Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ This document describes how to contribute to the project, with emphasis on the *

## Creating a new collector

**Start here:** [docs/Tutorial_building_a_collector.md](docs/Tutorial_building_a_collector.md) — step-by-step walkthrough (scaffolding, `AbstractCollector` hooks, testing, YAML/Celery, deployment) with a worked `heartbeat_demo` example.

Use the **`startcollector`** management command to generate a new Django app with the usual collector layout (stub `models.py`, `services.py`, `AbstractCollector` + `BaseCollectorCommand`, `tests/` package, `migrations/0001_initial.py`, and `schedule_snippet.yaml`). Run it from the **repository root** so the new package sits next to the other apps.

```bash
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ If you see `relation "cppa_user_tracker_githubaccount" does not exist` (or simil
python manage.py run_scheduled_collectors --schedule daily --group github
```

7. To **add a new collector app** (boilerplate, management command, and schedule snippet template), use **`python manage.py startcollector <name>`** and follow **[CONTRIBUTING.md](CONTRIBUTING.md#creating-a-new-collector)**.
7. To **add a new collector app**, follow **[docs/Tutorial_building_a_collector.md](docs/Tutorial_building_a_collector.md)** (walkthrough), then **`python manage.py startcollector <name>`** and **[CONTRIBUTING.md](CONTRIBUTING.md#creating-a-new-collector)** (checklist).

For local development you can start the dev server: `python manage.py runserver`.

Expand Down
2 changes: 1 addition & 1 deletion core/_version.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
# file generated by setuptools-scm; do not edit
version = "0.1.1.dev561+g0dd532eaf.d20260527"
version = "0.1.1.dev2+g0dd532eaf.d20260527"
2 changes: 1 addition & 1 deletion core/collectors/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Collector orchestration shared by every `run_*` management command.

## Usage

New collectors should subclass `AbstractCollector` and wire a management command through `BaseCollectorCommand`. See [How to add a collector](../../docs/How_to_add_a_collector.md) and the parent [core README](../README.md).
New collectors should subclass `AbstractCollector` and wire a management command through `BaseCollectorCommand`. See [Tutorial: building a collector](../../docs/Tutorial_building_a_collector.md) (walkthrough), [How to add a collector](../../docs/How_to_add_a_collector.md) (checklist), and the parent [core README](../README.md).

## Tests

Expand Down
4 changes: 4 additions & 0 deletions docs/How_to_add_a_collector.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# How to add a collector

**Tutorial (start here):** [Tutorial_building_a_collector.md](Tutorial_building_a_collector.md) — end-to-end walkthrough with design decisions for scaffolding, `AbstractCollector` hooks, testing, Celery scheduling, and deployment.

**Preferred:** scaffold a new app from the repo root with **`python manage.py startcollector <app_label>`** (see [CONTRIBUTING.md — Creating a new collector](../CONTRIBUTING.md#creating-a-new-collector)), then follow the manual steps there (`INSTALLED_APPS`, YAML schedule, migrations, **cross-app-dependencies.md**).

**Layout note:** `startcollector` places the collector class and `BaseCollectorCommand` in **`management/commands/run_<app>.py`**. Split into a separate **`collectors.py`** when the command grows (see the tutorial §2.5). Section 4 below uses a `collectors.py` layout as an alternate copy-paste skeleton, not the default scaffold output.

If you used **`startcollector`**, section 1 below is mostly satisfied (you still add the app to **`INSTALLED_APPS`**). Otherwise, this checklist assumes you already have a Django app (or are creating one) with a `management/commands/run_<your_app>.py` entry point. For a high-level diagram and GitHub pipeline notes, see the **Architecture** section in [Development_guideline.md](Development_guideline.md).

## 1. App and command
Expand Down
9 changes: 5 additions & 4 deletions docs/Onboarding.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ For setup steps (venv, migrate, tests), start with the root **[README.md](../REA
| 3 | [Architecture_data_flow.md](Architecture_data_flow.md) | Sources → collectors → DB / workspace → Pinecone (diagrams). |
| 4 | [Workflow.md](Workflow.md) | YAML schedules, Celery Beat, execution order. |
| 5 | [CONTRIBUTING.md](../CONTRIBUTING.md) | Service-layer rule for DB writes. |
| 6 | [Workspace.md](Workspace.md) | Where files land under `WORKSPACE_DIR`. |
| 7 | [Schema.md](Schema.md) — § Overview + diagrams for your area | Cross-app tables (identity, GitHub, Boost libraries). |
| 8 | [Service_API.md](Service_API.md) + `service_api/<app>.md` | Callable surface for writes you must use. |
| 9 | [operations/README.md](operations/README.md) | Shared I/O (GitHub, etc.), not the same as services. |
| 6 | [Tutorial_building_a_collector.md](Tutorial_building_a_collector.md) | **First collector:** scaffold → hooks → tests → schedule → deploy. |
| 7 | [Workspace.md](Workspace.md) | Where files land under `WORKSPACE_DIR`. |
| 8 | [Schema.md](Schema.md) — § Overview + diagrams for your area | Cross-app tables (identity, GitHub, Boost libraries). |
| 9 | [Service_API.md](Service_API.md) + `service_api/<app>.md` | Callable surface for writes you must use. |
| 10 | [operations/README.md](operations/README.md) | Shared I/O (GitHub, etc.), not the same as services. |

Deep dives when you touch an area: **[Docker.md](Docker.md)**, **[Deployment.md](Deployment.md)**, per-app notes under **`docs/service_api/`** and **`docs/operations/`**.

Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Documentation is organized **by topic**, not by app. Each doc covers one cross-c
| **Architecture overview** | [Architecture_overview.md](Architecture_overview.md) | **Start here for system design:** all 15 domain apps + `core`, persistence, coupling, links to app READMEs and service API. |
| **Workflow** | [Workflow.md](Workflow.md) | Main application workflow, execution order, and project details. |
| **Architecture (data flow)** | [Architecture_data_flow.md](Architecture_data_flow.md) | Data flow (sources → collectors → DB / workspace → Pinecone), orchestration diagram, per-app component map. |
| **Tutorial: building a collector** | [Tutorial_building_a_collector.md](Tutorial_building_a_collector.md) | End-to-end walkthrough: `startcollector`, hooks, tests, YAML/Celery, deploy. |
| **Cross-app dependencies** | [cross-app-dependencies.md](cross-app-dependencies.md) | FK/import matrix, import-linter contracts, regeneration via `list_cross_app_imports.py`. |
| **CODEOWNERS / reviews** | [CODEOWNERS_and_branch_protection.md](CODEOWNERS_and_branch_protection.md) | CODEOWNERS behavior, enabling branch protection, verification checklist. |
| **Onboarding walkthroughs** | [onboarding/](onboarding/README.md) | 1:1 session runbooks (Leo, Jonathan) and session logs. |
Expand Down
Loading
Loading