feat(signals): artefacts as report response log#61545
Open
oliverb123 wants to merge 39 commits into
Open
Conversation
Contributor
|
Hey @oliverb123! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Contributor
Prompt To Fix All With AIFix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
products/signals/backend/report_generation/research.py:123-128
The multi-field validator always returns the same static message listing all three fields, even though it is called once per individual field. When only `contents` is empty, the error will say "file_path, contents, and relevance_note must not be empty", which is misleading. A single generic message (or a field-specific one) would be clearer for callers debugging validation failures.
```suggestion
@field_validator("file_path", "contents", "relevance_note")
@classmethod
def fields_must_not_be_empty(cls, v: str) -> str:
if not v.strip():
raise ValueError("must not be empty or whitespace-only")
return v
```
### Issue 2 of 2
products/signals/backend/report_generation/research.py:146-151
Same issue as in `CodeReference`: the error message always names all three fields even though the validator fires once per field. Changing to a generic message makes each failure unambiguous.
```suggestion
@field_validator("file_path", "diff", "relevance_note")
@classmethod
def fields_must_not_be_empty(cls, v: str) -> str:
if not v.strip():
raise ValueError("must not be empty or whitespace-only")
return v
```
Reviews (1): Last reviewed commit: "feat(signals): add code reference and co..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Signals’ SignalReportArtefact taxonomy to support attaching code-level evidence to reports by introducing two new artefact types (code_reference and code_diff) and documenting/validating their JSON shapes, without changing any producer or API write paths.
Changes:
- Added
CODE_REFERENCEandCODE_DIFFtoSignalReportArtefact.ArtefactType, with a corresponding choices-only Django migration. - Introduced Pydantic content schemas
CodeReferenceandCodeDiff(including basic non-empty validation and aend_line >= start_linevalidator). - Updated documentation and expanded the existing PUT rejection test matrix to include the new non-editable types.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
products/signals/backend/test/test_signal_report_artefact_api.py |
Extends the parameterized PUT-rejection coverage to include code_reference and code_diff. |
products/signals/backend/report_generation/research.py |
Adds Pydantic schemas for the new artefact content shapes, including validators. |
products/signals/backend/models.py |
Adds the two new enum values to SignalReportArtefact.ArtefactType. |
products/signals/backend/migrations/max_migration.txt |
Bumps the recorded max migration to 0028. |
products/signals/backend/migrations/0028_alter_signalreportartefact_type.py |
Updates type field choices to include the new artefact types (metadata-only change). |
products/signals/ARCHITECTURE.md |
Documents the new artefact types and their expected JSON payload shapes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Migration SQL ChangesHey 👋, we've detected some migrations on this PR. Here's the SQL output for each migration, make sure they make sense:
|
Contributor
🔍 Migration Risk AnalysisWe've analyzed your migrations for potential risks. Summary: 0 Safe | 2 Needs Review | 0 Blocked
|
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewThis PR adds report-response log artefacts for Signals, including backend endpoints and related task/GitHub integration for storing and acting on report artefact entries such as reviewers, repository selections, pushed branches, commits, and task associations. The PR has made progress with 4 issues already addressed, but 7 security issues remain open. The most significant remaining risks are that writable report artefacts can alter canonical pipeline status and reviewer/task flow, and GitHub-related artefacts can be used to access repository data through installation-token-backed endpoints without sufficient binding or input validation. Other open issues involve overly broad write capability for signal-report agents, weak task/report association trust, and mutation paths that do not consistently enforce per-task boundaries. Overall, the current posture still has clear attacker-visible impact and needs more tightening before it is safe to merge. Open issues (7)
Fixed/addressed: 4 · PR risk: 8/10 |
oliverb123
force-pushed
the
posthog-code/signals-code-artefact-types
branch
from
f1f9691 to
58f34c1
Compare
oliverb123
changed the title
oliverb123
force-pushed
the
posthog-code/signals-code-artefact-types
branch
from
3dd3f58 to
fdfd6c1
Compare
Contributor
MCP UI Apps size report
|
Contributor
|
Size Change: -38.1 kB (-0.06%) Total Size: 62.2 MB 📦 View Changed
ℹ️ View Unchanged
|
Contributor
|
⏭️ Skipped snapshot commit because branch advanced to The new commit will trigger its own snapshot update workflow. If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:
|
| operation_id="signals_report_artefacts_partial_update", | ||
| ) | ||
| def partial_update(self, request: ValidatedRequest, *args, **kwargs) -> Response: | ||
| artefact = cast(SignalReportArtefact, self.get_object()) |
There was a problem hiding this comment.
Medium: Artefact update/delete bypasses per-task access boundaries
A user with object-level write access to one task may be able to PATCH or DELETE log artefacts on reports that are not tied to that task. The view is authorized as scope_object = "task", but get_object() loads a SignalReportArtefact; that model does not map to the task resource for object permissions, so the detail action may be allowed after the generic “has any specific task access” fallback. Add an explicit resource-level task write check for these artefact mutations, or protect artefacts with a resource that maps to the report/artefact being modified.
Contributor
Author
There was a problem hiding this comment.
These are team-scoped resources, not task-scoped: safely_get_queryset filters to the report_id from the URL plus self.team (and excludes deleted reports), so a token can only ever reach artefacts belonging to its own team's reports. The task:write scope is reused deliberately as the capability gate (see ARTEFACT_LOG_PLAN.md) — the same model the pre-existing suggested_reviewers PUT uses. Reports aren't task-scoped, so there's no per-task object mapping to enforce here. Leaving as-is by design.
oliverb123
changed the title
2 tasks
Contributor
Query snapshots: Backend query snapshots updatedChanges: 3 snapshots (0 modified, 3 added, 0 deleted) What this means:
Next steps:
|
oliverb123
added a commit
to PostHog/code
that referenced
this pull request
Jun 9, 2026
The backend now appends a new suggested_reviewers status artefact on each edit (latest-wins) rather than mutating in place (PostHog/posthog#61545). The detail pane already derives current reviewers as the latest row, so update the optimistic mutation to append a synthetic latest row — mirroring the server — instead of patching the existing one, keeping the prior row in the work log as history. Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| ("code_diff", {"file_path": "a.py", "diff": "@@ -1 +1 @@", "relevance_note": "x"}), | ||
| ("line_reference", {"file_path": "a.py", "line": 3, "note": "here"}), | ||
| ("pushed_branch", {"repository": "PostHog/posthog", "branch": "fix/foo", "base_branch": "master"}), | ||
| ("task_run", {"task_id": "abc", "relationship": "signals_research"}), |
Contributor
There was a problem hiding this comment.
Invalid content schema for task_run artefact. The test passes {"task_id": "abc", "relationship": "signals_research"} but according to TaskRunArtefact schema in artefact_schemas.py (lines 112-131), the required fields are task_id, product, and type (not relationship). This will store malformed data that fails when parsed.
("task_run", {"task_id": "abc", "product": "signals", "type": "research"}),
Suggested change
| ("task_run", {"task_id": "abc", "relationship": "signals_research"}), | |
| ("task_run", {"task_id": "abc", "product": "signals", "type": "research"}), | |
Spotted by Graphite
Is this helpful? React 👍 or 👎 to let us know.
Contributor
Author
There was a problem hiding this comment.
Good catch — fixed in 81dd23d. Updated to {"task_id": "abc", "product": "signals", "type": "research"} to match the current TaskRunArtefact schema (the relationship field was replaced by product/type earlier in this PR).
| # Append a new status row rather than mutating in place: a human reviewer edit becomes a | ||
| # point-in-time entry in the work log, and latest-wins keeps it current. Appending a | ||
| # reviewers status also re-evaluates auto-start (handled in `append_status`, on commit). | ||
| new_artefact = SignalReportArtefact.append_status( |
There was a problem hiding this comment.
Medium: Reviewer edit can start a task as another user
A user who can PUT a suggested_reviewers artefact can add another org member, and append_status() then re-evaluates auto-start and may create an implementation task with that reviewer as user_id. Do not trigger auto-start from this generic reviewer-edit path unless the selected assignee explicitly initiated or approved the run.
oliverb123
force-pushed
the
posthog-code/signals-code-artefact-types
branch
from
3291995 to
1fdb3bf
Compare
Auto-start only ran at report-creation time (agentic + custom-agent paths). With suggested_reviewers now append-only and editable via the artefact API, a human adding a reviewer whose autonomy threshold qualifies never re-triggered it. The reviewer PUT now re-evaluates auto-start from the report's current artefacts via a new `maybe_autostart_from_report_artefacts` helper (reconstructs the latest actionability / priority / repo-selection / reviewers). It delegates to the existing `maybe_autostart_implementation_task`, which is idempotent — no-ops if an implementation task already exists. Best-effort and post-commit so it never fails the edit or starts a task that gets rolled back. Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
Move the reviewer-change auto-start re-evaluation out of the artefact viewset and into SignalReportArtefact.append_status, so any path that appends a suggested_reviewers status re-runs the (idempotent) auto-start check rather than each caller having to remember to. Scheduled via transaction.on_commit (post-commit, not inside the atomic block) with a lazily-imported helper to avoid a models <-> auto_start cycle; best-effort so it never breaks the write. Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
Route the agentic pipeline and custom-agent persistence through the SignalReportArtefact append helpers (append_status / append_finding) instead of bulk_create, so the model is the single artefact write path. Adds append_finding for the signal_finding type (neither status nor log). Auto-start behaviour is unchanged: these paths orchestrate it explicitly on the worker with full in-hand context, so their suggested_reviewers append passes reevaluate_autostart=False to opt out of the model's on-commit auto-start hook (which still serves the human reviewer-edit path and any future caller). Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
The custom agent's report is fully persisted (artefacts + title/summary) before auto-start runs, so it can use the same reconstruct-from-artefacts entry point (`maybe_autostart_from_report_artefacts`) the in-app reviewer edit uses, instead of passing in-hand objects to `maybe_autostart_implementation_task`. Behaviour is unchanged. The agentic pipeline keeps passing inputs in hand, since its research-refined title/summary aren't written to the report row until later. Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
Serialize implementation-task auto-start under a report row lock so the several trigger paths (reviewer-edit on-commit hook, agentic pipeline, custom agents) can't race into duplicate tasks. Make the agentic research reload legacy-tolerant for priority/finding artefacts, cap the branch diff size, and validate the diff endpoint's repo/branch before any GitHub call. Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
Master took 0035/0036 for the signal team config auto-start default, so the artefact updated_at + latest-index migrations move to 0037/0038. Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
…ts per task
Reworks the report artefact log around three properties:
- Unified content model: every artefact type has a pydantic schema in
artefact_schemas.py, collected in a registry and enforced on the write
path (model helpers + API serializers). Reads stay legacy-tolerant.
- Attribution: every write declares a producer — a user, a task, or
explicitly the system — via a required ArtefactAttribution argument,
persisted as nullable created_by/task FKs. Agent writes are attributed
deterministically through a sandbox-set X-PostHog-Task-Id header
forwarded by the MCP server; the LLM never handles its own task id.
- Commits, not branches: the pushed_branch type and branch-diff endpoint
are replaced by per-commit artefacts with an on-demand single-commit
diff (GitHubIntegration.get_commit_diff).
Task↔report associations lose their relationship label: tasks simply
associate (unique per pair, free-form via POST reports/{id}/tasks/ or
the agent's own header task), and purpose is derived from task_run
artefacts — which now also drive auto-start idempotency, inside the
existing report-row lock. Sandbox agents get the write tool surface via
a signals_report MCP scope preset (read scopes, read-only mode off).
Generated-By: PostHog Code
Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
Generated-By: PostHog Code Task-Id: 3afae508-37cd-4bd9-9624-cffcd7c1a486
…port Adds register_artefact (typed on the artefact_schemas content models, so validation fails at the call site) plus register_note / register_code_reference / register_code_diff / register_line_reference conveniences to CustomSignalAgent. Queued entries are persisted via add_log in the same transaction as the report, attributed like every other component (the sandbox task when one exists, else system), and cleared by report_and_continue. commit and task_run stay excluded — they're owned by the signed-commit hook and report persistence respectively. Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
Adds an `agent_note` field to the MCP tool YAML: a brief work-protocol note the generated handler appends to the tool *response* as `_agentNote`, so the agent sees it when it actually fetches the resource instead of growing the always-loaded tool description. inbox-reports-retrieve uses it to outline the work-log protocol — associate your task once, then log notes / code references / diffs / out-of-band commits as artefacts — deferring content shapes to each tool's description. Also refreshes the stale tool-schema snapshots from the pushed_branch era (the create tool's type list now includes `commit`). Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
Status vs log classifies semantics, not ownership — nothing is pipeline-owned. POST now appends any artefact type via SignalReportArtefact.append, which routes to the type's semantics: status types are latest-wins (a new version supersedes the previous as the report's canonical status, with auto-start re-evaluation on reviewers), signal_finding stays keyed by signal_id, dismissals stack, log types accumulate. PATCH edits any artefact (validated against the row's type; editing the latest reviewers row re-evaluates auto-start), and DELETE works on any artefact — deleting the latest status row reverts the canonical status to the previous version. Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
…ontract The artefact list/retrieve endpoints were excluded from the OpenAPI schema, so agents genuinely couldn't read judgments or findings through MCP — and the retrieve tool's description wrongly implied that was by design. Un-excludes them (the bespoke reviewers PUT stays app-only) and ships inbox-report-artefacts-list / -retrieve as read tools. Tool copy now answers what test agents kept asking: report lifecycle is pipeline-managed (act on `ready`; `pending_input` is waiting on a human — surface it, don't act on it), and status judgments are consequential, not advisory — canonical priority/actionability steer what humans, agents, and PostHog's automation pick up next. Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
A free-form task↔report association only created the link row, so the Runs section showed the task while the activity log stayed empty — the two read different sources (link rows vs artefacts). Associating now also appends a task_run artefact (skipped on idempotent re-association; pipeline-created links already write their own), labelled by optional product/type body fields following the custom-agent identifier convention (defaults tasks/agent_run). Invalid identifiers 400 and roll the link back. Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
…tion
Removes the /signals/reports/{id}/tasks/ surface (viewset, serializers,
routes, MCP tool) and all SignalReportTask reads and writes — the link table
was redundant with task_run artefacts, which carry the same association on
their task attribution FK plus the run's identity.
Associating is now appending a task_run artefact: content.task_id defaults
to the X-PostHog-Task-Id header ("associate me"), product/type default to
tasks/agent_run, attribution is always the recorded task, and re-associating
is idempotent (returns the existing entry). The reports ?task_id= filter,
implementation_pr_url, and the merged-PR webhook resolution all derive from
artefacts. The deprecated SignalReportTask model survives only so existing
rows can be converted (backfill now also covers unlabelled rows); the table
drop is a follow-up migration.
Generated-By: PostHog Code
Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
… edit The tasks-create removal span accidentally swallowed the adjacent inbox-reports-list entry; restore it and regenerate. Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
…er master rebase Master took 0037 for signalscoutemission tags; the artefact migrations shift to 0038-0040. Regeneration picks up the scout-config-create tool and the priority dollar_value field alongside the artefact changes. Generated-By: PostHog Code Task-Id: 03657aa4-20d8-4dea-aeef-e8c1ab4bbce2
oliverb123
force-pushed
the
posthog-code/signals-code-artefact-types
branch
from
f196aed to
797a037
Compare
…er pipeline Review-pass simplifications over the artefact-log feature: - Artefact content is strongly typed end to end: parse_artefact_content is the single boundary parser (API writes, reads of stored rows), the model helpers take pydantic content models and derive the row's type from the model class, and validate_artefact_content's string-in/string-out gate is gone. Stored content is the normalized model dump. - Auto-start idempotency moves off the freeform artefact log onto a real SignalReport.implementation_task gate column, compare-and-set via the shared record_implementation_task helper (auto-start + manual start path). - The research agent is read-only: its findings/judgments are extracted from the multi-turn session and persisted by the pipeline, which now appends a new artefact version only when something changed (the agent can confirm a still-correct finding/judgment via the *Update wrapper schemas). A research task_run association is recorded on every run. Implementation agents run with full scopes so they can log their work; the signals_report preset is removed. - register_artefact collapses to one generic, typed method (no registerable subset, no typed convenience wrappers); persistence routes any type through append's latest-wins/status semantics. - The inbox notification gate is simply PR presence from any associated task, polled until timeout behind a new workflow patch — no task-shape heuristics. - code_diff artefact type removed; code/line references get line-length and line-count bounds; commit SHA shape and GitHub ref "safety" validation removed (GitHub's to reject). - Migrations collapsed to two: one atomic schema migration and one non-atomic concurrent index using the idempotent CreateIndexConcurrently helper instead of bare AddIndexConcurrently. Generated-By: PostHog Code Task-Id: f65424ca-a7fe-49f5-9f75-7a3cf004b417
|
|
||
| try: | ||
| response = self._github_api_get( | ||
| f"https://api.github.com/repos/{repo_path}/commits/{sha}", |
There was a problem hiding this comment.
High: GitHub API path injection
repository and sha come from a writable commit artefact and are interpolated directly into the GitHub API path. A caller can set repository to a path like owner/repo/contents/path?ref=main#; the access check uses the same raw string, and this request then returns the contents API response through the installation token instead of a commit diff. Validate repository as exactly owner/repo, validate sha as a commit SHA, and URL-encode path segments before making the request.
mypy: annotate the schema unions at multi-turn send_followup call sites so the generic return type resolves to the union instead of BaseModel; guard against anonymous-user attribution; drop a stale type: ignore and an over-narrow test helper annotation. semgrep: add team filters to the SignalReport lookups in auto-start, the implementation-task gate write, and the reviewers PUT lock (idor-lookup-without-team). Generated-By: PostHog Code Task-Id: f65424ca-a7fe-49f5-9f75-7a3cf004b417
Contributor
🕸️ Eager graphHow much code each root forces the browser to download and decode through static imports — the regression class total bundle size can't see.
✅ Largest files eagerly reachable from
|
| Size | File |
|---|---|
| 878.2 KiB | src/styles/global.scss |
| 609.0 KiB | public/hedgehog/burning-money-hog.png |
| 541.9 KiB | public/hedgehog/waving-hog.png |
| 448.2 KiB | public/hedgehog/stop-sign-hog.png |
| 362.0 KiB | public/hedgehog/phone-pair-hogs.png |
| 354.8 KiB | ../node_modules/.pnpm/@posthog+icons@0.36.6_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 335.6 KiB | public/hedgehog/desk-hog.png |
| 323.2 KiB | public/hedgehog/3-bears-hogs.png |
| 297.4 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 287.5 KiB | src/lib/api.ts |
Largest files eagerly reachable from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 878.2 KiB | src/styles/global.scss |
| 760.0 KiB | src/queries/validators.js |
| 609.0 KiB | public/hedgehog/burning-money-hog.png |
| 541.9 KiB | public/hedgehog/waving-hog.png |
| 448.2 KiB | public/hedgehog/stop-sign-hog.png |
| 398.7 KiB | ../node_modules/.pnpm/chart.js@4.5.1/node_modules/chart.js/dist/chart.js |
| 362.0 KiB | public/hedgehog/phone-pair-hogs.png |
| 354.8 KiB | ../node_modules/.pnpm/@posthog+icons@0.36.6_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 335.6 KiB | public/hedgehog/desk-hog.png |
| 323.2 KiB | public/hedgehog/3-bears-hogs.png |
Posted automatically by check-eager-graph · sizes are input-source bytes from the esbuild metafile · part of #32479
…tools Agents reading a report via inbox-reports-list never saw the association instruction (it lived only on the retrieve tool's note), and the task_run artefact's role as THE association was buried mid-list in the create tool description. Point both report read tools at the artefact log and state plainly: any work prompted by a report — research included — starts by appending a task_run artefact, or commits never link back to the report. Generated-By: PostHog Code Task-Id: f65424ca-a7fe-49f5-9f75-7a3cf004b417
Conflict resolutions: research.py judgment schemas live in artefact_schemas.py (master's priority-explanation wording ported there); business-knowledge prompt wiring combined with the typed schema-union call sites; generate-tools.ts keeps master's enrich-url suffix inside the agent_note wrapper, and master's new confirmed_action return path gains the agent_note fields its result type requires. Generated-By: PostHog Code Task-Id: f65424ca-a7fe-49f5-9f75-7a3cf004b417
Generated-By: PostHog Code Task-Id: f65424ca-a7fe-49f5-9f75-7a3cf004b417
| reports = ( | ||
| SignalReport.objects.filter(report_tasks__task_id=task_id) | ||
| SignalReport.objects.filter( | ||
| id__in=SignalReportArtefact.objects.filter( |
There was a problem hiding this comment.
Medium: User-mutable task associations resolve reports
task_run artefacts are appendable through the report artefact API, and that path only checks that the task belongs to the same project. A caller with task:write can associate an unrelated report with a task they control; when that task's PR is merged, this webhook will mark the report resolved. Use a trusted server-written link for resolution, such as SignalReport.implementation_task_id, or otherwise require the association to be a server-created implementation task rather than any task_run artefact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overall idea: reports are a living object, the "core" around which work happens autonomously within posthog. To that end, we need a way to produce a "log" of that work that's happening - different agents will run, they'll push commits or create insights or flags, or suggest reviewers, or whatever, and when they do, they should add a small note to the report about what they found or did. The UI for this is rough right now (see PostHog/code#2557), but as a directional idea it looks like a log of artifacts:
We'll work on the naming - "artefact log" is obviously incorrect, "report log" is probably better, or (lord help me) "responder log".
Notable artefact types:
The existing assessment/judgement artefacts still exist, and are slightly unique, in so far as they (unlike some of the later ones) are closer to a "status" than a "log item" - as such, they've been changed slightly to act as a kind of "last write wins" class of objects, where the "status" of a report (it's assigned reviewers, it's priority, etc) is derived from whatever the most recently written artefact of that type is. For example, here are two items in the log:
The final set of assigned reviewers for the report is then:
Priority and actionability work similarly.
We also have a new task-run artefact, which replaces
SignalReportTaskentirely - atask_runartefact is the task↔report association now, and the task's purpose is derived from the artefact's(product, type)(I've included a management command to migrate the legacy rows to artefacts, and will run it once this is in place; dropping the now-dead table is a follow-up migration). This allows much more free-form association of tasks to reports, which I think will matter a lot as a given report results in a higher number of individual tasks (e.g. with Peters "workstream" concept, if a work stream is a report (and it should be), there can be a quite large number of tasks. Other products building autonomous features on top of reports/the inbox will also need more freeform association). A running agent can now also associate its own task with any report it decides its work relates to, then start producing artefacts. They look like this:We also now have "code reference", "diff proposed" and "line reference" artefacts. The first and third will be useful, in my opinion, for helping an Human/LLM walk through a PR or set of PR's during review, letting responder agents highlight particular findings or suggest point changes without pushing a whole branch.
Then there is also the full on diff view - this started life as a "pushed branch" artefact diffed against a base branch, but I've since decided against that model: we now track individual commits instead. The agent harness records one commit artefact per pushed commit (automatically, from the signed-commit tool - see the Code PR), and the diff view renders that commit's diff inline (again, this needs to be quite a bit prettier really):
Since the first cut, the original to-do list has landed in this PR:
Wiring up artefact creation for custom agents, and for the agents run by the pipeline, so they emit these— done: asignals_reportMCP preset gives the pipeline's research/implementation sandboxes the write tools, and commits are recorded automatically by the harnessArtefacts should, wherever possible, point back to the thing that created them— done: every new artefact is attributed to a user, a task, or (explicitly) the system, enforced as a required argument at the model helpers. Agent writes are attributed deterministically via anX-PostHog-Task-Idheader baked into the sandbox MCP config — the LLM never handles its own task idAlmost certainly the existing auto-start logic doesn't handle this properly— done: auto-start idempotency now keys on the implementation task-run artefact, inside the existing report-row lockSharing mostly so people can pull it, play around with it, and so people know what I mean when I say "artefacts as log"
Update from the review pass: the research agent no longer holds the artefact write tools — its findings and judgments are extracted from the multi-turn session by the pipeline, which persists them deterministically and now only appends a new artefact version when something actually changed (on re-research the agent confirms a still-correct finding/judgment instead of regenerating it). Implementation agents do get write access — they now run with
fullMCP scopes so they can log notes, code references, and diffs while they work.AI slop below (probably not useful):
Problem
A
SignalReportArtefactwas a write-once, replaced snapshot: the agentic pipeline deleted and re-created its artefact types on every run, and nothing else wrote them. We want artefacts to become an append-only log of the work done on a report — so a signal report reads as a living document: the evidence the research agent gathered, the commits it pushed, the task runs that executed, and free-form notes, all accumulating over time — with every entry validated against a per-type schema and attributed to whoever produced it.Changes
Turns
SignalReportArtefactinto an attributed, schema-validated work-log with a typed write surface.Model + migrations
updated_at(nullable,auto_now) — log artefacts are editable in place.created_by(FKposthog.User,SET_NULL) andtask(FKtasks.Task,SET_NULL), nullable for legacy rows. Every write helper takes a requiredArtefactAttribution(user | task | system) so no write site can silently skip attribution.code_reference,code_diff,line_reference,commit,task_run,note). Expressed asSTATUS_ARTEFACT_TYPES/LOG_ARTEFACT_TYPES.signal_findingis keyed by(report, signal_id)anddismissalentries stack; both have dedicated appenders.append_status,add_log,append_finding,append_dismissal,update_content— all funnelled through a single validated, attributed create.SignalReportTaskis fully deprecated — no longer read or written anywhere; atask_runartefact is the sole task↔report association (the table drop is a follow-up migration). Purpose is derived from the artefact's(product, type);implementation_pr_urlis the newest PR from any artefact-associated task; auto-start idempotency keys on the implementationtask_runartefact inside the existingselect_for_updateblock.updated_at+ choices migration, a concurrent latest-wins index on(report, type, -created_at), and the attribution/constraint migration (all nullable ADD COLUMNs and metadata-only ALTERs).Content schemas —
artefact_schemas.py, a dependency-light pydantic module, is now the canonical home of every artefact content shape, collected in anARTEFACT_CONTENT_SCHEMASregistry (a test asserts exact coverage of the type enum).validate_artefact_contentis the single write gate, called by the model helpers and the API serializers — validation is a gate, not a rewrite, and reads stay generic JSON for back-compat with legacy rows.Write API —
SignalReportArtefactViewSetgains POST / PATCH / DELETE for artefacts of any type (writing a status type appends a new latest-wins row), plus schema-documented list/retrieve, reusingscope_object = "task"(no new scope), team-scoped via the existing queryset, with per-type schema validation. POSTing atask_runartefact is the "associate me" operation:content.task_iddefaults from the task-id header (socontent: {}suffices for an agent),product/typedefault totasks/agent_run, the named task must belong to the team, and re-association is idempotent. Writes are attributed to the task named by anX-PostHog-Task-Idheader (set automatically for sandbox agents and forwarded by the MCP server) or to the requesting user. The bespokesuggested_reviewersPUT path is untouched bar attribution. Adiffaction renders a commit artefact's unified diff viaGitHubIntegration.get_commit_diff. The reports list accepts?task_id=, resolved throughtask_runartefacts.MCP tools —
inbox-report-artefacts-create/-update/-delete(task:write) and-list/-retrieve(task:read), surfaced to the pipeline's sandboxes via a newsignals_reportscope preset (scope-identical toread_only, but read-only mode no longer strips thetask:writetools). A newagent_noteYAML field lets any tool append brief point-of-use guidance to its responses;inbox-reports-retrieveuses it to teach agents the artefact protocol (associate via atask_runartefact, then log work as artefacts) without bloating tool descriptions.Backfill —
backfill_task_run_artefactsconverts legacy relationship-labelledSignalReportTaskrows intotask_runartefacts (idempotent, with--dry-runand--team-id); unlabelled rows get defaulttasks/agent_runartefacts, since artefacts are now the only association.Pipeline — the agentic pipeline's findings/judgments/reviewers are attributed to the research sandbox task, repo selection to its selection task (when one ran), and the safety judge explicitly to the system.
Out of scope (follow-ups): dropping the
SignalReportTasktable once the backfill has run, commit artefacts forgit_signed_rewrite, and a review of the fulltask:writetool surface thesignals_reportpreset exposes.How did you test this code?
I'm an agent. Automated checks run in a full dev environment:
pytestacross the signals, tasks, integration-model, and oauth suites — 1,750+ passing (write API incl. attribution + per-type validation, model helpers, commit diff action, artefact-based association incl. idempotency and header defaulting, auto-start idempotency, backfill), plus the full MCP vitest suite.ruff check+ruff format --check— clean;tach check— clean.makemigrations --check --dry-run— no drift;sqlmigrateon 0037/0038/0039 — nullable ADD COLUMNs, a concurrent index, and metadata-only ALTERs.hogli build:openapi— regenerated the OpenAPI spec, frontend types, and MCP tool definitions.Automatic notifications
Docs update
ARTEFACT_LOG_PLAN.mddocuments the shipped design;products/signals/backend/management/AGENTS.mddocuments the backfill command.🤖 Agent context
Autonomy: Human-driven (agent-assisted) — Oliver Browne.
Authored via the PostHog Code agent. The PR was reworked in place mid-flight: the original
pushed_branch+ branch-diff surface was replaced by per-commit artefacts before ever shipping, the relationship labelling on task links was removed in favour of artefact-derived purposes, and finally theSignalReportTasklink table itself was retired — atask_runartefact is the association. Design notes: attribution is enforced at the model-helper signature (a required discriminated argument) rather than a DB constraint, because legacy rows and explicit system writes legitimately carry NULLs; the task-id header is attribution metadata, not an authorization boundary — the token is already team-scoped and the named task must belong to the same team; content validation is wired server-side through a single registry gate, but stored payloads keep forward-compatible extra keys and reads never re-validate legacy rows.Created with PostHog Code