SOW-0004: codex adapter (Scan + Tail + forks + reasoning + turn-boundary evolution)

.agents/sow/pending/SOW-0020-20260530-catalog-idempotency.md → .agents/sow/done/SOW-0020-20260530-catalog-idempotency.md

@@ -2,10 +2,9 @@
 
 ## Status
 
-Status: pending
+Status: closed
 
-Sub-state: proposed (2026-05-30) during SOW-0003 Round-6 review. Awaiting operator sign-off.
-Independent of the adapters; an ingester-layer correctness fix.
+Sub-state: SUPERSEDED 2026-05-30 by SOW-0004 (codex adapter), which absorbed this catalog-idempotency-under-re-emission fix as a prerequisite: `onOpStarted` counts a call once per op (insert-signal) and migrates the contribution on an identity change keyed on the effective post-upsert identity (empty→prior); `onOpFinalized` applies a now-minus-prior delta. The fix is adapter-agnostic and benefits aiagent_v2/v3 + claude_code + codex. See SOW-0004 `## Reviews` rounds 4–6 and `internal/ingest/{catalog.go,catalog_migrate.go,writer.go}` + `catalog_idempotency_test.go`. Moved to `done/` as a closed (superseded) record. Originally: proposed during SOW-0003 Round-6 review as an independent ingester-layer correctness fix.
 
 ## Requirements
 

.agents/sow/pending/SOW-0021-20260530-turn-extras-carrier.md

@@ -0,0 +1,77 @@
+# SOW-0021 - turn-extras carrier (populate turns.extras_json)
+
+## Status
+
+Status: open
+
+Sub-state: proposed follow-up, awaiting operator prioritization. Discovered during SOW-0004 (codex adapter) Chunk B. Not blocking SOW-0004 — codex ships an interim no-loss workaround.
+
+## Requirements
+
+### Purpose
+
+Make `turns.extras_json` reachable. The `turns` table defines an `extras_json TEXT` column (data-model.md:112, with `codex_turn_id` cited as the canonical example value), but **no canonical turn event carries an `Extras` field** (`TurnStartedEvent`/`TurnFinalizedEvent` in `internal/canonical/events.go:220,233` have none) and the ingest writer never populates `turns.extras_json`. So every per-turn extra the specs promise is structurally unreachable from any adapter. This SOW adds a turn-extras carrier to the canonical event model + ingest writer so adapters can populate `turns.extras_json`, and migrates the codex adapter's interim surface onto it.
+
+### User Request
+
+Implied by the data-model + adapter specs, which document `turns.extras_json.{codex_turn_id,sandbox,ttft_ms}` (adapter-codex.md "Canonical Model Gaps" #2/#3/#8) and `claude-code system.subtype='turn_duration'` (data-model.md:112) as the durable home for per-turn metadata. SOW-0004 surfaced that this home is unwired.
+
+### Assistant Understanding
+
+Facts:
+
+- `internal/canonical/events.go`: `TurnStartedEvent` = {EventBase, SessionNativeID, Seq}; `TurnFinalizedEvent` = {EventBase, SessionNativeID, Seq, Status, ErrorClass, EndTs, Tokens*, CostUSD}. Neither carries `Extras`.
+- `internal/ingest/writer.go`: the `turns` UPSERT paths never write `extras_json`; the `graftAiViewerExtras` extras handling is ops/sessions-only.
+- `data-model.md:98-112`: the `turns` table has `extras_json TEXT`.
+- SOW-0004 codex adapter computes per-turn `codex_turn_id`, `sandbox`, and `ttft_ms` on its `turnState` and currently surfaces them via a single informational `turn_meta` LogEntry at turn finalize (no silent loss) — `internal/adapters/codex/ops_event.go` (`turnExtrasLog`).
+
+Inferences:
+
+- The cleanest carrier is an `Extras map[string]any` field on `TurnFinalizedEvent` (turn extras are known by the time the turn finalizes), mirroring how ops/sessions carry `Extras`; the writer then marshals it into `turns.extras_json` on the turn UPSERT, mirroring the ops/sessions extras write. A `TurnUpdatedEvent` is an alternative if mid-turn extras are ever needed, but no current adapter needs that.
+- This is shared infrastructure: claude-code (`turn_duration`), codex, and future adapters all benefit. It is deliberately out of SOW-0004's codex-only-additive blast radius.
+
+Unknowns:
+
+- Whether `turns.extras_json` needs the same per-key graft protection the ops/sessions paths use (re-emit safety). Turn finalize is terminal and single-shot per (session,seq), so a wholesale write is likely safe — confirm against the idempotent-write model (SOW-0015) during the gate.
+
+### Acceptance Criteria
+
+1. `TurnFinalizedEvent` carries an `Extras` field (or an equivalent carrier); `internal/canonical` tests cover it. **Verification**: `go build`/`go test` for canonical.
+2. The ingest writer marshals turn `Extras` into `turns.extras_json` on the turn UPSERT, idempotently. **Verification**: an ingester test asserts a `TurnFinalizedEvent{Extras:{...}}` lands in `turns.extras_json`, and a re-emit does not corrupt it.
+3. The codex adapter populates `turns.extras_json.{codex_turn_id,sandbox,ttft_ms}` via the carrier and **removes** the interim `turn_meta` LogEntry. **Verification**: codex golden/mapper tests assert the turn extras on the event; the `turn_meta` LogEntry is gone.
+4. Specs reconciled: adapter-codex.md "Canonical Model Gaps" v1-reachability note removed/updated; data-model.md + canonical-events.md describe the turn-extras carrier. **Verification**: spec-drift sweep clean.
+
+## Analysis
+
+Sources checked: `internal/canonical/events.go`, `internal/ingest/writer.go`, `.agents/sow/specs/{data-model.md,canonical-events.md,adapter-codex.md}`, `internal/adapters/codex/ops_event.go`.
+
+Current state: discovered 2026-05-30 during SOW-0004 Chunk B. Codex ships an interim no-loss `turn_meta` LogEntry; this SOW migrates to the real column.
+
+Risks:
+
+- **R1 — Shared-surface change.** Touches `internal/canonical` + `internal/ingest`, used by every adapter. Mitigation: additive field (no existing adapter sets it → no behavior change for v2/v3/claude-code until they opt in); full gate + external review.
+- **R2 — Idempotency.** Re-emitted turn finalize must not corrupt `turns.extras_json`. Mitigation: confirm against SOW-0015 idempotent-write model in the gate; test the re-emit path.
+
+## Pre-Implementation Gate
+
+(To be filled by the assistant picking this SOW up. Required before moving to `current/`.)
+
+## Implementation
+
+(Empty placeholder.)
+
+## Validation
+
+(Empty placeholder.)
+
+## Reviews
+
+(Empty placeholder.)
+
+## Outcome
+
+Pending.
+
+## Lessons / Follow-Ups
+
+Pending.

.agents/sow/pending/SOW-0022-20260530-codex-duplicate-rollout-id.md

@@ -0,0 +1,75 @@
+# SOW-0022 - codex duplicate-rollout-id disambiguation
+
+## Status
+
+Status: open
+
+Sub-state: proposed follow-up, awaiting operator prioritization. Discovered during SOW-0004 (codex adapter) round-6 review (codex P3). Not blocking — unobserved edge, accepted for v1.
+
+## Requirements
+
+### Purpose
+
+Honor `adapter-codex.md` edge case #14: when two codex rollout files carry the same `session_meta.payload.id`, they must become SEPARATE canonical sessions keyed on `(source_id, native_id + ":" + file_basename)` with a `LogEntry` warning — instead of collapsing into one session as they do in SOW-0004's v1.
+
+### User Request
+
+Implied by `adapter-codex.md` edge #14 (the spec documents the disambiguation behavior). Surfaced by codex's round-6 review of SOW-0004 as the one remaining (P3, non-blocking) gap.
+
+### Assistant Understanding
+
+Facts:
+
+- SOW-0004's codex adapter sets `SessionStartedEvent.NativeID = session_meta.payload.id` (authoritative; `internal/adapters/codex/mapper_turn.go`), and the ingester upserts sessions on `(source_id, native_id)` (`internal/ingest/writer.go`). So two rollout files with the same `payload.id` upsert into ONE `sessions` row — they collapse.
+- The spec (`adapter-codex.md` edge #14) intends them kept separate via `native_id + ":" + file_basename` + a warning.
+- This is **unobserved**: 0 of the 2,566 modern files on the reference workstation have duplicate ids. It would require codex to resume into a forked thread writing the same id to two files.
+- The per-file adapter cannot detect the collision alone — it has no cross-file view. Detection needs either the ingester (which sees the `(source_id, native_id)` conflict) to disambiguate, or the adapter to track seen ids across files within a Scan/Tail run.
+
+Inferences:
+
+- The cleanest home is likely the ingester: on a sessions upsert where the incoming `start_ts`/file differs from the existing row in a way that indicates a DIFFERENT physical file with the same native_id, disambiguate by appending the basename. But that needs the file basename threaded into the SessionStarted extras (the adapter has it).
+- Alternatively the adapter carries the basename in extras and the ingester composes the disambiguated native_id when it detects a collision. Design decision for the gate.
+
+Unknowns:
+
+- Whether to disambiguate in the adapter (cross-file id set per run) or the ingester (collision-on-upsert). Resolve in the Pre-Implementation Gate.
+- Whether the disambiguated `native_id` breaks parent/child linkage (parent_thread_id/forked_from_id reference the bare id) — the resolver may need to match the bare id prefix.
+
+### Acceptance Criteria
+
+1. Two codex rollout files with the same `session_meta.payload.id` produce TWO distinct canonical sessions, each keyed on a basename-disambiguated native id, with a `LogEntry` warning per the spec. **Verification**: a golden/integration test with two same-id fixtures asserts two sessions.
+2. Parent/child + fork linkage still resolves when a parent/child id collides (the resolver matches the bare id). **Verification**: a linkage test across the disambiguated ids.
+3. The single-file common case (unique ids — 100% of observed data) is byte-for-byte unchanged. **Verification**: existing codex goldens unchanged.
+
+## Analysis
+
+Sources checked: `adapter-codex.md` edge #14 (:504), `internal/adapters/codex/mapper_turn.go` (NativeID assignment), `internal/ingest/writer.go` (sessions upsert + resolver), SOW-0004 round-6 review (codex P3).
+
+Risks:
+
+- **R1 — cross-file state.** Disambiguation needs a view the per-file adapter lacks; the ingester is the natural place but it must not regress the common single-file path. Mitigation: gate the disambiguation strictly on a detected `(source_id, native_id)` collision from a DIFFERENT file.
+- **R2 — linkage.** Disambiguated native ids must not break parent_thread_id/forked_from_id resolution. Mitigation: the resolver matches the bare id.
+
+## Pre-Implementation Gate
+
+(To be filled by the assistant picking this SOW up. Required before moving to `current/`.)
+
+## Implementation
+
+(Empty placeholder.)
+
+## Validation
+
+(Empty placeholder.)
+
+## Reviews
+
+(Empty placeholder.)
+
+## Outcome
+
+Pending.
+
+## Lessons / Follow-Ups
+
+Pending.