Repository navigation
Add trace schema v1, replay and dataset export - #211
Conversation
Every model call, decision and later outcome is now kept durably, so the
expensive models' work can be replayed and turned into datasets for cheaper
models plus workflows. The copilot and the review bot share one schema.
- trace_store (trace/1): append-only JSONL records (the source of truth),
content-addressed gzip blobs for prompts and replies (records hold only
sha256 refs; reads re-verify the hash), and a SQLite index by kind, run,
repo, change set, PR, rule ID, role, model and outcome. The index can be
rebuilt from the JSONL. trace_context binds run/playbook/repo/change-set
context through contextvars. Every string of a record (keys, model and
usage included) and every blob is redacted before it is hashed or written: known token shapes, private-key blocks, and the value of every
environment variable whose name marks it as a secret. It uses only the
standard library and is exported through sdk.v1 (TraceStore, TRACE_SCHEMA,
trace_context, redact) without loading private modules.
- knowledge service:
- ModelGateway records every call with the pinned provider/model/effort
and the call's final verdict: transport failures and truncated, empty,
unparseable or schema-failing replies are recorded as failures.
- gate_and_stage allocates the change set ID first, judges under its
context, then writes a decision record.
- merge.advance writes outcomes (merged, closed_unmerged, head_changed).
- retirements write rule_retired.
- intake, sweep and calibration are bound to their playbooks. Generator
calls precede their change set, so each drafting has a unique draft key
(event, or release + page) and each call its attempt number; the
decision lists only the ACCEPTED attempt, so rejected replies and earlier
tries never inherit its labels.
- Tracing never breaks the service.
- replay: re-asks a recorded call with a substitute model and compares the
decision-bearing fields. It touches no GitHub, outbox or ledger; the
substitute call and the comparison are traced too.
- export: judge or generator rows of (input, expensive output, gate decision,
outcomes, later rule outcomes), generator rows joined through draft keys.
Calibration and replay calls, failed calls, and any prompt that cites a
calibration case's evidence reference anywhere or its title are dropped.
- CLI: kb traces, kb replay --record ID --model P:M[:E], kb export --out F.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 074a27ada7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| with trace_context(playbook="kb-calibrate", run_id=f"calibrate-{Path(directory).name}"): | ||
| return _run_calibration(directory, gateway=gateway, judge=judge) |
There was a problem hiding this comment.
When calibration is run through infermatrix-copilot kb calibrate, cli.py passes a plain ModelGateway(Settings()) with no recorder, so this context manager does not produce any model_call records. Consequently, all calibration model activity is absent from trace/1 despite the new every-call audit contract; construct the gateway with the state directory's TraceStore and trace_recorder before invoking run_calibration.
Useful? React with 👍 / 👎.
| identity = {"role": role.name, "requested": role.label(), "provider": role.provider, | ||
| "model": role.model, "effort": role.effort} |
There was a problem hiding this comment.
When a configured provider CLI is missing or unauthenticated, _transport() raises ModelUnavailable before the new identity and exception-recording path is reached. These common unavailable-model failures therefore leave no model_call record at all, contradicting the promised inclusion of failed calls and preventing operators from replaying or auditing them; move transport acquisition inside the recorded failure path.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
PR 8 of the knowledge-service redesign: trace schema v1 (shared by the copilot and the review bot), plus replay and dataset export.
trace_store(trace/1):sdk.v1.kb replay: re-asks a recorded call with a substitute model and compares the decision fields. It makes no GitHub, outbox or ledger writes.kb export: dataset rows of (input, output, decision, outcomes). Calibration and replay calls, failed calls, and prompts citing calibration evidence are dropped.Commit hook approved; full suite and SPEC freshness pass locally.
🤖 Generated with Claude Code