Repository navigation
Add knowledge intake, the quality gate and judge calibration - #207
Conversation
The knowledge service can now turn upstream changes into gated knowledge change sets. The merge flow, activation, sweep and scheduler build on this. - models: generator and judge pinned as provider:model[:effort], by default claude-code:claude-opus-5-5 and codex:gpt-6-sol:medium. The judge must come from a different model family. An unavailable, timed-out or unparseable model raises ModelUnavailable and the work stays queued; there is no weaker fallback. Every call is recorded with inputs, outputs and usage. - providers: complete() takes an `effort` pin. codex passes it as model_reasoning_effort and validates it; the others accept and ignore it. - sources: read-only reads of the knowledge repository clone (knowledge and external texts via git cat-file), GitHub (merged PRs, bounded PR evidence, releases, tags) and the host's Copilot run-lesson inbox. - intake: the generator drafts typed operations (add, edit_same_meaning, replace, retire). It sees the owner pages and existing rules and gets the evidence fenced as untrusted data. apply_operations must accept the result, with up to two repair rounds fed the exact error. - gate: L1, then a per-block L2 judge answering only the dimensions each block kind needs, then a consistency check per owner directory. Any "no" or conflict fails; any "unsure", unavailable model, protected rule, external reference or circuit breaker (counted per repository) goes to people. L1 failure makes no model call. - verdict: the signed kb-gate verdict schema (identity binding, a patch manifest compared by pre/post blob IDs, block and consistency judgements, upstream facts, auto vs human-approved), vendorable. - intake runs under the single-writer ledger lease. A keeper thread renews it during slow model calls, and every ledger write goes through one transaction fenced on lease ownership, so a worker that lost its lease stages nothing. Discovery uses a (merge time, PR number) cursor over all result pages. Conflicting drafts stay pending; exhausted repairs are `rejected`, not "no rules". - runtime and the kb-intake playbook (candidate, planner-invisible): collect, intake, publish. Shadow records what it would open; auto_merge writes a signed open_pr outbox item. `kb run` runs it through the standard executor. - calibration: `kb calibrate` scores the judge on the repository's set; every bad case must be caught and at most 20% of good ones rejected. vllm-omni's set ships in its adapter: the 8 human-approved rules of #202 as known-good, plus 10 known-bad (mutations and the historical MMH3-4e contradiction), each with real upstream evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 763c8da890
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| an edit that changes meaning, an invented API) and the historical bot-written | ||
| MMH3-4e that contradicted MMH3-4a (folded by a human in 03b0b62a2). | ||
|
|
||
| Evidence is the real upstream PR (title, body, bounded diff excerpt). |
There was a problem hiding this comment.
The calibration cases check full upstream PR bodies and diff excerpts into the adapter tree, even though the repository contract requires raw PR material to remain temporary and outside the repository. Replace these replay fixtures with compliant evaluation artifacts or externally supplied evidence rather than shipping the raw PR payloads in every installation.
AGENTS.md reference: AGENTS.md:L12-L17
Useful? React with 👍 / 👎.
| validate(data) | ||
| except Exception as exc: # any malformed shape is a controlled refusal | ||
| raise ModelUnavailable(f"{role.label()} reply failed its schema: {exc!r}") from exc | ||
| return ModelReply(role, data, text, record["served_model"], record["usage"], seconds) |
There was a problem hiding this comment.
When a transport returns a successful reply whose model differs from role.model—for example because the backend silently substituted a fallback—the gateway records the mismatch but still accepts the response. This breaks the no-fallback pin and is especially unsafe for the judge because calibration is subsequently stored under the requested label, allowing the substituted model to authorize auto-merge; fail closed on a nonempty incompatible served-model value.
Useful? React with 👍 / 👎.
| return "calibration_required" | ||
| data = rt.load_changeset_files(changeset_id) | ||
| detail = changeset["detail"] | ||
| rt.outbox.issue(lifecycle.repo, "open_pr", { |
There was a problem hiding this comment.
If the process exits after issue() writes the uniquely identified signed open_pr item but before the later changeset status update, the changeset remains gated; retrying or resuming publication therefore emits another independently executable item for the same changeset. Concurrent calls have the same race, so the publisher can receive duplicate GitHub actions; persist a deterministic issuance marker/status atomically with the outbox operation or otherwise deduplicate by changeset.
Useful? React with 👍 / 👎.
| from .runtime import record_calibration | ||
|
|
||
| case_dir = lifecycle.adapter_dir / lifecycle.calibration_set | ||
| report = run_calibration(case_dir, gateway=ModelGateway(Settings()), judge=judge) |
There was a problem hiding this comment.
When operators run kb calibrate, this gateway is constructed without the recorder used by KbRuntime, so none of the calibration prompts, replies, served models, or usage are written to $KB_STATE_DIR/traces/model_calls.jsonl. This leaves the model calls that authorize auto-merge unaudited despite the documented every-call trace contract; construct the gateway with the same state-directory recorder.
Useful? React with 👍 / 👎.
Fifth PR (5a) of the knowledge-base redesign: intake, the quality gate and judge calibration. The merge flow, snapshot activation, release sweep and scheduler come next (5b).
What changes
claude-code:claude-opus-5-5and judgecodex:gpt-6-sol:medium(configurable viaKB_GENERATOR/KB_JUDGE).$KB_STATE_DIR/traces/model_calls.jsonl.complete()takes aneffortpin. Codex passes it asmodel_reasoning_effortand validates it.apply_operationsmust accept the result, with up to two repair rounds fed the exact error.rejected, not "no rules".kb-gate-verdict/1), vendorable.kb-intakeplaybook (candidate, not visible to the planner) runs collect → intake → publish.auto_mergemode it writes a signedopen_proutbox item, but only with a stored, passing calibration for the exact judge and case set.kb calibraterequires every bad case to be caught and at most 20% of good ones rejected. The vllm-omni set ships in its adapter:Test Plan
test/test_kb_intake_gate.py(35 tests):kb run --state-dirwithout mutating the environment;check_spec_freshness --strict,check_doc_links.Test Result
kb calibrateagainst the real judge. That needs the bot host's Codex login and is an owner step before any repository leaves shadow mode.🤖 Generated with Claude Code