Repository navigation
RFC: Strict review for a strong generator — deep passes, per-comment verification, self-proving evidence #72
Description
RFC — Strict review for a strong generator: deep passes, per-comment verification, self-proving evidence
- Status: implemented behind flags (all default ON); measured on the 20-PR
campaign splits (val + one-shot frozen test) - Owner: review pipeline (
engine/steps/review/,review/planner.py) - Evidence:
doc/EVAL-goal-strict-vs-opus5.md,
eval/dataset/judgments/goal_v13_val/,eval/dataset/judgments/goal_v13_test/
Motivation
The Strict review pipeline was engineered around a weak generator: four
narrow lenses forcing enumeration inside templates, tight tool budgets, a
tool-less reducer, and a 5-comment severity cap. With the official
deepseek-v4-pro (a large capability jump over the [1m] preview), that
scaffolding became the bottleneck — measured symptoms across judged sweeps:
- Recall losses were mechanical, not analytical: the budget cap deleted
reducer-kept maintainer concerns; depth planning misrouted small
high-blast-radius diffs to the no-ensemble light tier; a 170k-char diff
lost 30% of its hunks to the 120k evidence cap; whole question classes
(blast radius, dependency floors, test integrity, lifecycle-on-abort) had
no owning lens. - Precision was stuck at 0.54–0.56 across five different configurations
while recall swung 0.55–0.86 — because the blind judge is TOOL-LESS: it
sees only the diff and the human threads, so any repo-side finding
(consumers elsewhere, CI lane rules, version floors) scores as
speculation unless the comment itself carries checkable proof.
Design
Three mechanisms, composed with the existing ensemble/reducer machinery:
1. Hybrid pass set (review_deep_engine, default on)
Full depth runs two deep passes — investigator (central-change-first
free investigation; verify-before-assert; budget discipline: reserve the
final rounds for filing) and adversary (independent hunt over the
systematically-missed classes) — plus the two highest-value breadth
lenses (behavior, verification). Standard depth runs
investigator+behavior. Light is unchanged. Deep passes get
review_deep_max_iters (32) tool rounds; measured alone they ground
claims well (precision .82 on the hardest train items) but under-cover
(val recall .55); measured with the breadth lenses they hold recall .71–.86.
2. Per-comment agentic verification (review_verify_comments, default on)
Every merged draft comment gets a small tool-loop (4 rounds, concurrency 6,
shared diff cache prefix) that must re-derive the claim on the PR-time
tree: refuted drops, unverifiable demotes one severity step,
confirmed may tighten wording/position — and a verification failure
keeps the comment (the pass can only raise precision, never silently
delete recall). Skip/failure reasons are traced
(review_comments_verified / review_coverage_skipped).
3. Self-proving evidence (drafting + verify + render)
The single biggest measured lever (4—11 → 8—7 on the val gate): every
comment's evidence field must QUOTE the decisive code line(s) verbatim
with file:line so a reader holding only the review and the diff can check
the claim. Enforced in the drafting contract, re-emitted by the verifier
from the code it actually read, rendered as-is.
Supporting changes shipped with this RFC: value_flips diff signal (a
default flip can never rule light), verification lens in the planner
fallback set, evidence caps 120k→260k / reducer 60k→280k, commit timeline
- deterministic changed-symbol consumer sweep (
git grep -nw) + per-file
hunk-location paging targets in the evidence pack, cross-lens
corroboration tags with protected-class reducer rules, comment budget
8 + rich-only overflow, coverage-promotion pass (mines the run's own
findings/blockers for uncovered protected-class concerns), knowledge page
repos/vllm-omni/review/guides/strict-review-checklist.mdinjected via
the new adapter manifest keyknowledge.review_checklist, and the
vllm-omni module map extended to 17 modules with 7 risk tiers.
Measured results (blind gpt-5.6 judge, 3 replicates, vs recorded CC+Opus 5)
| gate | config | head-to-head (arm—opus) | arm r/p | opus r/p |
|---|---|---|---|---|
| pre-campaign | preview model, 4 lenses | 1—4 | .42/.51 | .82/.54 |
| v8 | official model, lenses | 3—12 | .86/.56 | .79/.66 |
| v12 | hybrid, narrative evidence | 4—11 | .71/.55 | .81/.66 |
| v13 val | hybrid + self-proving | 8—7 | .857/.555 | .889/.589 |
| v13 frozen test (one shot) | same | 7—8 | .689/.687 | .762/.719 |
Combined fresh verdicts: 15—15 at 1/3 the baseline's cost$0.75–1.2/item vs ~$1.5–3). GOLD latent-gap items: pr4810 2/3, pr4834
(
swept 3/3 clear. Rubric-mean gaps (.03–.07) are at the magnitude of
measured judge drift on identical baseline reviews (±.05–.10).
Alternatives considered
- MoA re-test with stronger members — excluded by owner direction.
- Deep passes only (no breadth lenses): precision transferred but val
recall collapsed to .55; rejected (gate v11, 2—13). - Budget/reducer tuning alone: five configurations moved precision by
≤.02; the deficit was structural (tool-less judge), not calibrational.
Rollout / compatibility
All three mechanisms sit behind settings with kill switches
(review_deep_engine, review_verify_comments; legacy lens set retained
verbatim and used when the deep engine is off). Offline tests cover pass
selection, verify verdict handling (drop/demote/tighten/fail-open),
budget/overflow, corroboration ordering, promotion, and the planner
signal; the full suite and both knowledge validators are green. Cost is
~2–3× the old lens pipeline per item and remains ~3× cheaper than the
baseline. ARM_JOBS in run_copilot_arm.py enables item-level parallel
sweeps (endpoint allows 500 concurrent requests on v4-pro).
Open questions
- Both tuning splits are spent (8 val submissions, 1 frozen-test shot).
Further iteration needs wave-2 items (gt/pr5957*,gt/pr5976*,
build_wave2.py) judged under the same protocol. - The judge protocol's tool-less-ness now shapes the reviews (verbose
quoted evidence). If the product's real audience is maintainers rather
than a repo-less judge, the evidence verbosity should become a render
option rather than a fixed behavior. pr4762/pr4954-class losses (broad multi-concern PRs) remain: the
baseline still out-reads the arm per finding. The next structural lever
is a coverage-driven second investigation round seeded by the reducer's
uncovered-GT-class checklist rather than a fixed pass count.
Implementation PR: #71
🤖 Generated with Claude Code
Activity
tzhouam commentedon Aug 13, 2026
Wave-2 holdout results (RFC open question 1 — resolved)
Ran the 10-PR wave-2 pure holdout (
build_wave2.pydraw: seeded, human-only GT, no adaptation, one shot) with v13 exactly as merged here, against a freshly generated pinned Opus 5 reference (run_baseline_pinned.py, identical frozen snapshots + PR-time trees for both arms — snapshot builder extended to carry the commit timeline so the information budget stays identical).ARM 14 — Opus 5 16 (30 blind verdicts, gpt-5.6, 3 replicates) — near-parity on fully fresh data, and the first full split where the arm's precision exceeds the baseline: 0.748 vs 0.672. Recall 0.492 vs 0.651 (wave 2's GT is much richer than wave 1's — median 6 human inline comments vs 2 — and the comment budget caps coverage there). Actionability 0.920 vs 0.936. Four PRs swept 3/3 each way (arm: 5703, 5715, 5840, 5957 incl. the IndexTTS-2.5 model add; baseline: 5550, 5610, 5884, 5976 incl. the vllm-0.27 rebase), two split.
Cumulative fresh-data verdicts for v13 (val + frozen test + wave-2): 29 — 31 at ~1/3 the baseline's cost.
Disclosures: two baseline items (5884, 5957) carry audit flags for the Opus agent reading its own /tmp task-transcript files — harness self-reads, no repo or discussion leakage; kept and disclosed. Judgments:
eval/dataset/judgments/goal_v13_wave2/.Next lever per the RFC's open question 3, now sharpened by wave-2: the remaining gap is pure coverage on GT-rich PRs — a coverage-driven second investigation round (seeded by uncovered checklist classes) rather than a fixed pass count.
🤖 Generated with Claude Code
tzhouam commentedon Aug 14, 2026
Generator ablation on wave-2 (same v13 pipeline, items, snapshots, judge): MiMo-v2.5 scores 2—27 vs Opus 5 (r .481 / p .595 / act .731 at $0.43/item) where DeepSeek-v4-pro scored 14—16 (r .492 / p .748 / act .920 at $0.79/item). Recall is pipeline-carried (near-identical); precision and actionability are generator-carried. Judgments:
eval/dataset/judgments/goal_mimo_wave2/(29/30 verdicts; one flaked after 3 retries).🤖 Generated with Claude Code
tzhouam commentedon Aug 14, 2026
Judge-protocol update (Sonnet 5) + fresh generation replicate — wave-2, DS-v4-pro v13: r1 reviews re-scored under
claude-sonnet-5: 10—20 (arm r .323 / p .833 vs opus r .437 / p .793). Fresh generation replicate r2, Sonnet-judged: 3—27 (r .278 / p .804 vs r .509 / p .816). Consolidated read: (1) under strict Sonnet recall-crediting the arm clearly loses wave-2 — the gpt-5.6 near-parity (14—16) was judge-dependent; (2) the precision gains are judge-robust (arm ≈/above baseline under both judges); (3) generation replicate variance (10—20 vs 3—27, same config+judge) exceeds most config effects — single-replicate generation claims should not be quoted. MiMo ablation conclusion unchanged under both judges (2—27 gpt / 2—25 sonnet). Earlier gpt-judged val/test gates (8—7, 7—8) require Sonnet re-scoring before being quoted under the new protocol. Judgments:goal_v13_wave2_sonnet,goal_ds_wave2_r2_sonnet,goal_mimo_wave2_sonnet. Remaining lever: coverage-driven second investigation round (RFC open q3), needs a wave-3 draw to validate.🤖 Generated with Claude Code
tzhouam commentedon Aug 14, 2026
Composer 2.5 arm on wave-2 (headless cursor-agent harness — its own agentic loop, not our pipeline — pinned trees, frozen snapshots, detective audit; judge claude-sonnet-5 ×3): 4—26 vs Opus 5 (r .381 / p .751 / act .773, mean 177s/item, per-run cost not exposed by cursor billing). Notable: swept pr5976 (the vllm-0.27 rebase) 3/3 — the item every other challenger arm lost. Audit: 9/10 clean; pr5884 carries one benign harness self-read (same class as the baseline's own flags, kept+disclosed). Sonnet-judged challenger table on identical items: DS-v4-pro (our pipeline) 10—20 / 3—27 across two generation replicates, Composer-2.5 (cursor harness) 4—26, MiMo-2.5 (our pipeline) 2—25. Under strict Sonnet recall-crediting, Opus 5 beats all challengers; Composer posts the best challenger recall (.381), DS the best precision (.80–.83). Judgments:
goal_composer_wave2_sonnet; runner:run_cursor_arm.py.🤖 Generated with Claude Code
tzhouam commentedon Aug 15, 2026
v14/v15 recall campaign — teacher distillation, waves 3–4 (2026-08-15)
Continuation of the goal campaign under the standing claude-sonnet-5 judge. Full detail:
doc/RFC-review-recall-v14.md, results table top section ineval/dataset/results/model_comparison.md.Method. Wave-2 (spent) forensics + a Claude Fable 5 teacher arm on the train split (
baselines/teacher_fable5/, 10 items, ~$43 — it caught the pr4870 GOLD latent gap exactly, via a shared-contract producer census). Distilled into v14/v15: nine new investigation duties (claim ledger, sibling contrast, PR-numbers falsification via a sandboxedcalc, semantic-merge audit, producer census, test/gate epistemics + CI economy, differential parity, dead-knob recompute), a docs claims-audit/IA pass, five read-only archaeology tools (diff_stat/file_at_base/show_commit/search_history/calc, also bridged to harness backends), a coverage-driven second round seeded per-hunk (RFC deep-engine open q3), a ranked verification ledger with[resolved]/[claim-*]channels, and hard machinery fixes — including two long-standing latent bugs: pass finals dying at the 16k completion ceiling with an 8k head-only repair, and the gray-zone LLM planner silently failing on every gray item since the official reasoning model landed (its thinking consumed the 400-token cap before any JSON).Results (arm—baseline wins, rubric means arm/baseline, 3 reps):
Where it stands. On a clean fresh human-GT holdout the v15 arm is above the baseline on precision for the first time, recall ratio .78 (v13/v14 fresh splits sat at .55–.75), at $0.97/item vs $3.09 (3.2×). It now wins the GT-richest items (pr5864 GT=17 inline, pr5958 GT=20, pr5608 swept) — exactly where earlier arms collapsed. The residual recall gap (−.07, ~1.5× measured judge drift) concentrates in mid-size items; wave-3 forensics characterized the mechanism (validation-bias inversion: the ledgers record the GT facts approvingly instead of raising them; each fresh split partially renews the unowned GT classes). The Composer cursor-backend row reproduces the wave-2 pattern on the v15 pipeline: recall parity with DS, precision behind.
Blocked / next. A second DS replicate was invalidated mid-sweep by a 402 Insufficient Balance on the DeepSeek account (stubs quarantined at
judgments/INVALID_apierror_goal_v15r2_holdout4_sonnet) — replication and any further DS iteration are blocked on a recharge. Wave-4 remains valid for one more confirmatory submission; ~29 undrawn pool items remain for a wave-5. Baseline disclosure: wave-4 pr5720/pr5864 carry the benign self-read audit flag (CC's own tool-result spool), recorded in their cost.json.(automated update from the goal session)
tzhouam commentedon Aug 15, 2026
v16 + wave-4 saturation (2026-08-15, cont.)
Post-recharge continuation. Three new measurements on the wave-4 gate (all committed, table in
eval/dataset/results/model_comparison.md):Highlights: v15's precision-above-baseline replicated (both independent generations). The role-split showed no gain — Composer-as-investigator underdelivers pro. v16's Fable seats delivered the campaign's best fresh-split recall (ratio .81) and reproduced the pr4870 GOLD-gap catch in-pipeline on train (the exact #4910 dual-axis mechanism, via the routed adversary), and its train probe had BOTH means above baseline (9—11, r .627/.625, p .805/.772) — but on the holdout it landed at precision parity with fewer wins.
The binding constraint is now measurement, not modeling: opus's recall on IDENTICAL baseline mds read .335 / .338 / .416 across the three wave-4 judgment sets (±.08 drift) — larger than the arm deltas under study. Pooled wave-4 DS-core (90 verdicts): 27—62—1, precision .805/.790 (above), recall .293/.363 (ratio .81). Wave-4 has hosted 3 DS-core configs + the Composer row and is saturated as a gate.
Remaining honest levers: a full-Fable generator run (~$10–20/item) and/or a fresh wave-5 draw with a pre-registered pooled protocol sized for the measured judge noise. Awaiting owner direction.
(automated update from the goal session)
tzhouam commentedon Aug 15, 2026
Correction: the wave-4 "precision above baseline" claim is withdrawn
Two earlier comments in this thread reported v15 as precision above the CC+Opus 5 baseline on wave-4 (.816/.787, then replicated at .800/.778). That was computed from each side's raw rubric mean, which is the wrong statistic for this design, and the claim does not survive the right one. Withdrawing it.
Why the raw means mislead. Three wave-4 judgment sets scored the same baseline review texts and returned baseline recall means of .335 / .338 / .416 — ±.08 of pure judge drift, larger than the arm-vs-baseline differences under study. Any comparison that reads the arm from one set and the baseline from another inherits that drift.
The correct statistic pairs the two candidates inside each verdict (one judge call scores both, so its leniency cancels) and clusters the three replicates by item (they share the item and the two fixed texts). Implemented reproducibly as
eval/dataset/paired_analysis.py:Every interval spans zero. The honest wave-4 conclusion is parity within measurement precision at ~1/3 the baseline's cost — the arm is not demonstrably better on precision, and the recall deficit the campaign spent its second half chasing is also not demonstrable at n=10 items. The point estimates still favor the baseline on recall (−.07) and the arm slightly on precision (+.015), and the aggregate recall gap is carried by 3 of 10 items (pr5723 −.20, pr5756 −.27, pr5978 −.27) with the other 7 inside ±.06.
What survives unchanged: the configuration effects replicate in sign across independent generations (v15 precision-leaning both times; v16 recall-leaning, and the only config that reproduced the pr4870 GOLD-gap catch inside the pipeline), the cost ratio, and Composer's measurably worse precision (−.045) as a breadth-not-closer proposer.
Next: power, not another 10-item verdict. At the observed item-level sd (.127), resolving a .07 difference at 95%/80% needs ~32 items; wave 5 (
build_wave5.py, drawn and frozen) takes the fresh pool to 20 for a pre-registered pooled v16 measurement.(automated update from the goal session)
tzhouam commentedon Aug 16, 2026
Wave-5: the pre-registered answer — recall deficit is real (−.049), precision at parity
The measurement question is settled.
goal-eval/PREREG-wave5.mdfixed the arm, items, judge, analysis and decision rule before any wave-5 number existed; wave 5 was drawn by the same seeded stratified procedure as waves 2–4 and pooled with wave 4 for the primary analysis.Primary result — v16, 20 items, 60 verdicts, paired within verdict and clustered by item:
Wave-5 alone (descriptive): 6—24, Δrecall −.019 [−.068, +.031], Δprecision +.005.
Read it honestly. The recall result is marginal — the upper bound is −.001 — so this is a real but small deficit, roughly five rubric points, not a large one. It is also concentrated rather than diffuse: across the 20 pooled items, 6 favour the arm, 4 sit inside ±.02, and a single item (pr5978, −.37) supplies about a third of the pooled gap. Head-to-head win share (.21) looks worse than the rubric gap because the judge weights recall heavily.
Per the pre-registration, no third wave was run. That instruction was written in precisely to stop a favourable-draw hunt, and the temptation was real: a marginal −.049 is exactly the result that invites "one more wave."
Campaign close-out
The goal as stated — both means strictly above CC+Opus 5 — is not achieved, and with corrected statistics it was never achieved at any point. What is supported: the arm matches the baseline on precision and trails on recall by ~.05, at roughly a third of the cost ($0.86–0.97/item vs $3.09). Two configurations are documented in
.env.templatewith their measured tradeoffs: v15 (all-DS, precision-leaning) and v16 (Fable in the adversary and second-round seats, recall-leaning, and the only configuration that reproduced the pr4870 GOLD latent-gap catch inside the pipeline).Full report:
doc/EVAL-v14-v16-recall-campaign.md. Successors should note the power arithmetic before spending a split: at the observed item-level sd (.103), a .05 effect needs ~38 items — wave-4's 10-item gates could not have resolved what three configurations were submitted against.(automated update from the goal session)
Paired item-clustered statistics; wave-5 draw + pre-registration
tzhouam commentedon Aug 26, 2026
Closing as a complete record. The campaign ended with the pre-registered wave-5 measurement (last comment above): Δrecall −.049 [−.097, −.001] — a real but small deficit — and Δprecision at parity, at ~1/3 of the baseline's cost. The two shipped configurations (v15 precision-leaning, v16 recall-leaning) are documented in
.env.templatewith their measured tradeoffs, and the surviving structural lever (coverage-driven second investigation round) is recorded in the close-out for any successor campaign.