[tzhouam]

Implements RFC #72 — Strict review reworked for the official deepseek-v4-pro: hybrid deep-investigation passes, a per-comment agentic verify pass, and self-proving verbatim evidence. Design, alternatives, and open questions in the RFC issue (#72, also committed as doc/RFC-strict-review-deep-engine.md).

Performance vs the Opus 5 baseline

Every arm is measured against the same opponent — Claude Code + Opus 5 reviewing the same PRs — judged blind by claude-sonnet-5, 3 replicates, paired inside each verdict and clustered by item. Regenerate with python tools/mkchart_vs_opus5.py.

Pipeline versions (Composer 2.5 backend, train n=10):

version	change	Δ recall	Δ precision	outcome
v17cb	Composer 2.5 on the v17 pipeline	−0.052	−0.097	start
v19cb	real dedupe rule + no invented anchors	−0.092	+0.060	kept
v20cb	merge duplicates to their richest statement	−0.005	+0.046	champion
v21cb	claim headlines + evidence quote blocks	−0.093	+0.079	reverted
v22cb	require trigger → path → consequence	−0.082	+0.040	reverted
v23cb	breadth sweep lens over every changed file	−0.093	+0.024	reverted

Other generators on the same pipeline family and judge: DeepSeek v4-pro (v17 clean, train) +0.024 / +0.016; DeepSeek in its own dsh harness (v18, val) +0.038 / +0.023 — the best recall row of the campaign; DeepSeek v4-pro (v13, wave-2) −0.115 / +0.040; grok-4.6 −0.080 / +0.004; Composer 2.5 standalone −0.127 / −0.048; grok-4.5 −0.164 / +0.030; MiMo v2.5 −0.212 / −0.084.

How to read these numbers

• The shaded band is the resolution limit. At n=10 the 95% half-width implied by the observed item-level spread is ±0.099 recall and ±0.115 precision. Most differences on the chart fall inside it and should not be read as real.
• Precision is the durable result; recall is not. v20's precision gain survives leave-one-out on val (every removal stays positive, +0.021 to +0.155). Its recall parity does not: train −0.005 → −0.068 without pr4970, val +0.003 → −0.046 without pr4825. Every other single-item removal moves train by ≤0.03.
• The variance is generation-side. Re-judging one fixed review 3× gives sd 0.075; re-generating from unchanged code gives sd 0.164 — 2.2× larger. More judge replicates cannot shrink it.
• v21/v22/v23 are not distinguishable from v20. Tested directly against the champion rather than via their baseline-relative intervals: −0.088 [−0.258, +0.082], −0.077 [−0.217, +0.063], −0.089 [−0.239, +0.062]. They were reverted for lack of evidence they help, not because they were shown worse.
• Judges do not pool. The chart is claude-sonnet-5 throughout. Earlier gpt-5.6-judged results from this branch (val 8—7, frozen test 7—8) credit recall more leniently and are reported separately in doc/EVAL-goal-strict-vs-opus5.md; do not read them on the same axis.
• train and val are spent. Both have been iterated against, so neither supports a claim about unseen PRs. The 5-item test split is deliberately unspent.

Contents

• Engine: deep investigator + adversary passes alongside the behavior/verification lenses (review_deep_engine); per-comment verification (review_verify_comments); self-proving evidence quotes end-to-end; coverage promotion; corroboration-ranked budget with rich-only overflow.
• Recall mechanics: value_flips planner signal, verification lens in the planner fallback, evidence caps 120k→260k / reducer 60k→280k, commit timeline + changed-symbol consumer sweep + hunk-location paging in the evidence pack.
• Provider registry: api | cursor | claude-code | codex | deepseek behind STRICT_BACKEND, with the MCP tool bridge — this is what let the same pipeline be measured across the generators above.
• Render: dedupe that actually fires (_same_finding: verbatim-or topic∧claim), merge-to-richest instead of drop-later, no invented ?:NN anchors, ledger capped at 6.
• Knowledge: strict-review-checklist page injected via new manifest key knowledge.review_checklist (linter extended); vllm-omni module map 8→17 with platform/diffusion/distributed risk-tiered.
• Evidence: arm review artifacts + judgment sets so every reported number is reproducible.
• Also ships the previously-untracked anchor.py + tests this pipeline imports, and ARM_JOBS parallel eval sweeps.

Disclosures: config.py, manifest.yaml, and run_copilot_arm.py carried small pre-existing in-flight local edits that ship as-is; superseded local arms (v7–v12) are not included; both tuning splits are now spent — further iteration should use the wave-2 snapshots (RFC open question 1). One quarantined arm (INVALID_moa_*) is excluded: it ran with Mixture-of-Agents on, so its lenses were partly served by other vendors.

🤖 Generated with Claude Code

[tzhouam]

Wave-2 holdout measured post-merge-base: ARM 14 — Opus 5 16 on 30 blind verdicts over 10 never-seen PRs; arm precision 0.748 vs 0.672 (above baseline), recall 0.492 vs 0.651. Details and disclosures in #72. Cumulative fresh-data verdicts for v13: 29—31 at ~1/3 cost.

🤖 Generated with Claude Code

[TaffyOfficial]

dsv4pro-0831已经撤回发布，不如预期