ci(codex-review): stop requiring the VERDICT trailer — the model omits it - #134
Conversation
…s it #133 made the trailer mandatory. It failed the first real PR it ran on. topcoder1/domain-rank#82, run 30330151877 at 2026-07-28T04:57Z. The prompt asking for the trailer is in that run's log. gpt-5.6-sol at reasoning=low answered: "No regressions found in the requested coverage, state-mutation, or contract-drift axes." with no trailer. The gate logged `state=no_verdict; enforcing=true` and reddened a clean PR. Compliance with a free-text output contract cannot be assumed, and that is now measured rather than argued. The failure signals are the ones that have actually proven reliable: - a `regression:` line. Three for three on real findings (#74, #79, #82). - an explicit `VERDICT: REGRESSION` trailer, when the model complies. - no output at all: empty, whitespace-only, missing file, or the workflow's own no-parseable-verdict sentinel. A review that did not happen must never read as "nothing found". Everything else passes, with a `::warning::` when the trailer is missing so report-only callers make the fleet-wide compliance rate visible — which is what would justify requiring it later. The prompt still asks for the trailer, and it is still read from the final line when present. It is a bonus signal now, not a gate. ACCEPTED AND ASSERTED: a finding phrased in neither form is missed. Every rule that caught that case also failed clean PRs, because the two are indistinguishable without reading English — "The new fallback branch has no test." and "Looks good to me." are both trailer-less prose. Blocking correct work is what gets a gate switched off, and then nothing is reviewed at all; a miss still reaches a human through the review comment. The selftest names these cases as known misses rather than quietly dropping them. Codex rounds: 2 Refs: topcoder1/domain-rank#80 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Risk class: This PR touches one of the blocked path categories from Auto-merge is refused by (This is a policy notice, not a code-quality failure. The classify job itself does not fail — required CI checks remain authoritative for "is the code green.") |
|
Coverage Floor — mode:
|
|
No issues found. The trailer→bonus-signal change is sound: sentinel and empty/missing-file cases still route to no_verdict (fail-closed), findings still outrank a CLEAN trailer, and the new clean-fallthrough behavior is the deliberate, well-tested point of the PR. Default report-only callers keep exit 0. |
1 participant
#133 made the
VERDICT:trailer mandatory. It failed the first real PR it ran on.The evidence
topcoder1/domain-rank#82, run
30330151877, 2026-07-28T04:57Z. The prompt asking for the trailer is verifiably in that run's log:gpt-5.6-solatreasoning=lowanswered:No trailer. The gate logged
Verdict: state=no_verdict; enforcing=trueand reddened a clean PR.Compliance with a free-text output contract cannot be assumed. That is now measured rather than argued — and it is the second time this gate has been corrected by real data instead of reasoning.
What fails now
Only signals that have actually proven reliable:
regression:line — three for three on real findings (ci(risk-paths): exclude .env.example/template/sample from blocked glob #74, Required-checks audit: 2026-W22 #79, and CI runs)VERDICT: REGRESSIONtrailer, when the model does complyEverything else passes, with a
::warning::when the trailer is absent, so report-only callers make the fleet-wide compliance rate visible. That warning is what would justify requiring the trailer later, if compliance turns out to be high on other models.The prompt still asks for the trailer and it is still read from the final line when present. It is a bonus signal now, not a gate.
The accepted risk, stated plainly
A finding phrased in neither form is missed. I am not going to pretend otherwise, and the selftest asserts it by name (
off-format finding is missed (accepted risk)) rather than quietly dropping the case.Every rule that caught that case also failed clean PRs, because the two are indistinguishable without reading English — "The new fallback branch has no test." and "Looks good to me." are both trailer-less prose. Blocking correct work is what gets a gate switched off, and then nothing is reviewed at all. A miss still reaches a human through the review comment.
This is the honest trade, and #82 is the reason it is made in this direction.
Verification
Codex rounds: 2
Auto-merge rationale: manual merge — PRs to
ci-workflowsare always manual merge, and this touches.github/workflows/**on a reusable consumed by 27 repos via@main. Behavior is unchanged for every caller that does not setfail_on_regression: true; only domain-rank does, on an open PR.🤖 Generated with Claude Code