Repository navigation
Give the reviewer a line budget for non-test code - #149
Merged
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
A PR body that says tests pass is not evidence that the guard cases CI runs for the changed feature were executed. The lane selects by marker, so running a test file directly can prove nothing about what the lane collects. REV-3a makes that comparison a gate: find the guard steps from the pipeline definition at the frozen head, compare their commands with what the PR reports running by selector rather than by literal text, and treat a missing, unrun or mismatched case as a blocking finding. The rule reaches every review channel it applies to. Direct consumers get it through the first-review checklist; the Strict reviewer gets the repo-shaped form through the injected checklist page. That injection had no room left. The page was 6,991 characters against a 7,000-character cap, so anything appended would have been cut off mid-sentence and the gate would have looked present while never reaching the model. The cap moves to 8,000, and the three places that documented it as 4k or 7k now agree with the code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCk5xJGRyydouK7tuhs8kk
Review attention is finite, so REV-3b puts a number on it: non-test added and modified lines stay under a thousand, and past that the review asks for a split plan or a stated exemption on the largest contributing file. The rule says where the number comes from, because that is the part easy to get wrong. The host supplies it; the reviewer reports it. A model counting the diff itself would be counting a patch that is truncated for exactly the large files the budget exists for. Tests, docs, lock files and pure renames stay outside the budget, and deletions never consume it, so the rule cannot be satisfied by writing fewer tests or punish a cleanup. The review-readiness prompt said in as many words that a large diff is never sufficient on its own, which would have argued against the new rule every time it fired. It now records volume above the budget as a concerns reason in its own right, while the code path that already requires two grounded reasons keeps size alone from reaching needs_rework. A test pins the two halves together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCk5xJGRyydouK7tuhs8kk
tzhouam
force-pushed
the
agent/review-rule-diff-budget
branch
from
September 10, 2026 09:32
b72817e to
79e765e
Compare
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Owner rule request: a PR's code diff excluding tests should not exceed ~1k lines.
REV-3binknowledge/general/review/rules.mdstates it, and says the part that is easy to get wrong -- where the number comes from. The host supplies it; the reviewer reports it. A model counting the diff itself would be counting a patch that is truncated for exactly the large files the budget exists for. The enforced counter for the bot's Direct lane is omni-reviewbot#57.Outside the budget: tests, documentation, lock files and pure renames. Deletions never count. So the rule cannot be satisfied by writing fewer tests, and a cleanup is not punished for its size.
Over budget, the review asks for a split plan or a stated exemption (vendored sync, generated code) on the largest contributing file. A PR body that already gives one is recorded instead. Size stays non-blocking and never raises another finding's severity.
Coverage matches #148:
DIRECT_REVIEW_CHECKLISTfor Direct consumers,strict-review-checklist.mdfor Strict (now 7,714 chars against the 8,000 cap raised in #148 -- 286 characters of headroom left, so the next addition to that page needs a trim or another bump).The contradiction this had to resolve
agent.assess_pr_quality's prompt said in as many words that a large diff is never sufficient by itself -- it would have argued against this rule every time it fired. It now records volume above the budget as aconcernsreason in its own right, while the existing code path that requires two grounded reasons keeps size alone from reachingneeds_rework. A test pins both halves so they cannot drift apart again.That assessor never sees
rules.md, so its prompt now carries the exclusions and the already-justified exemption itself. Without that, a docs-only PR could collect a size concern for a budget it cannot exceed. (Both points came from the commit review gate; the second caught a real error in my first wording, which said a deletion-heavy PR cannot exceed the budget -- 1,500 added lines are over budget beside any number of removed ones.)SPEC freshness
engine/steps/review.mdsaid size may never be a needs-rework reason on its own -- still true, and still enforced in code -- but it now also records that volume can stand alone as aconcernsreason, and why the exclusions travel in the prompt rather than by reference to the knowledge tree.Test Plan
PYTHONPATH=$PWD/src python -m pytest -q python knowledge/tools/check_knowledge_tree.py python knowledge/tools/check_wiki_lint.pyNew coverage:
test_quality_prompt_keeps_size_a_concerns_reason_onlyasserts the budget, the exclusion list, the deletion distinction, the exemption, and that size alone never reachesneeds_rework.Note:
PYTHONPATH=$PWD/srcis required here -- the shared venv hasinfermatrix_copilotinstalled editable against another checkout and this repo's pytest config sets nopythonpath, so a barepytestsilently tests the other tree.Test Result
Full suite on this branch: 1 failure,
test_phase1_cluster.py::test_drift_guard_unimportable_checked_class_fails_guard, identical toorigin/mainate40258d3band to #148.Knowledge validators:
check_knowledge_tree.py0 errors / 22 pre-existing size reminders;check_wiki_lint.py0 errors, 0 warnings across 387 pages.Stacked on #148 (shared checklist page, shared
DIRECT_REVIEW_CHECKLIST, and it consumes the 8k injection cap raised there). Merge #148 first.Not covered: no live review was run, so how often the ask fires on real PRs is unmeasured. The 1,000-line number lives in the rule text on this side and in a code constant on the reviewbot side; there is no single source of truth binding them yet.
🤖 Generated with Claude Code
https://claude.ai/code/session_01JCk5xJGRyydouK7tuhs8kk