Repository navigation
Make test evidence answerable against the CI definition - #148
Merged
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
A PR body that says tests pass is not evidence that the guard cases CI runs for the changed feature were executed. The lane selects by marker, so running a test file directly can prove nothing about what the lane collects. REV-3a makes that comparison a gate: find the guard steps from the pipeline definition at the frozen head, compare their commands with what the PR reports running by selector rather than by literal text, and treat a missing, unrun or mismatched case as a blocking finding. The rule reaches every review channel it applies to. Direct consumers get it through the first-review checklist; the Strict reviewer gets the repo-shaped form through the injected checklist page. That injection had no room left. The page was 6,991 characters against a 7,000-character cap, so anything appended would have been cut off mid-sentence and the gate would have looked present while never reaching the model. The cap moves to 8,000, and the three places that documented it as 4k or 7k now agree with the code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JCk5xJGRyydouK7tuhs8kk
tzhouam
force-pushed
the
agent/review-rule-test-evidence
branch
from
September 10, 2026 09:31
fb5a9c4 to
dc0edd5
Compare
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Owner rule request: when reviewing a PR, check the description's Test Plan / Test Result for (1) whether the CI guard cases covering the changed features/models were actually run, and (2) whether the commands match what CI really executes.
REV-3ainknowledge/general/review/rules.mdstates the gate. The comparison is answerable rather than a matter of taste, because CI declares both halves at the frozen head:commandsare the commands CI runs;source_file_dependenciesdeclare which changed paths that step guards;pyproject.tomlowns the marker semantics those commands select on.Equivalence is by selector -- test targets,
-mexpression, run-level -- never by literal text, sotimeout,--cov*and container prefixes are not mismatches. Running a test file directly is explicitly not evidence that the lane collects it: the marker deselects it there.Coverage: Direct consumers (including
/imreview) get it viaDIRECT_REVIEW_CHECKLIST; the Strict reviewer gets the repo-shaped form via the injectedstrict-review-checklist.md. The enforced version for the bot's Direct lane is omni-reviewbot#56.The injection budget had no room
strict-review-checklist.mdwas 6,991 characters against a 7,000-character cap (engine/steps/review/steps.py). Anything appended would have been silently cut mid-sentence -- the gate would have looked present in the page while never reaching the model. The cap moves to 8,000 and the page now sits at 7,484.Three places documented the cap as 4k or 7k while the code said 7,000; they now all say 8k (
doc/knowledge/writing.md,doc/features/review-recall.md, thecheck_wiki_lint.pycomment, the test docstring).SPEC freshness
Touching
direct_routing.pyandengine/steps/review/made their SPEC pages stale and failed the--strictgate on the first push. Both are re-verified against the change, andengine/steps/review.mdnow states the injection cap as a contract, since that is the thing this PR moves.Test Plan
PYTHONPATH=$PWD/src python -m pytest -q python knowledge/tools/check_knowledge_tree.py python knowledge/tools/check_wiki_lint.pytest_review_checklist_resolves_from_adapter_knowledgenow pins the budget from both sides: a marker past the old 7k cap must be injected, and a tail past the new 8k cap must not be.Note for anyone running the suite here: the shared venv has
infermatrix_copilotinstalled editable against another checkout, and this repo's pytest config does not setpythonpath. WithoutPYTHONPATH=$PWD/srcthe tests import the other tree and silently pass against unmodified code -- that is how the first run of the budget test looked green-adjacent while the injection was still 7k.Test Result
Full suite on this branch: 1 failure,
test_phase1_cluster.py::test_drift_guard_unimportable_checked_class_fails_guard, identical toorigin/mainate40258d3b(verified by running the full suite on a pristine baseline worktree). Atest_testing_substrate.pyprocess-reaping test failed once under full-suite load and passes in isolation on both trees; it did not reproduce on rerun.Knowledge validators:
check_knowledge_tree.py0 errors / 22 pre-existing size reminders;check_wiki_lint.py0 errors, 0 warnings across 387 pages.Two ratchets caught a first draft that named the target repo in a core comment (
test_repo_neutral_core,test_parity_vocabulary_is_contained); the comment was neutralized rather than the ceilings bumped.Not covered: no live review was run, so the rule's finding quality on real PRs is unmeasured.
🤖 Generated with Claude Code
https://claude.ai/code/session_01JCk5xJGRyydouK7tuhs8kk