Repository navigation
Record per-arm bench cost and apply the cost check - #1981
Merged
Merged
Conversation
rival_eval_cost.py (a wrapper around the unchanged, hash-pinned rival_eval.py; rival-eval-premerge.sh now runs it) writes cost_per_arm (prompt_tokens, completion_tokens, spend_usd) taken from the PCSS response or claude -p JSON, never estimated, with a per-output sidecar so resumed runs keep it. bench_verdict.py --cost prints per-arm cost per item and the Perun/rival ratio; exit 0 Perun cost <= rival, 1 Perun costs more, 2 usage error, 3 unmeasured. A billed retry adds to the item's usage, and an attempt with unknown usage leaves the item unmeasured. Zero cost on both arms reads unmeasured, and an unreadable results file exits 3. Tests: 21/21 (test_bench_cost_capture) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1980 and #1962. The same change, rebuilt as one commit on current main, with three fixes from an independent review of #1980.
The bench recorded
spend_usd: 0.0for every arm, so the cost half of the keep rule could never be applied.scripts/rival_eval_cost.py(new) wraps the hash-pinnedscripts/rival_eval.py. It records prompt and completion tokens per item, and spend in USD when the provider reports it. An arm counts as measured only when every item has usage. Unlike Record per-arm bench cost and apply the cost check #1962,rival_eval.pyitself is left unchanged, so its pinned hash still holds.scripts/bench_verdict.py --cost results.jsonreports the Perun-to-rival cost ratio. It exits 0 when Perun costs no more than the rival, 1 when it costs more, 2 on a usage error and 3 when either arm is unmeasured, the units differ, or the rival cost is zero. Older results read as unmeasured.scripts/rival-eval-premerge.shcalls the wrapper.docs/perun-vs-rivals.mddocuments the harness entry point.scripts/test_bench_cost_capture.pyand run fromscripts/ci-tests.d/bench-cost-capture.sh.The review of #1962 found one 🔴: a partial run could read as measured. It is fixed here, because every item must carry usage. The review of #1980 found three 🟡, also fixed here:
Tests: 21/21 in
scripts/test_bench_cost_capture.py.Local CI repro on this commit: size-ratchet ok against main, mustload ok, and every
scripts/ci-tests.dhook passes.run-ledger.shfailed once only because a second copy ran at the same time; run alone it passes 26/26.test-ci-gates.shshows 295 passed and 9 failed. The 9 are the known sandbox-only failures (dcr-gates opt-in, peer pains, reap_own pgrep, tracker socket bind).🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z