Repository navigation
Record per-arm bench cost and apply the cost check - #1962
Closed
remigiusz-antczak wants to merge 2 commits into
Closed
remigiusz-antczak wants to merge 2 commits into
remigiusz-antczak wants to merge 2 commits into
Conversation
rival_eval.py now writes cost_per_arm (prompt_tokens, completion_tokens, spend_usd) taken from the PCSS response or claude -p JSON, never estimated, with a per-output sidecar so resumed runs keep it. bench_verdict.py --cost prints per-arm cost per item and the Perun/rival ratio and exits 1 unless Perun cost <= rival cost. Tests: 14/14 (new) + bench_verdict, rival_eval, prepush parity 6/6 green Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
Collaborator
Author
|
Superseded by #1980. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bench results recorded
spend_usd 0.0for every arm, so the cost leg of "superior" was unmeasured on every row of the scoreboard. This change records real per-arm usage and makes the verdict check cost.scripts/rival_eval.pyrecords prompt and completion tokens per item, plus spend in USD when the provider reports it. A missing price is stored as unmeasured, never as 0.scripts/bench_verdict.pyreports the Perun/rival cost ratio and fails the cost check closed when either arm is unmeasured, when the arms report in different units, or when the rival cost is zero.Older results.json files still read as unmeasured, so this changes no existing scoreboard row. Rows get cost numbers on their next run.
An independent review found one 🔴, fixed in this commit: a partial run could read as measured.
Tests: 14/14 in scripts/test_bench_cost_capture.py; test_bench_verdict and test_rival_eval green; prepush parity 6/6.
🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z