grade: add --write to persist the score back into the trace - #15
Open
saikrishnarallabandi wants to merge 1 commit into
Open
grade: add --write to persist the score back into the trace#15saikrishnarallabandi wants to merge 1 commit into
saikrishnarallabandi wants to merge 1 commit into
Conversation
cmd_grade computed a fresh grading_result and only printed it; the trace file was never updated. Re-grading a finished --no-judge run to add an LLM judge was therefore a silent no-op. --write strips any existing grading_result and appends the fresh one, which makes re-grading cost judge tokens only rather than a full agent re-run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cmd_graderecomputes a fullgrading_result— dimensions,task_score,passed— and then only prints it. The trace file is never updated._append_grading_to_traceexists and is used by the run/batch path, butcmd_gradedoes not call it.So
claw-anything gradecannot be used to re-score an existing trace. It looks like it worked (it prints the new scores, exits 0), but nothing persists.Why this matters
Adding an LLM judge to an already-finished
--no-judgerun should be cheap: the judge readsaudit_datafrom the trace, so no mock services and no agent re-run are needed — only judge tokens.We tried exactly that across 226 finished traces. Every invocation printed a fresh judged score and exited 0. Nothing changed: 0 of 148 completion values differed from the un-judged originals. The re-grade was a no-op we only caught by diffing against a backup and comparing file mtimes.
With
--write, the same operation works and is dramatically cheaper than re-running the agent — for us it avoided re-spending ~27M tokens on a metered model purely to attach a judge.Reproduce
The printed
completion:differs from the one stored in the file.Fix
Add an opt-in
--writeflag. When set, drop any existinggrading_resultlines and append the fresh one, so a re-grade replaces rather than stacks (twograding_resultevents in one trace make downstream "last wins" readers ambiguous).Default behaviour is unchanged — without
--write,gradestill only prints.Verified: a trace re-graded with a judge goes
completion 0.163 → 0.628, and the file reflects it.Environment
graderun inside the trial image)gpt-4o-miniviahttps://api.openai.com/v1f9a0f63and currentmain(e19dce2)Note
This is the only one of my three PRs that adds a flag rather than fixing a bug — happy to rename it or gate it differently if you would prefer
--in-place, or for it to be the default.