Skip to content

grade: add --write to persist the score back into the trace - #15

Open
saikrishnarallabandi wants to merge 1 commit into
LiberCoders:mainfrom
saikrishnarallabandi:feat-grade-write
Open

grade: add --write to persist the score back into the trace#15
saikrishnarallabandi wants to merge 1 commit into
LiberCoders:mainfrom
saikrishnarallabandi:feat-grade-write

Conversation

@saikrishnarallabandi

Copy link
Copy Markdown

Problem

cmd_grade recomputes a full grading_result — dimensions, task_score, passed — and then only prints it. The trace file is never updated. _append_grading_to_trace exists and is used by the run/batch path, but cmd_grade does not call it.

So claw-anything grade cannot be used to re-score an existing trace. It looks like it worked (it prints the new scores, exits 0), but nothing persists.

Why this matters

Adding an LLM judge to an already-finished --no-judge run should be cheap: the judge reads audit_data from the trace, so no mock services and no agent re-run are needed — only judge tokens.

We tried exactly that across 226 finished traces. Every invocation printed a fresh judged score and exited 0. Nothing changed: 0 of 148 completion values differed from the un-judged originals. The re-grade was a no-op we only caught by diffing against a backup and comparing file mtimes.

With --write, the same operation works and is dramatically cheaper than re-running the agent — for us it avoided re-spending ~27M tokens on a metered model purely to attach a judge.

Reproduce

# a trace produced with --no-judge
claw-anything grade --trace run/T04_x/T04_x.jsonl --task benchmark/tool/T04_x
grep -c grading_result run/T04_x/T04_x.jsonl   # unchanged

The printed completion: differs from the one stored in the file.

Fix

Add an opt-in --write flag. When set, drop any existing grading_result lines and append the fresh one, so a re-grade replaces rather than stacks (two grading_result events in one trace make downstream "last wins" readers ambiguous).

Default behaviour is unchanged — without --write, grade still only prints.

Verified: a trace re-graded with a judge goes completion 0.163 → 0.628, and the file reflects it.

Environment

OS Ubuntu 22.04.5 LTS, kernel 6.8.0-124
Python 3.12.10
Docker 29.1.3 (grade run inside the trial image)
Judge gpt-4o-mini via https://api.openai.com/v1
Base commit verified on f9a0f63 and current main (e19dce2)

Note

This is the only one of my three PRs that adds a flag rather than fixing a bug — happy to rename it or gate it differently if you would prefer --in-place, or for it to be the default.

cmd_grade computed a fresh grading_result and only printed it; the trace file
was never updated. Re-grading a finished --no-judge run to add an LLM judge was
therefore a silent no-op. --write strips any existing grading_result and appends
the fresh one, which makes re-grading cost judge tokens only rather than a full
agent re-run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant