[tzhouam]

Summary

• Retain the selected best-performing v20cb train and validation arm records: manifests, reviews, costs, and packed traces.
• Retain the matching blind Sonnet judgment records and generated reports against the existing claudecode_opus5 baseline.
• Leave the tracked baseline unchanged and intentionally exclude the local v18dsh, v19, v21, v22, v23, and invalid MoA records.

Why

v20cb is the performance point worth preserving from the local campaign. Keeping its exact outputs, provenance, packed traces, and judgments alongside the frozen baseline lets other maintainers inspect and reproduce the comparison without retaining every exploratory variant.

Impact

This is a data-only change with no runtime behavior change. It adds 94 evaluation files: 15 item records with costs and packed traces, two manifests, two judge reports, and 45 blind verdicts.

Validation

• PYTHONPATH=src python -m pytest -q test/test_trace_pack.py test/test_campaign_tooling.py — 15 passed
• python eval/dataset/trace_pack.py verify eval/dataset/arms/copilot_v20cb_train eval/dataset/arms/copilot_v20cb_val — verified all 15 packed traces
• Parsed all committed JSON records with jq empty
• Passed gzip -t for all 15 trace archives
• Passed git diff --check origin/main...HEAD
• Confirmed the PR diff contains exactly the four v20cb artifact directories and no baseline changes

Known limitations

• The ignored private run directories were generated without payload-level events.jsonl, so the raw-run completeness gate reports those runs as non-replayable at the request/response payload level. The committed packed traces pass the repository's integrity, hash, and path-scrubbing verifier.
• validate_replicate.py still hard-codes the old copilot_v2 / opus_baseline blinding labels. Against these records it reports label-name mismatches; heads, return codes, counts, content hashes, and verdict structure otherwise validate.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c2adf7115

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Populate cost totals from recorded metrics

All 15 new cost records set attempt_usd_total and the top-level token counts to zero even though their nested metrics.cost contains nonzero event-derived values (this item records $0.5684 and 1,690,401 input tokens). eval/dataset/aggregate_costs.py treats a present attempt_usd_total as authoritative and reads tokens only from the top level, so aggregating these retained arms reports $0 and zero tokens rather than $2.6971 for train and $1.4340 for validation, corrupting cost and cost-per-success comparisons. Populate these fields from the recorded metrics, or omit unavailable totals so the aggregator's metrics fallback is used.

Useful? React with 👍 / 👎.