[remigiusz-antczak]

Supersedes #1966. The same change, rebuilt as one commit on current main (after #1981), with six fixes from an independent review.

Four rival-bench rows (business-ops, growth-analytics, idea-critic, contribution) had no outcome score a cost check could pair with. This adds deterministic answer keys, so those rows can be scored without a judge model and without a paid call.

• keys.json for the 40 held-out scenarios of the four skills: 3-6 must-hit and 1-3 must-not regex items per scenario. Keys test what an answer adds beyond its prompt (echo baseline at most 0.25), accept number forms and paraphrases, and carry no Perun verdict tokens.
• scripts/rival_eval_keys.py wraps the cost-capture wrapper from Record per-arm bench cost and apply the cost check #1981; rival_eval.py stays unchanged and hash-pinned. It adds key_score (per-arm hit rate and violation rate), an answer_key block in results.json, a keys subcommand that rescores cached outputs offline, and key hygiene in check. rival-eval-premerge.sh now runs it.

Fixes from the review of the rebuilt branch:

• A missing keys.json for a skill that ships keys now fails closed instead of silently skipping scoring.
• check validates every key entry's shape, so a malformed key fails before a paid run rather than at score time.
• Must-not matching no longer blanks bracketed or angle-bracketed spans, which could hide a violation. Negation only counts when it sits in the few words right before the match in the same clause, so "No, you should definitely form an LLC now" is now a violation. A generic placeholder such as [Company] is still fine.
• A scenario missing for one arm now counts as a miss for that arm instead of being dropped for every arm.
• The draft check now needs a non-trivial drafted message, not one quoted line or an empty fence.
• The offline keys rescore picks each output by the current prompt hash, so stale outputs from an older prompt no longer make it depend on file order.

Tests: 19/19 in scripts/test_rival_answer_keys.py, and 47/47 in test_rival_eval. Local CI repro on this commit: size-ratchet ok against main, mustload ok, and the rival-answer-keys and bench-cost-capture hooks pass. The full scripts/ci-tests.d loop and test-ci-gates.sh ran on the pre-fix rebuild with only the 9 known sandbox-only failures.

🤖 Generated with Claude Code

https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z