Bench: answer keys and a deterministic scorer for four stalled rival rows

changelog.d/bench-answer-keys.md

@@ -0,0 +1,2 @@
+### Added
+- Bench: answer keys (`keys.json`) for all 40 held-out scenarios of the business-ops, growth-analytics, idea-critic and contribution rival rows, 10 per row, 3-6 must-hit and 1-3 must-not regex items each. Keys test what an answer adds beyond the prompt (an echo baseline of each prompt scores at most 0.25 hit rate), accept number forms (`$280k`, `100,000`, `13 percent`) and paraphrases, and use no Perun verdict tokens. `scripts/rival_eval_keys.py` (a wrapper around the unchanged, hash-pinned `scripts/rival_eval.py`) adds `key_score` (per-arm hit rate and violation rate, deterministic, no judge). Must-nots are negation-aware ("never name X" is not a leak), skip bracketed placeholders, and for contribution scan only the drafted message; a scenario that needs a draft and has none counts as a violation (`no_draft`). Results land in `results.json` as `answer_key` next to the pairwise verdict. `rival_eval_keys.py keys SKILL RUNDIR` rescores cached outputs offline and fails on a missing arm directory. `check` rejects an unkeyed or malformed key. Tests: `scripts/test_rival_answer_keys.py`.

docs/perun-vs-rivals.md

@@ -1,6 +1,6 @@
 # Perun vs other skill bundles
 
-Harness: run rival rows with `scripts/rival_eval_cost.py run ...`, a wrapper around the hash-pinned `scripts/rival_eval.py` that also records per-arm cost into `results.json`; `scripts/bench_verdict.py --cost results.json` reads it (exit 0 Perun cost <= rival, 1 Perun costs more, 2 usage error, 3 unmeasured).
+Harness: run rival rows with `scripts/rival_eval_keys.py run ...`, a wrapper around the hash-pinned `scripts/rival_eval.py` that also records per-arm cost and, for skills with a `keys.json`, a deterministic answer-key score (`rival_eval_keys.py keys SKILL RUNDIR` rescores cached outputs) into `results.json`; `scripts/bench_verdict.py --cost results.json` reads it (exit 0 Perun cost <= rival, 1 Perun costs more, 2 usage error, 3 unmeasured).
 
 **Bottom line (2026-10-17 to 2026-10-19 held-out runs, free PCSS models, judge protocol v2).** Five skills have a confirmed preference-only win over their rival (the judge preferred Perun with the 95% lower bound above 0.5; cost and user outcome were not measured, so none is a win on cost or outcome) (product-output-safety's is narrow: lower bound 0.518 on its preregistered replication, one earlier fresh set below 0.5); idea-critic won once (round 9, narrowly) and the preregistered replication on a second fresh set did not reproduce it, so it is not counted; four more are not wins (growth-analytics among them: its round-4 edit shipped on a paired gain but its win-rate lower bound is 0.483, point estimate 0.550; a round-5 candidate got no verdict because no judge was inside the slot-1 gate, and was not shipped) and none is unproven. A win needs the 95% CI lower bound above 0.5; every row below is a fresh held-out set of n=40 or more, with ties counted as 0.5. Rows with n=10 are not mixed in: they are the older history further down and are not comparable to this table.
 

scripts/ci-tests.d/rival-answer-keys.sh

@@ -0,0 +1,2 @@
+# Registers the rival answer-key scorer tests with CI (see the "Per-PR tests" step in ci.yml). Offline; no model is called.
+python3 scripts/test_rival_answer_keys.py

scripts/eval-fixtures/rival/business-ops/keys.json

@@ -0,0 +1,123 @@
+{
+ "07": {
+  "must": {
+   "unit_margin_37": "\\b37\\b",
+   "direct_margin_72": "\\b72\\b",
+   "volume_ratio": "1\\.9\\d?\\s*(x|times)|1\\.95|nearly (twice|double)|(roughly|about|almost) (twice|double)",
+   "inventory_cash_540k": "(?<![\\d,.])[$\u00a3\u20ac]?540,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?540k\\b",
+   "counter_offer": "counter|negotiat|minimum order|tiered|price floor|volume tier"
+  },
+  "mustnot": {
+   "commits_followon": "20,?000 (units )?(are|is|will be) (committed|guaranteed|certain)",
+   "no_risk": "guaranteed profit|no downside|risk[- ]free"
+  }
+ },
+ "08": {
+  "must": {
+   "liability_shield": "veil|liability shield|separate legal|asset protection",
+   "state_registration": "foreign (qualification|registration)|nexus|register(ed)? in|state (tax|filing|registration)",
+   "payroll_100k": "(?<![\\d,.])[$\u00a3\u20ac]?100,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?100k\\b",
+   "pro_threshold": "(CPA|tax (professional|advisor|adviser)|accountant|attorney|lawyer|solicitor|counsel).{0,120}(\\$\\s?\\d|\\d{2,3}k|\\d+ (employees|hires)|first (hire|employee)|payroll)|(\\$\\s?\\d|\\d{2,3}k|\\d+ (employees|hires)|first (hire|employee)|payroll).{0,120}(CPA|tax (professional|advisor|adviser)|accountant|attorney|lawyer|solicitor|counsel)"
+  },
+  "mustnot": {
+   "definitive_entity": "you (should|must) (definitely )?(form|elect|choose) an? (S-?corp|LLC) (now|immediately|today)"
+  }
+ },
+ "09": {
+  "must": {
+   "heavy_200": "\\b200\\b",
+   "revenue_at_risk": "(?<![\\d,.])[$\u00a3\u20ac]?60,?200(?!\\d|,\\d)|(?<![\\d,.])[$\u00a3\u20ac]?100,?200(?!\\d|,\\d)|\\b301\\b|\\b501\\b",
+   "churned_30": "\\b30 (heavy|high|power)?[- ]?(usage )?(accounts|customers|users)",
+   "breakeven": "break-?even",
+   "softer_alternative": "overage|spend cap|usage alert|committed[- ]use|bill[- ]shock (alert|protection)|prepaid"
+  },
+  "mustnot": {
+   "claims_churn_fixed": "capping (will|would) (eliminate|solve|fix) (the )?churn",
+   "guarantee": "guarantee[sd]? (retention|lower churn)"
+  }
+ },
+ "10": {
+  "must": {
+   "emi": "\\bEMI\\b",
+   "value_10k": "(?<![\\d,.])[$\u00a3\u20ac]?10,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?10k\\b",
+   "salary_gap_20k": "(?<![\\d,.])[$\u00a3\u20ac]?20,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?20k\\b",
+   "hmrc": "HMRC",
+   "adviser_before_grant": "(solicitor|lawyer|advis[eo]r).{0,80}before (you )?(grant|issu|sign|offer)|before (you )?(grant|issu|sign|offer).{0,80}(solicitor|lawyer|advis[eo]r)"
+  },
+  "mustnot": {
+   "definitive_tax": "(will|would) (pay|owe) no tax|guaranteed (tax|relief)|tax[- ]free for (sure|certain)"
+  }
+ },
+ "11": {
+  "must": {
+   "payers_2000": "\\b2,?000 (paying|paid|subscribers|payers|users|customers)",
+   "mrr_20k": "19,?9\\d\\d|\\b20k|\\$20,?000",
+   "costs_19k": "(?<![\\d,.])[$\u00a3\u20ac]?19,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?19k\\b",
+   "control_test": "A/B|control group|holdout|split test",
+   "cannibalization": "cannibal|downgrade|trade down|substitut"
+  },
+  "mustnot": {
+   "asserts_gain": "will (definitely )?increase (total )?revenue",
+   "invented_elasticity": "elasticity (is|of) -?\\d"
+  }
+ },
+ "12": {
+  "must": {
+   "special_category": "special[- ]categor|article 9|explicit consent",
+   "dpia": "DPIA|data protection impact",
+   "lawyer_trigger": "(CPA|tax (professional|advisor|adviser)|accountant|attorney|lawyer|solicitor|counsel).{0,100}(when|once|before|if you|threshold|trigger)|(when|once|before|if you|threshold|trigger).{0,100}(CPA|tax (professional|advisor|adviser)|accountant|attorney|lawyer|solicitor|counsel)|\\bDPO\\b",
+   "total_users_15k": "(?<![\\d,.])[$\u00a3\u20ac]?15,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?15k\\b"
+  },
+  "mustnot": {
+   "no_lawyer": "no (privacy )?(lawyer|counsel) (is )?needed|you are (fully )?(exempt|compliant)"
+  }
+ },
+ "13": {
+  "must": {
+   "target_fee": "(?<![\\d,.])[$\u00a3\u20ac]?24,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?24k\\b|(?<![\\d,.])[$\u00a3\u20ac]?23,?400(?!\\d|,\\d)",
+   "concentration": "concentration|both clients|two clients|single point",
+   "downside_floor": "floor|base fee|retainer|collar|minimum fee",
+   "effective_rate": "\\b15[34]\\b",
+   "value_evidence": "baseline|outcome|ROI|case stud"
+  },
+  "mustnot": {
+   "risk_free": "risk[- ]free|no risk|guarantee[sd]? (a )?(20|higher)"
+  }
+ },
+ "14": {
+  "must": {
+   "cac_50": "\\$50\\b",
+   "payback": "\\b6\\.[23]\\d?\\b",
+   "ltv_margin": "\\b(160|144)\\b",
+   "ratio": "\\b(3\\.2|3\\.1|3\\.0|2\\.9|2\\.88)\\b",
+   "m2_decline": "\\b10 ?(%|percent)",
+   "nrr": "contraction|expansion|net revenue retention|\\bNRR\\b|gross (revenue )?churn"
+  },
+  "mustnot": {
+   "unmargined_ltv": "LTV (is|=|of|equals) \\$?200\\b"
+  }
+ },
+ "15": {
+  "must": {
+   "weekly_5k": "((?<![\\d,.])[$\u00a3\u20ac]?5,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?5k\\b) ?(per|a|/|each) ?week|\\$5k",
+   "exceeds_grant": "(?<![\\d,.])[$\u00a3\u20ac]?65,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?65k\\b|(?<![\\d,.])[$\u00a3\u20ac]?130,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?130k\\b|(?<![\\d,.])[$\u00a3\u20ac]?260,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?260k\\b",
+   "net_negative": "net (negative|loss)|grant (does not|doesn't|won't) cover|exceeds the grant|more than the grant|costs more than",
+   "hours_25": "\\b25 (hours|hrs|h)\\b"
+  },
+  "mustnot": {
+   "free_money": "(pure|free) (money|profit)|grant (is|will be) (clearly )?worth it"
+  }
+ },
+ "16": {
+  "must": {
+   "vietnam_180k": "(?<![\\d,.])[$\u00a3\u20ac]?180,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?180k\\b",
+   "volume_not_revenue": "take rate|commission|GMV",
+   "minimum_performance": "minimum|quota|milestone|performance (target|threshold)",
+   "exit_clause": "terminat|revert|lose exclusivity|exit",
+   "total_1_3m": "1\\.3 million|(?<![\\d,.])[$\u00a3\u20ac]?1,?300,?000(?!\\d|,\\d)|(?<![\\d.])[$\u00a3\u20ac]?1300k\\b|\\$1\\.3m"
+  },
+  "mustnot": {
+   "volume_as_revenue": "1\\.3 million (in )?revenue|\\$?500,?000 (in )?revenue"
+  }
+ }
+}

scripts/eval-fixtures/rival/contribution/keys.json

@@ -0,0 +1,135 @@
+{
+ "07": {
+  "must": {
+   "numbered_plan": "PR ?1.{0,200}PR ?2.{0,200}PR ?3|first PR.{0,200}second PR.{0,200}third|(#1|\\b1\\)).{0,150}(#2|\\b2\\)).{0,150}(#3|\\b3\\))",
+   "each_builds": "(each|every) (PR|pull request|step|commit).{0,60}(builds?|compiles?|passes|green|CI)",
+   "dependency": "depends on|stacked|based on|merge (in )?order|follow-?up",
+   "generic_rename": "(rename|placeholder|neutral|generic) (names?|identifiers?|variables?|constants?)",
+   "tone": "thank|appreciate"
+  },
+  "mustnot_draft": {
+   "leaks_company": "Acme Logistics|warehouse code",
+   "leaks_schedule": "deploy(ment)? schedule"
+  },
+  "needs_draft": true
+ },
+ "08": {
+  "must": {
+   "advisory": "security advisories?|GitHub's private|SECURITY\\.md|security@|PGP|encrypted",
+   "no_payload": "without (the )?(payload|exploit|proof)|(do not|don't|will not|won't) (include|attach|share) (the )?(payload|exploit|proof)|(payload|exploit).{0,30}(upon|on) request|withhold|omit",
+   "timeline": "90[- ]day|\\b(30|45|60|90) days|disclosure (deadline|window|timeline)",
+   "escalation": "CERT|CVE|MITRE|coordination (center|service)"
+  },
+  "mustnot": {
+   "public_issue_advice": "(should|recommend|go ahead and) (open|file) a public issue"
+  },
+  "mustnot_draft": {
+   "leaks_company": "Globe Finance"
+  },
+  "needs_draft": true
+ },
+ "09": {
+  "must": {
+   "retry_request": "(re-?run|retry|restart) (the )?(job|check|workflow|CI)|manual retry|runner (memory|config|resources?|size)",
+   "log_evidence": "link to (the )?(log|run)|logs? (attached|below|linked)|CI (log|run) (link|url)",
+   "unrelated": "unrelated to (my|the) (change|diff|patch)|(change|diff|patch) (doesn't|does not) (touch|affect)",
+   "passes_locally": "(passes|pass(ed)?|green) (locally|on my machine)",
+   "gracious": "happy to (help|adjust|test)|thanks for (maintaining|your time)|appreciate"
+  },
+  "mustnot_draft": {
+   "leaks_fork": "Zenith|internal (fork|CI|setup)"
+  },
+  "needs_draft": true
+ },
+ "10": {
+  "must": {
+   "acknowledge": "understand (the|your) (preference|goal|design|reasoning)|appreciate",
+   "joint_benchmark": "(shared|joint|public|reproducible|common) (benchmark|test|harness)",
+   "middle_ground": "middle ground|compromise|hybrid|behind a flag|opt-in",
+   "synthetic": "synthetic|generic workload|public (data|dataset)|representative",
+   "interim": "interim|temporary|stopgap|in the meantime|backport"
+  },
+  "mustnot_draft": {
+   "leaks_numbers": "BlueWave|15 ?%|fifteen percent"
+  },
+  "needs_draft": true
+ },
+ "11": {
+  "must": {
+   "no_api_change": "(no|without) (any )?(public )?API (change|changes|break)|backward[- ]compat|opt-in|off by default|behind a flag",
+   "rfc_first": "(before|prior to) (sending|opening|writing) (a|any) (PR|pull request|code)|\\bRFC\\b|design discussion|gauge interest",
+   "small_scope": "small|minimal|self-contained|isolated|separate (module|file|header)|maintenance (cost|burden)",
+   "will_maintain": "(I|we)('| wi)ll (maintain|own|keep|add tests)|happy to (maintain|own)|commit(ted)? to maintain",
+   "generic_case": "(any|every|all) (application|user|client)s?|(general|generic)[- ]purpose|(common|typical) (use case|workload)"
+  },
+  "mustnot_draft": {
+   "leaks_business": "Apex|latency target"
+  },
+  "needs_draft": true
+ },
+ "12": {
+  "must": {
+   "preferences": "if you have (a )?(preferred|preference)|prefer(red)? (test|commit|style)",
+   "small_patch": "single (commit|file|change)|minimal (patch|diff|fix)|small (patch|diff|fix)",
+   "follow_existing": "follow(ing)? (the )?existing|match(es|ing)? (the )?(existing|surrounding)",
+   "fallback": "(if|in case) (you('re| are)|this is) (no longer|not) (maintain|accept|review)|(happy|glad) to (maintain|fork|help co-?maintain)|no pressure",
+   "test_included": "(regression|unit) test.{0,60}(included|attached|covers|reproduc)|added a (regression )?test"
+  },
+  "mustnot_draft": {
+   "leaks_internal": "Silent Hill|session (architecture|structure)|internal test"
+  },
+  "needs_draft": true
+ },
+ "13": {
+  "must": {
+   "ask": "(what|how) (do you|would you) (prefer|want)|could you (confirm|advise|point me)",
+   "maintainer_regenerates": "(maintainer|you|someone)( can| could| to| should)? (run|re-?generate|regenerate)|regenerat(e|ion) (on|during|at) (merge|release)",
+   "no_hand_edit": "(I|we) (have not|haven't|didn't|did not) (hand-?edit|edit)",
+   "changelog": "changelog (entry )?(will be|is) (added|generated) (by|at)|leave (the )?changelog|not (touching|editing) the changelog"
+  },
+  "mustnot_draft": {
+   "leaks_bypass": "DataCrunch|bypass|workaround"
+  },
+  "needs_draft": true
+ },
+ "14": {
+  "must": {
+   "interactive": "git rebase (-i|--interactive)",
+   "backup": "git branch (backup|bak)|backup branch|reflog",
+   "force_with_lease": "--force-with-lease",
+   "comments_persist": "(comments|review|conversation).{0,60}(stay|remain|persist|preserved|are not lost)",
+   "reword": "\\breword\\b|\\bfixup\\b|filter-repo|git commit --amend"
+  },
+  "mustnot_draft": {
+   "bare_force_push": "git push\\b[^\\n]*?(\\s-f\\b|\\s--force(?![-\\w]))",
+   "leaks_org": "Prism Solutions"
+  },
+  "needs_draft": true
+ },
+ "15": {
+  "must": {
+   "acknowledge": "thank(s| you) (for|to)|appreciate (the|everyone's) (work|effort)",
+   "root_cause": "(initiali[sz]ation|initial count|initial value).{0,80}(root cause|underlying|actual cause)",
+   "open": "(would|if) (you|the maintainers) (be open|like|prefer)|happy to (close|withdraw|fold)|not (trying|meaning) to (compete|pile)",
+   "test": "(regression|reproduc(tion|er)|failing) test",
+   "refs": "#\\d+|(PR|pull request) (#|number)"
+  },
+  "mustnot_draft": {
+   "leaks_setup": "Velocity Systems|stress test|modified (version|broker)|internal"
+  },
+  "needs_draft": true
+ },
+ "16": {
+  "must": {
+   "separate": "(i|we)('d| would|'ll| will) (split|send|open|submit|file)[^.]{0,60}separate|(docs?|documentation)[^.]{0,40}(in|as) (a )?(separate|second|follow-?up)|(two|2) (PRs|pull requests)",
+   "ordering": "(after|once) the fix (lands|merges|is merged)|follow-?up (PR|pull request)|docs? (PR )?(follows|after)",
+   "fictional": "(fictional|made-up|hypothetical|toy|contrived) (service|counter|cache|queue|scenario|worker)",
+   "concept": "happens-before|data race|mutex|atomic|synchroni[sz]ed|immutable",
+   "ask": "(open|file) an issue|ask (the )?maintainers|check (with|whether)"
+  },
+  "mustnot_draft": {
+   "leaks_product": "Nexus Analytics|proprietary"
+  },
+  "needs_draft": true
+ }
+}