Repository navigation
Bench: blind held-out score of the secret gate and destructive guard against ported rivals - #1988
Merged
Merged
Conversation
…against ported rivals Four synthetic corpora (30 leaks, 30 decoys, 40 dangerous, 40 benign) were written before the rules or the first corpus were read, and their hashes committed first (preregistration commit ad168d8c6; secret-leaks.txt d7aef600ac9ad7c9b511c7f467e066d12f5bf99d98197192b9ec6eadbcc9217c, decoys c5f82cec87517931a0fed914c2fa8ece8a096868b535260856686a4560329c12, dangerous b0962b37d1bab25264ce28b9ae09488412496a6afd622c86f44ddb4accee3a4f, benign 257f644fa288203f17f29545c5b30c8261b527f0a19ebed5201654280b4b3f32). Three leak rows later gained privacy markers only; content with markers removed is byte-identical. Held-out result, no rule changed after scoring: Perun 21/30 leaks (2/30 decoys flagged), 26/40 dangerous blocked (3/40 benign); gitleaks port 23/30 (0 FP); hooks-mastery port 9/40 (6 FP); safety-net subset port 17/40 (2 FP). Misses are logged as leads in the README. score.py gains --corpus-dir; pins run in CI. Tests: scripts/test_security_gates_heldout.py ok; test-mustload-split.sh 0 FAIL; size, size-ratchet, fragments, privacy, diff --check clean Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1987, whose CI failed the bench-integrity gate because the bench had no
ledger.jsonl. This version adds one:SHA256-frozen.txtkeeps the four preregistered lines (commit ad168d8c6) and appends the twosecret-leaks.txtmarker amendments, and the ledger records the preregistration and each amendment with its time. The pins test checks the last line per file.docs/INDEX.mdis regenerated.Every security-gate number so far was in-sample: the rules were tuned after seeing the corpus misses. This adds a held-out corpus written blind to Perun's rules and the earlier corpus, with hashes committed before scoring. No rule changed after scoring.
Held-out results (
docs/bench/security-gates-heldout-1024/):git stash drop,find -delete,rm -rf /tmp/...) are debatable labels, noted in the README.Integrity: the preregistration commit was squashed away, but the README and
SHA256-frozen.txtrecord its hash, andscripts/test_security_gates_heldout.pyasserts that the@@-stripped content matches it. Marker-only edits came after preregistration: three rows flagged by the repo privacy gate before scoring, and three flagged by GitHub push protection after scoring. They don't change the stripped content or any score.Caveat: the corpus author is from the same model family as the rule author, so the two may share blind spots. Rival numbers come from ported rules, not the binaries.
An independent review found 0 🔴 and 2 🟡 (the prereg-reachability wording and the AWS-doc-key label caveat), both folded in.
Local CI repro: pins test ok; privacy,
diff --check, size, size-ratchet, fragments and mustload are clean.test-ci-gates.shgives 293 passed and 11 failed with the same failure list as main.🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z