Repository navigation
Close the generalizable held-out security-gate misses and score them out of sample - #1989
Merged
Merged
Conversation
…ctive guard, and score them out of sample Secret gate: provider-format patterns for the AWS secret access key (docs example stays allowed), Twilio SID and auth token, JWT, Azure AccountKey/SharedAccessKey, Discord bot token, Linear, Mailgun, Docker Hub PAT and Telegram bot token. Destructive guard: aws s3 rm --recursive / rb --force, ec2 terminate-instances, rds delete-db-*, gcloud projects delete, az group delete, terraform/tofu destroy, kubectl delete namespace / all --all, redis FLUSHALL/FLUSHDB, mongo dropDatabase, wipefs, diskutil erase*, and eval/sh of decoded base64. Held-out-1024 before to after: secret 21/30 to 30/30 (FP 2/30), guard 26/40 to 40/40 (FP 3/40); it is now in-sample. First corpus stays 30/31 FP 1/34 and 46/46 FP 0/45. Out of sample on the preregistered held-out-2 corpus (rules frozen): secrets Perun 23/40 FP 2/40 vs ported gitleaks 26/40 FP 0; guard Perun 37/50 FP 4/50 vs ported safety-net 18/50 FP 2 and hooks-mastery 9/50 FP 5. Perun still trails gitleaks on secrets. The landed held-out-1024 corpus is immutable, so the new patterns' hits on its unmarked rows (and a pre-existing hit in held-out-2 decoys) are handled by a narrow privacy-gate skip for the frozen security-gates secret-leaks.txt and secret-decoys.txt files, tested in both directions. score.py accepts id<TAB>cmd rows and hashes from SHA256-frozen.txt; bench_check.py resolves a bare frozen filename from the bench dir first; the held-out-1024 pin test cross-checks ledger amendments against the frozen list; the README notes the ledger was reconstructed after the fact. Tests: scripts/test_security_gates_heldout_fixes.py (35+15 caught, 25+11 allowed, exemption both ways), heldout, heldout2 and corpus pins pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds rules for the held-out misses from #1988 that generalize, and scores them on a second blind corpus that was frozen before the rules were written.
Rule changes:
.banlist.txt: provider token formats.destructive_guard.py: cloud, Kubernetes, Terraform, NoSQL, disk-tool and base64-eval rules.Results on the second held-out corpus (
docs/bench/security-gates-heldout2-1024/). The corpus was preregistered in 8abdefc93 before these rules existed, and no rule changed after scoring:git filter-branch, some cloud and Helm deletes, printf-hex andeval $(echo …)obfuscation,redis-clireading FLUSHALL from stdin, andaws s3 sync --delete.docs/perun-vs-rivals.mdkeeps its blind numbers as "as scored blind". The original corpus is unchanged.Tooling:
score.pyaccepts id-prefixed rows and checks hashes againstSHA256-frozen.txt.bench_check.pyresolves a bare frozen filename from the bench directory first.security-gates-*file still fails.The independent review found 2 🔴, both fixed, and a re-review checked the fixes:
Four 🟡 were also folded in: a
kubectl delete -n <ns> ns <name>bypass, the in-sample row label, the freeze-order wording, and the known gaps. Re-scoring after these fixes gives identical numbers.GitHub push protection blocked one placeholder decoy key, row D2-08. Its
@@marker is recorded as an amendment inledger.jsonlandSHA256-frozen.txt. The pin test checks that the first five lines still hash to the preregistered value, and the@@-stripped content and every score are unchanged.Local CI repro:
test-ci-gates.shgives 297 passed and 9 failed. The same 9 fail on the parent commit (sandboxpsand network). main had 11 failures before.🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z