Repository navigation
Make the secret-gate and destructive-guard rows reproducible and fix their misses - #1985
Merged
Merged
Conversation
…the misses they exposed
The 20-leak and 32-string corpora behind rows H5 and H8 were lost, so their 17/20 and 30/32 could not be reproduced. Add frozen synthetic corpora with an offline scorer (docs/bench/security-gates-1024: 31 leaks, 34 decoys, 46 dangerous commands, 45 benign) and fix the generalizable misses.
Before the fixes the secret gate caught 20/31 with 1/34 decoys flagged and the guard blocked 29/46 with 0/45 false positives. After: 30/31 (1/34, the public AWS example key) and 46/46 (0/45). The fixes post-date the corpus, so these numbers are in-sample, and rival tools were not run (not installed, owner gate).
- .banlist.txt: token shapes for GitLab, Slack app, SendGrid, Twilio, npm, PyPI, Hugging Face, Shopify and DigitalOcean, plus credentialed connection URLs.
- destructive_guard.py: ${IFS} and brace-expanded commands, variable-built command words, ssh remote commands, docker volume rm/prune, system prune --volumes/-a and bulk rm via substitution, kill -1/0 (also bare `kill 0`), shutdown/reboot, shred, truncate to 0, rsync --delete, SQL DROP/TRUNCATE through a client, fork bombs, force-with-lease to main/master, worktree remove --force, string-built rm -rf in interpreter one-liners, secret variables in a URL query.
- Held-out variants and decoys for every fix in scripts/test_security_gates_corpus.py.
- docs/perun-vs-rivals.md H5/H8 rows: old 17/20 and 30/32 marked as not reproducible.
Not fixed: a 35-character ghs_ fixture (one short of the real format) and the AWS example key, which an ERE cannot exclude; a secret piped to curl -d @- is still allowed.
Tests: test_security_gates_corpus (held-out, timing), test-destructive-guard 15/15, test-secret-gate-recall, ci-tests.d bench-check/rivals-outcome-audit/scoreboard; test-ci-gates 295 passed, 9 failed (sandbox-only: dcr-gates opt-in x7, pgrep/ps x2)
No-Mechanism-Reason: destructive_guard.py is a script, not skill prose; mechanism is scripts/test_security_gates_corpus.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The H5 (secret gate) and H8 (destructive-command guard) rows in
docs/perun-vs-rivals.mdcited 17/20 and 30/32, but the corpora behind them were never committed and are lost. This change makes both rows reproducible and fixes the misses the new corpora exposed.docs/bench/security-gates-1024/: frozen synthetic corpora (31 fake secret leaks, 34 decoys, 46 dangerous and 45 benign command strings), SHA256-pinned, plus an offline scorerscore.py. Secret lines carry@@markers so they never trip the repo privacy gate or push protection..banlist.txt): adds GitLab, Slack app, SendGrid, Twilio, npm, PyPI, Hugging Face, Shopify, DigitalOcean and URL-embedded-credential patterns. Recall goes from 20/31 to 30/31, with false positives unchanged at 1/34 (the public AWS documentation example key).kill 0/kill -1, docker volume and system prune with--volumes/-a(routinedocker rm/rmistays allowed),--force-with-lease=<ref>to main, comment-splitDROP/**/TABLE, IFS and brace tricks, and interpreter-wrapped commands. Blocked goes from 29/46 to 46/46, with 0/45 false positives. A timing test bounds the dynamic-command regex at under 50 ms on a 25 KB input.Caveat: the fixes post-date the corpora, so the after-numbers are in-sample. The held-out variants in
scripts/test_security_gates_corpus.pyare the generalization check. The guard is best-effort string classification, not a security boundary.An independent review found one 🔴 (
kill 0allowed) and five 🟡 issues (docker false positives, the docker-Hbypass,--force-with-lease=bypass, the SQL comment split, regex backtracking). All are fixed in this commit.Local CI repro: score.py 30/31 and 46/46; every
scripts/ci-tests.dhook passes;test-ci-gates.sh295 passed and 9 failed, the known sandbox-only failures.🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z