Repository navigation
Close blind held-out-4 guard misses and false positives; score on a second blind corpus - #1990
Merged
Merged
Conversation
… 4 and 5 Adds the blind, preregistered held-out-4 and held-out-5 destructive-command corpora (60 dangerous, 60 benign each). Held-out-4 blind score on main: guard 35/60 blocked, 9/60 benign blocked; ported safety-net subset 14/60 (FP 1), hooks-mastery 7/60 (FP 4). Held-out-5 (rules frozen first): guard 50/60 FP 4/60, main before the fixes 41/60 FP 6/60, safety-net 20/60, hooks-mastery 8/60. destructive_guard.py now allows usage flags, aws/kubectl/helm --dry-run forms and rm/find under a named $TMPDIR or /tmp child, and blocks package unpublish/yank/purge, cloud and cluster deletes, terraform state rm, parted mklabel, hex/printf decode-eval, alias and one-step variable indirection, redirects over system files and git filter-branch. A --help after other arguments of rm/chmod/chown/dd/shred/mkfs/diskutil is still checked. The decode-eval scan is windowed and capped, with timing tests. The held-out-4 in-sample re-score is 60/60 with 0/60 FP and is labelled in-sample in RESULTS.md and docs/perun-vs-rivals.md, with the blind row kept. score.py tolerates absent secret corpus files; bench_check accepts the sha256_frozen_txt ledger key. Held-out-1024 and held-out-2 pins raised upward only, with notes. Tests: all security gates, test-destructive-guard 16/16, size-ratchet, bench_check green Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This closes the destructive-guard misses and false positives found on a blind corpus, and checks the result on a second blind corpus frozen before the rules were written.
prereg/heldout4(e8dd4d4) andprereg/heldout5(384ae0d). Ledger line 1 of each is byte-identical to its prereg commit.docker exec,kubectl exec -- rm …) is the largest gap.Rule changes in
destructive_guard.py:--dry-runforms for aws, kubectl and helm (judged on the last dry-run token), and rm/find under a named child of$TMPDIRor/tmp.terraform state rm;parted mklabel; hex and printf decode-then-execute; alias and one-step variable indirection; truncating redirects onto system files;git filter-branch.The independent review found 2 🔴, both fixed, and a re-review checked the fixes with probes:
--helpanywhere in the arguments exempted the whole command. On macOS,rm -rf ~ --helpruns for real. The exemption now never applies to rm, chmod, chown, dd, shred, mkfs or diskutil, including behind sudo, env, xargs, nohup or timeout.Of the 7 🟡 from the first review:
sh $(xxdrepeated. They are now a windowed, capped scan with timing tests; 10k-character adversarial inputs finish in about 0.1 s.//and>|redirects, apt purge scope, the/tmpresiduals documented, the ledger key, and the RESULTS citation.The re-review's 0 🔴 / 3 🟡 are folded into the next guard wave:
>>onto system files.openssl encdecode shapes.Local gates:
test_security_gates*.py: all pass. The held-out-4 and held-out-5 pins are new; held-out-1024 and held-out-2 pins were raised only.test-destructive-guard.sh: 16/16. bench_check tests: 9/9 and 2/2.bench_check --changed: 2 dirs clean. Size-ratchet: ok. Privacy on changed files: clean.git diff --check: clean.🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z