Skip to content

Close the generalizable held-out security-gate misses and score them out of sample - #1989

Merged
remigiusz-antczak merged 1 commit into
mainfrom
land/gatefix2
Oct 25, 2026
Merged

remigiusz-antczak merged 1 commit into
mainfrom
land/gatefix2

Conversation

@remigiusz-antczak

Copy link
Copy Markdown
Collaborator

This adds rules for the held-out misses from #1988 that generalize, and scores them on a second blind corpus that was frozen before the rules were written.

Rule changes:

  • .banlist.txt: provider token formats.
  • destructive_guard.py: cloud, Kubernetes, Terraform, NoSQL, disk-tool and base64-eval rules.

Results on the second held-out corpus (docs/bench/security-gates-heldout2-1024/). The corpus was preregistered in 8abdefc93 before these rules existed, and no rule changed after scoring:

Gate Perun Rivals (ported rules)
Secret leaks found (40) / decoys flagged (40) 23 / 2 gitleaks v8.30.1: 26 / 0
Dangerous commands blocked (50) / benign blocked (50) 37 / 4 claude-code-safety-net subset: 18 / 2; hooks-mastery: 9 / 5
  • Out of sample, Perun still trails gitleaks on secrets (23 against 26). Combining the two would miss only 7 of 40. The next change adds the gitleaks rules to Perun's secret gate and scores the combination on a third preregistered corpus.
  • Perun leads on destructive commands (37 against 18 and 9). The remaining misses are listed in RESULTS.md as leads for a later rule wave, scored on a fresh corpus. They are: bare SQL, redirects into system files, git filter-branch, some cloud and Helm deletes, printf-hex and eval $(echo …) obfuscation, redis-cli reading FLUSHALL from stdin, and aws s3 sync --delete.
  • The first held-out corpus is now in-sample. Its 30/30 and 40/40 are labelled that way, and docs/perun-vs-rivals.md keeps its blind numbers as "as scored blind". The original corpus is unchanged.

Tooling:

  • score.py accepts id-prefixed rows and checks hashes against SHA256-frozen.txt.
  • bench_check.py resolves a bare frozen filename from the bench directory first.
  • The privacy gate skips exactly four frozen corpus files by exact repo-relative path. A test shows that a key placed in any other security-gates-* file still fails.

The independent review found 2 🔴, both fixed, and a re-review checked the fixes:

  • The first version of the privacy skip used a glob that crossed directories.
  • The base64-eval rule, and the command-substitution scan under it, could be driven into polynomial and exponential backtracking. The fix is linear, and a regression test enforces a time bound.

Four 🟡 were also folded in: a kubectl delete -n <ns> ns <name> bypass, the in-sample row label, the freeze-order wording, and the known gaps. Re-scoring after these fixes gives identical numbers.

GitHub push protection blocked one placeholder decoy key, row D2-08. Its @@ marker is recorded as an amendment in ledger.jsonl and SHA256-frozen.txt. The pin test checks that the first five lines still hash to the preregistered value, and the @@-stripped content and every score are unchanged.

Local CI repro:

  • Security tests, size-ratchet and privacy are green.
  • test-ci-gates.sh gives 297 passed and 9 failed. The same 9 fail on the parent commit (sandbox ps and network). main had 11 failures before.

🤖 Generated with Claude Code

https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z

…ctive guard, and score them out of sample

Secret gate: provider-format patterns for the AWS secret access key (docs example stays allowed), Twilio SID and auth token, JWT, Azure AccountKey/SharedAccessKey, Discord bot token, Linear, Mailgun, Docker Hub PAT and Telegram bot token. Destructive guard: aws s3 rm --recursive / rb --force, ec2 terminate-instances, rds delete-db-*, gcloud projects delete, az group delete, terraform/tofu destroy, kubectl delete namespace / all --all, redis FLUSHALL/FLUSHDB, mongo dropDatabase, wipefs, diskutil erase*, and eval/sh of decoded base64.

Held-out-1024 before to after: secret 21/30 to 30/30 (FP 2/30), guard 26/40 to 40/40 (FP 3/40); it is now in-sample. First corpus stays 30/31 FP 1/34 and 46/46 FP 0/45. Out of sample on the preregistered held-out-2 corpus (rules frozen): secrets Perun 23/40 FP 2/40 vs ported gitleaks 26/40 FP 0; guard Perun 37/50 FP 4/50 vs ported safety-net 18/50 FP 2 and hooks-mastery 9/50 FP 5. Perun still trails gitleaks on secrets.

The landed held-out-1024 corpus is immutable, so the new patterns' hits on its unmarked rows (and a pre-existing hit in held-out-2 decoys) are handled by a narrow privacy-gate skip for the frozen security-gates secret-leaks.txt and secret-decoys.txt files, tested in both directions. score.py accepts id<TAB>cmd rows and hashes from SHA256-frozen.txt; bench_check.py resolves a bare frozen filename from the bench dir first; the held-out-1024 pin test cross-checks ledger amendments against the frozen list; the README notes the ledger was reconstructed after the fact.

Tests: scripts/test_security_gates_heldout_fixes.py (35+15 caught, 25+11 allowed, exemption both ways), heldout, heldout2 and corpus pins pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
@remigiusz-antczak
remigiusz-antczak merged commit e42fed4 into main Oct 25, 2026
1 check passed
@remigiusz-antczak
remigiusz-antczak deleted the land/gatefix2 branch October 25, 2026 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant