Repository navigation
Bench: rival head-to-head on the security-gates corpus (ported rules) - #1986
Merged
Merged
Conversation
…les ported to Python Port gitleaks v8.30.1 default rules (keywords, entropy, allowlists), the hooks-mastery rm and .env regexes, and a documented subset of claude-code-safety-net, and score them on the 2026-10-24 corpus. Rival sources are pinned by tag or commit with sha256 in rivals/SOURCES.md; nothing downloaded is executed. Perun vs ported rivals (in-sample, own corpus): - secrets: Perun 30/31 recall, 1/34 FP; gitleaks 22/31, 0/34 - destructive: Perun 46/46, 0/45 FP; safety-net subset 27/46 (at most 31, 4 rows unported), 0/45; hooks-mastery 13/46, 5/45 Caveat recorded in the README and perun-vs-rivals rows: rules ported to Python, not the rival binary; ported-rule fidelity is the main threat. scripts/test_rival_h2h.py pins the scores and runs from scripts/ci-tests.d/rival-h2h.sh. Gates: size-ratchet ok, mustload ok, privacy clean, rival-h2h test ok. test-ci-gates.sh 293 passed, 11 failed (baseline 295/9; sandbox-environment failures, run-ledger lsof and bench-check timeout seen). Tests: 293/304 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The security-gates corpus from #1985 had Perun's numbers but no rival numbers, because gitleaks and the two guard projects are not installed here. This change scores the rivals on the same frozen corpus by porting their published rules to Python.
docs/bench/security-gates-1024/rivals/: the rival rule sources are vendored as data at pinned versions with sha256 and URLs (SOURCES.md);score_rivals.pyports them and is never given the downloaded code to execute. The gitleaks port keeps the keyword prefilter, entropy thresholds, secret groups, and rule and global allowlists.scripts/test_rival_h2h.pyplus aci-tests.dhook pin the scores, the rule count (221 gitleaks rules, 0 compile failures) and the vendored hashes.Caveats, also in the README and the rows of
docs/perun-vs-rivals.md:rm -rfand.envaccess by design, so 13/46 is not a general quality comparison.An independent review found 0 🔴 and 4 🟡, all wording, all folded in: drop an invalid upper bound, list unported families, put the asymmetry caveat in the rows, add the hooks-mastery scope note.
Local CI repro: size-ratchet and mustload ok, privacy grep clean,
git diff --checkclean,test_rival_h2hok.test-ci-gates.shgives 293 passed and 11 failed on this branch and on main (d68fb67) alike, with identical failure lists, so the branch adds no failures.🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z