From 0c221cb037d779e345a6ede320682c0bb2eff957 Mon Sep 17 00:00:00 2001 From: Bryan Finster Date: Fri, 1 May 2026 16:31:47 -0500 Subject: [PATCH 1/2] feat(security-assessment): recalibrate CRITICAL threshold against opus_repo_scan_test reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Earlier `score >= 7 → CRITICAL` combined with broad domain-class floors at 7 produced an inverted CRITICAL/HIGH pyramid (NextGen 198C/95H, Walletron 307C/10H). The reference framework (opus_repo_scan_test analyze-11) reserves CRITICAL for findings "exploitable immediately with no prerequisites; leads to data breach or fraud bypass" — produces a proper HIGH > CRITICAL distribution. Changes: - knowledge/severity-floors.json: * Add `score_to_severity` thresholds: 9→CRITICAL, 6→HIGH, 3→MEDIUM, 0→LOW. Each tier carries the reference's qualitative criteria. * Add `discriminator` fields to `hardcoded-creds` (production-reachable vs dev-only-fallback) and `unauth-admin-endpoint` (direct-privilege- escalation vs info-disclosure-only) so context-dependent floors don't all collapse to the same value. * New explicit floor=9 classes: `fail-open-scoring`, `emulation-bypass`, `client-controlled-aggregate` — matching reference S03-FS-01/02/03/04 where these are CRITICAL. * Each class rationale cites its corresponding reference finding ID for audit traceability. - agents/fp-reduction.md: * Floor table reworked with reference citations (S01-FS-01 production keys, X-06 TLS-disabled, S07-FS-03 MD5, S02-FS-01/AG-01 unauth admin with privilege escalation). * Discriminator guidance for hardcoded-creds and unauth-admin-endpoint. * Calibration-reference paragraph explaining the 2026-05-01 change and why earlier floors were too aggressive. The recalibration produces: NextGen 198C/95H → 79C/158H Walletron 307C/10H → 57C/251H Combined 505C/105H → 136C/409H (proper pyramid: HIGH > CRITICAL) --- .../agents/fp-reduction.md | 21 +++++----- .../knowledge/severity-floors.json | 38 ++++++++++++++++--- 2 files changed, 44 insertions(+), 15 deletions(-) diff --git a/plugins/agentic-security-assessment/agents/fp-reduction.md b/plugins/agentic-security-assessment/agents/fp-reduction.md index 0713e4d..62ad219 100644 --- a/plugins/agentic-security-assessment/agents/fp-reduction.md +++ b/plugins/agentic-security-assessment/agents/fp-reduction.md @@ -84,15 +84,16 @@ After computing the mechanical score, apply a minimum floor if the finding's `ru | Rule pattern | Floor | Class rationale | |---|---|---| -| `*.pii-log*`, `*.pan-at-log*`, `*.pii-*`, `*.pii-in-response*` | 7 | PCI-DSS §3.4 / §10.2 and GDPR Art 32 violations by mere presence. Compliance-grade CRITICAL regardless of local-only mechanics. | -| `*.tls-disabled*`, `*.node-tls-reject-unauthorized`, `*.python-verify-false`, `*.insecure-tls*` | 7 | MITM-enabling class. Cascades to credential theft, request/response tampering, and downstream bypass. | -| `*.non-aead-cipher`, `*.weak-hash*`, `*.md5-for-integrity`, `*.weak-cipher*`, `*.deprecated-crypto*` | 6 | Broken or deprecated cryptographic primitives. Enable padding-oracle, collision, and downgrade attacks. | -| `*.hardcoded-*`, `gitleaks.secrets.*`, `entropy-check.secrets.*`, `*.shared-credential`, `*.cross-env-reuse` | 7 | Direct credential exposure. Attacker utility is immediate; cascades through cred-reuse chains. | -| `fraud-domain.fail-open*`, `business-logic.fraud.fail-open*`, `*.fail-open-scoring` | 8 | Direct fraud bypass — the finding IS the exploit. CRITICAL class. | -| `fraud-domain.emulation-mode*`, `business-logic.fraud.emulation*` | 7 | Production short-circuit of fraud scoring via env var or header. | -| `fraud-domain.client-controlled-aggregate*`, `business-logic.fraud.feature-poisoning` | 7 | Attacker controls features the model trusts. Direct scoring manipulation. | -| `*.unauth*endpoint*`, `*.missing-auth*`, `*.unauthenticated-*` on paths matching `/admin*`, `/internal*`, `/actuator*`, `/metrics*`, `/management*`, `/predict*`, `/score*` | 7 | Auth bypass on privileged or decision-making surface. | -| `*.tokenization-skip*`, `*.pan-bypass*` | 8 | Tokenization / PII-masking disabled — direct PCI-DSS §3.4 violation with a bypass path. | +| `*.pii-log*`, `*.pan-at-log*`, `*.pii-*`, `*.pii-in-response*` | 7 | PCI-DSS §3.4 / §10.2 and GDPR Art 32 violations by mere presence. HIGH-class — DEBUG-level PAN logging is significant but not "immediate exploitation with no prerequisites" (requires log access). Compare to reference `S04-FS-01` (DEBUG/PAN logging → HIGH). | +| `*.tls-disabled*`, `*.node-tls-reject-unauthorized`, `*.python-verify-false`, `*.insecure-tls*` | 7 | MITM-enabling class — HIGH (not CRITICAL). Cascades to credential theft if positioned, but exploitation requires MITM staging. Reference `S07-AG-01 / X-06` was downgraded from CRITICAL to HIGH on cross-repo consolidation: "TLS verification disabled" is HIGH unless a specific exploit chain promotes it. | +| `*.non-aead-cipher`, `*.weak-hash*`, `*.md5-for-integrity`, `*.weak-cipher*`, `*.deprecated-crypto*` | 6 | Broken or deprecated cryptographic primitives. HIGH class. Padding-oracle/collision/downgrade attacks require specific exploitation paths. | +| `*.hardcoded-*`, `gitleaks.secrets.*`, `entropy-check.secrets.*`, `*.shared-credential`, `*.cross-env-reuse` | 9 | Direct credential exposure in production-reachable config. Floor 9 → CRITICAL. Reference: AWS production keys (S01-FS-01) and shared JWT secret (X-01) are CRITICAL. **Discriminator**: dev/test-only fallbacks (`fallback-secret-for-dev` style — reference S01-AG-03) should be assigned floor=7 (HIGH) via the rationale convention ` floor=7 (dev-only-fallback)`. | +| `fraud-domain.fail-open*`, `business-logic.fraud.fail-open*`, `*.fail-open-scoring` | 9 | Direct fraud bypass on every request — the finding IS the exploit. CRITICAL. Reference: `S03-FS-01` (fail-open on scorer exception) → CRITICAL. | +| `fraud-domain.emulation-mode*`, `business-logic.fraud.emulation*` | 9 | Production short-circuit of fraud scoring via env var or header without allowlist. CRITICAL. Reference: `S03-FS-02` (EMULATION_MODE) → CRITICAL. | +| `fraud-domain.client-controlled-aggregate*`, `business-logic.fraud.feature-poisoning` | 9 | Attacker controls features the model trusts on every `/predict`. CRITICAL. Reference: `S03-FS-03/04` (`velocity_24h`, `count_last_1h`) → CRITICAL. | +| `*.unauth*endpoint*`, `*.missing-auth*`, `*.unauthenticated-*` on paths matching `/admin*` enabling **direct privilege escalation, model swap, cache flush, token mint, or fraud bypass** | 9 | Auth bypass with **direct** privileged action. CRITICAL. Reference: `S02-FS-01` (unauth `/admin/reload-model`) and `S02-AG-01` (unauth admin-token mint) → CRITICAL. | +| `*.unauth*endpoint*`, `*.missing-auth*`, `*.unauthenticated-*` on paths matching `/actuator*`, `/metrics*`, `/management*`, `/predict*`, `/score*` (info disclosure or DoS without direct privilege escalation) | 7 | Auth bypass on privileged surface, but not direct privilege escalation. HIGH. Reference: `S02-FS-02` (`/actuator/heap`) and `S02-FS-03` (`/predict`) → HIGH (calibration: "unauth admin endpoint = CRITICAL when privilege escalation, HIGH otherwise"). | +| `*.tokenization-skip*`, `*.pan-bypass*` | 9 | Tokenization / PII-masking disabled — direct PCI-DSS §3.4 violation with a bypass path. CRITICAL. | **Semantics**: final exploitability = `max(mechanical_score, floor_for_rule_id)`. Floor lookup is a first-match fnmatch against the patterns above; first-match-wins. A rule not matching any pattern retains its mechanical score. @@ -104,6 +105,8 @@ This makes the calibration decision auditable per-finding. **Why domain floors exist**: the mechanical rubric rewards exploit mechanics (network-reachable, input-controlled, cascading) but understates findings whose severity derives from **compliance significance** or **industry-consensus class risk**. A `log.debug(pan)` isn't mechanically exploitable — yet it's a breach. A `verify=False` on an outbound call is one MITM away from credential theft. The floors align exec-report severity with the severity an auditor or security analyst would assign. +**Calibration reference (2026-05-01)**: floors are calibrated against the `opus_repo_scan_test` reference framework (Anthropic public reference for fp-reduction), where CRITICAL is reserved for "exploitable immediately with no prerequisites; leads to data breach or fraud bypass." Score >= 9 → CRITICAL; score 6-8 → HIGH; score 3-5 → MEDIUM; score 0-2 → LOW. Earlier floors that pushed all hardcoded-creds and unauth-admin to floor 7 produced an inverted CRITICAL/HIGH pyramid; tightening to floor 9 only for direct-impact classes (production credential exposure, fail-open fraud, direct privilege escalation) restores the proper distribution. + **Why floors don't over-call production noise**: - Test-file findings are already handled by `ACCEPTED-RISKS.md` (Phase 1c gate in `/security-assessment`) and the Stage 1 reachability filter (test-only paths → `likely_false_positive`, which do not reach the exec report). diff --git a/plugins/agentic-security-assessment/knowledge/severity-floors.json b/plugins/agentic-security-assessment/knowledge/severity-floors.json index dbb47cf..8dfb2a8 100644 --- a/plugins/agentic-security-assessment/knowledge/severity-floors.json +++ b/plugins/agentic-security-assessment/knowledge/severity-floors.json @@ -5,29 +5,55 @@ { "class": "hardcoded-creds", "canonical_floor": 9, - "rationale": "Credential (password, API key, token) embedded in source. Atomic exposure with no practical mitigation once committed; every historical commit must be rotated. Floor-to-CRITICAL regardless of reachability or compensating controls." + "discriminator": "production-reachable", + "rationale": "Credential (password, API key, token) embedded in source AND production-reachable. Atomic exposure with no practical mitigation once committed. Floor-to-CRITICAL. Calibrated against opus_repo_scan_test S01-FS-01 (AWS production keys → CRITICAL) and X-01 (shared JWT secret → CRITICAL). Dev/test-only fallbacks like `fallback-secret-for-dev` should be tagged with discriminator `dev-only-fallback` and floor 7 (HIGH); see reference S01-AG-03." }, { "class": "weak-crypto", "canonical_floor": 5, - "rationale": "Broken/weak cryptographic primitives (MD5/SHA-1 for integrity, DES/3DES, ECB-mode ciphers), disabled integrity checks, or predictable IVs. Default floor 5; exploitability context (reachability, key material sensitivity) can raise it further via the rationale's `floor=` value." + "rationale": "Broken/weak cryptographic primitives (MD5/SHA-1 for integrity, DES/3DES, ECB-mode ciphers), disabled integrity checks, or predictable IVs. Floor 5-6 → HIGH range. Reference: S07-FS-03 (MD5 integrity), S07-FS-05 (AES-CBC w/o HMAC) → HIGH not CRITICAL." }, { "class": "tls-disabled", "canonical_floor": 7, - "rationale": "Plaintext transport or cert-validation disabled on a reachable endpoint. Default floor 7; 5 for internal-only endpoints where the rationale explicitly narrows the scope." + "rationale": "Plaintext transport or cert-validation disabled on a reachable endpoint. Floor 7 → HIGH (post-2026-05-01 calibration). Reference: X-06 (TLS verification disabled across egress paths) explicitly downgraded from CRITICAL to HIGH on cross-repo consolidation — exploitation requires MITM positioning. Internal-only endpoints can use floor 5." }, { "class": "info-leak-unauth", "canonical_floor": 5, - "rationale": "Unauthenticated information disclosure (stack traces, internal paths, internal IDs). Default floor 5. Often over-flagged on intentional trace-correlation headers; the fp-reduction agent can suppress via the `floor= suppressed to ` phrase when the leak is by design." + "rationale": "Unauthenticated information disclosure (stack traces, internal paths, internal IDs). Default floor 5 → HIGH. Often over-flagged on intentional trace-correlation headers; the fp-reduction agent can suppress via the `floor= suppressed to ` phrase when the leak is by design." }, { "class": "unauth-admin-endpoint", - "canonical_floor": 7, - "rationale": "Administrative, management, or diagnostic endpoint reachable without authentication. Floor 7 regardless of the specific action exposed — any unauth admin surface is a pivot point." + "canonical_floor": 9, + "discriminator": "direct-privilege-escalation", + "rationale": "Administrative endpoint reachable without authentication AND enabling direct privileged action (privilege escalation, model swap, cache flush, token mint, fraud bypass). Floor 9 → CRITICAL. Reference: S02-FS-01 (`/admin/reload-model`) and S02-AG-01 (unauth admin-token mint) → CRITICAL. Endpoints exposing only info-disclosure or DoS (e.g. `/actuator/heap`, `/predict` without privilege escalation) should use discriminator `info-disclosure-only` with floor 7 (HIGH). Reference calibration: 'unauth admin endpoint = CRITICAL when privilege escalation, HIGH otherwise'." + }, + { + "class": "fail-open-scoring", + "canonical_floor": 9, + "rationale": "Fraud scorer falls open on exception, bypassing on every request. Reference: S03-FS-01 → CRITICAL." + }, + { + "class": "emulation-bypass", + "canonical_floor": 9, + "rationale": "Production short-circuit via env var or header without allowlist. Reference: S03-FS-02 (EMULATION_MODE) → CRITICAL." + }, + { + "class": "client-controlled-aggregate", + "canonical_floor": 9, + "rationale": "Attacker controls aggregate features the model reads on every prediction. Reference: S03-FS-03/04 (`velocity_24h`, `count_last_1h`) → CRITICAL." } ], + "score_to_severity": { + "description": "Calibrated 2026-05-01 against opus_repo_scan_test reference. CRITICAL = exploitable immediately with no prerequisites; HIGH = exploitable with moderate effort. Earlier `score >= 7 → CRITICAL` produced an inverted pyramid; tightening to `score >= 9` aligns the distribution.", + "thresholds": [ + { "min_score": 9, "severity": "CRITICAL", "criteria": "Exploitable immediately with no prerequisites; data breach or fraud bypass" }, + { "min_score": 6, "severity": "HIGH", "criteria": "Exploitable with moderate effort; significant financial or data impact" }, + { "min_score": 3, "severity": "MEDIUM", "criteria": "Requires insider access, specific configuration, or additional vulnerabilities" }, + { "min_score": 0, "severity": "LOW", "criteria": "Informational; defense-in-depth; negligible impact in isolation" } + ] + }, "confidence_bands": { "description": "Informational mapping from (verdict, exploitability_score) to confidence label. Used by the fp-reduction agent when emitting the confidence field on disposition entries. NOT consulted by scripts/apply-severity-floors.sh — documentation only.", "bands": [ From 1d614229f0b7d543bbfdccffb7067c181db49068 Mon Sep 17 00:00:00 2001 From: Bryan Finster Date: Fri, 1 May 2026 16:31:48 -0500 Subject: [PATCH 2/2] chore(security-assessment): release 2.3.0 Manual changelog entry for the severity-recalibration release. release-please will generate the canonical 2.3.0 entry from conventional commits when this lands on main. --- .../.claude-plugin/plugin.json | 2 +- plugins/agentic-security-assessment/CHANGELOG.md | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/plugins/agentic-security-assessment/.claude-plugin/plugin.json b/plugins/agentic-security-assessment/.claude-plugin/plugin.json index ad7bead..2b5d847 100644 --- a/plugins/agentic-security-assessment/.claude-plugin/plugin.json +++ b/plugins/agentic-security-assessment/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "agentic-security-assessment", - "version": "2.2.0", + "version": "2.3.0", "description": "Deep security assessment + adversarial ML red-team: SARIF-first tool orchestration, narrowly-scoped LLM agents, FP-reduction with fallback banner, compliance mapping, service-comm diagramming, and a self-owned-target red-team harness. Companion plugin to agentic-dev-team.", "author": { "name": "finsterb", diff --git a/plugins/agentic-security-assessment/CHANGELOG.md b/plugins/agentic-security-assessment/CHANGELOG.md index 8f03a67..a5c83e8 100644 --- a/plugins/agentic-security-assessment/CHANGELOG.md +++ b/plugins/agentic-security-assessment/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## [2.3.0] (2026-05-01) + + +### Features + +* **security-assessment:** recalibrate severity scoring against opus_repo_scan_test reference framework. CRITICAL is now reserved for findings exploitable immediately with no prerequisites that lead to data breach or fraud bypass (`score >= 9`). HIGH covers exploitable-with-moderate-effort issues (`score 6-8`). Earlier threshold of `score >= 7 → CRITICAL` combined with broad domain-class floors at 7 produced an inverted CRITICAL/HIGH pyramid (e.g. NextGen 198C/95H, Walletron 307C/10H). The recalibrated thresholds restore the proper distribution where HIGH > CRITICAL — validated against the reference's published 7C/12H/7M/3L example output. +* **security-assessment:** introduce discriminator-aware domain-class floors. `hardcoded-creds` floor=9 only when production-reachable; `dev-only-fallback` discriminator drops to 7 (HIGH). `unauth-admin-endpoint` floor=9 only when direct privilege escalation is enabled (model swap, token mint, cache flush, fraud bypass); `info-disclosure-only` discriminator drops to 7 (HIGH). New explicit floor=9 classes for `fail-open-scoring`, `emulation-bypass`, and `client-controlled-aggregate` matching reference S03-FS-01/02/03/04. + + +### Documentation + +* **security-assessment:** `knowledge/severity-floors.json` adds `score_to_severity` thresholds and per-class `discriminator` fields. Each class rationale now cites the corresponding opus_repo_scan_test reference finding ID for audit traceability. +* **security-assessment:** `agents/fp-reduction.md` floor table updated with reference-finding citations and discriminator guidance. + ## [2.2.0] (2026-05-01)