diff --git a/.github/workflows/creusot-gate.yml b/.github/workflows/creusot-gate.yml new file mode 100644 index 00000000..6d20f8fd --- /dev/null +++ b/.github/workflows/creusot-gate.yml @@ -0,0 +1,96 @@ +# SPDX-License-Identifier: MPL-2.0 +name: Creusot Gate + +# Rust deductive verification, as the estate requires it: the Creusot gate in +# hyperpolymath/cicd-suite, called rather than copied. This file adds no +# steps of its own, exactly like main-estate-audit.yml — a copy would freeze +# this repository at whatever cicd-suite commit it happened to hold. +# +# Why this file exists at all: the estate audit's own `languages` job runs the +# Idris2, Zig, proof-runner and BoJ checks, not this one, so before this file +# nothing in this repository watched for a Creusot claim. The gate's checks are +# therefore new coverage here, not a duplicate of the audit's. +# +# KNOWN LIMIT OF THE PINNED GATE (reviewed, upstream fix required): the pinned +# script recognises a Creusot dependency by its literal `creusot-std` / +# `creusot-contracts` key. A crate that renames the dependency +# (`creusot = { package = "creusot-std" }`) and writes contracts would set +# `obligations=true` while leaving the selected-crate list empty, so the proof +# loop could run over nothing and still exit green. That is a gap in the gate, +# not a pass for this tree, and it cannot be fixed from here without copying +# the gate (which is what this repository refuses to do). It is recorded so the +# first aliased dependency arrives with someone looking; the honest fix is a +# cicd-suite release that resolves the alias and rejects an empty crate list +# when obligations exist. +# +# WHAT THIS GATE DOES, AND WHAT IT DOES NOT CLAIM +# ----------------------------------------------- +# Three phases, run by the reusable: +# +# detect does any crate carry a Creusot obligation (a creusot-std / +# creusot-contracts dependency, or a #[requires]/#[ensures] and +# friends)? does any prose or source header CLAIM Creusot +# verification? +# theatre T1: a claim with no obligations behind it is a hard failure — +# "verified with Creusot" without contracts is proof theatre. +# T2: a creusot-std dependency that annotates nothing is named. +# T3: every #[trusted] axiom is listed, because a proof is only as +# strong as the set it trusts. +# prove only runs when detect found obligations. It installs Creusot at +# a pinned release, proves a planted FALSE contract and REQUIRES +# the prover to refuse it, proves a planted TRUE one and requires +# success, and only then runs `cargo creusot prove` in every crate +# with obligations. A prover that cannot say no is not a prover. +# +# MEASURED ON THIS TREE (2026-10-28, cicd-suite bf7dd95 creusot-check): +# obligations=false claims=false contract_count=0 trusted_count=0 +# so the theatre phase reports, in its own words, "no Creusot obligations and +# no verification claims in this tree. Nothing was proved and nothing claims +# to be — this gate is silent, not green-as-assurance." +# +# That is the honest state, and it is the state PROOF-PROGRESS.adoc records: +# Pons has no machine-checked soundness proof. The gate exists here so that +# the first time anyone claims, in a README, a module header or a doc, that +# this code has been proved by Creusot — with no contracts behind it — the +# build goes red (T1) instead of the claim shipping. (The gate reads that +# claim phrase out of prose lines, so even quoting it here would be a mistake +# if workflows were scanned; this comment deliberately paraphrases.) It is a guard against theatre today and the +# wiring for real obligations tomorrow: the day a crate depends on +# creusot-std, detect flips obligations to true, prove runs, and the tree has +# to discharge its contracts or fail. +# +# A first obligation therefore needs, together: a creusot-std dependency, the +# nightly pin Creusot is built against (the reusable's `rust_nightly` input), +# and a crate whose contracts `cargo creusot prove` actually discharges. There +# is no way to have the third without the toolchain, and no honest way to +# claim it before it runs. +# +# Pin: the same cicd-suite commit main-estate-audit.yml uses (bf7dd95), so the +# audit's `languages` component and this job run the same gate revision. +# DE-ONBOARDED from actions.lock deliberately, like the secret-scanner and +# estate-audit callers: a reusable onboarded cross-repo startup-fails with zero +# jobs under sha_pinning_required. Do not run a full-directory `gh actions-lock` +# that re-onboards this file. + +on: + push: + branches: [main] + pull_request: + branches: [main] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + creusot: + uses: hyperpolymath/cicd-suite/.github/workflows/creusot-gate.yml@bf7dd95cf49733019343ff485c2f45c6b261839c # cicd-suite main + # A creusot-std dependency that annotates nothing is a hollow dependency: + # this repository adds one only when it also writes the contract, so the + # warning is escalated to a failure. It costs nothing today — there is no + # such dependency — and it means the first one cannot arrive alone. + with: + enforce_hollow: true diff --git a/ARCHITECTURE.adoc b/ARCHITECTURE.adoc index 1056a5c9..f9ae3986 100644 --- a/ARCHITECTURE.adoc +++ b/ARCHITECTURE.adoc @@ -95,8 +95,9 @@ Components marked *planned* carry the milestone that delivers them; see * Inline comment allowances; strings cannot suppress findings * Suppressed findings are counted rather than silently erased . *Reporter* — *built* -* Human-readable output with SPECULATIVE visually demoted -* JSON 0.3.0 envelope with skip, transcode and suppression accounting +* Human-readable output with SPECULATIVE visually *and* numerically demoted +* JSON 0.5.0 envelope (0.4.0 frozen and still validatable) with skip, transcode, + suppression, demoted-count and corroboration-key accounting * SARIF 2.1.0 with invocation notifications and honest artifact offsets * Generated catalogue and man page from the live registry/CLI model diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index a029a39b..ae8a1e18 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -3,6 +3,76 @@ == Unreleased — 0.1.0-dev.1 +=== Added + +* The three things a repinning consumer needs are now published, and each one is + a fact the code checks rather than a promise the docs make. + + **The report contract is versioned, and 0.4.0 is frozen.** The envelope moves + to 0.5.0 with three appended keys: `findings[].semantic_fingerprint`, + `counts.actionable` and `counts.demoted_speculative`. Additive in both + directions, and the bump is still taken, because a consumer has to be able to + tell whether a report carries the corroboration key at all. `0.4.0` does not + disappear: `schemas/scan-0.4.0.json` is byte-pinned by + `FROZEN_0_4_0_SCHEMA_BLAKE3` (a test recomputes the digest, so editing the + frozen file fails the build instead of a consumer), and `pons + validate-report` still accepts a 0.4.0 report — the version *declared inside + the document* selects the schema, so a 0.4.0 declaration cannot carry a 0.5.0 + key. `schemas/index.json` lists every published artifact with its version, + status and BLAKE3 digest, and `schemas/README.adoc` states the freeze rule and + how to add a version. + + **The severity × evidence matrix is published as data.** + `schemas/verdict-matrix-1.0.0.json` carries all twelve (severity, evidence) + cells with `sarif_level`, `aggregate_weight`, `counts_toward_verdict`, + `gate_eligible`, `count_bucket` and `demotion_suffix`, plus both scales and the + corroboration recipe. It is generated from `crates/pons-core/src/verdict.rs`, + which is now the single home of the SARIF level and rank tables, the + evidence-contract confidence labels and the `--fail-on` rank, so the published + document cannot state a different number from the reporter; a drift test fails + if the committed file and the code disagree. + + **Every finding carries a semantic fingerprint.** + `findings[].semantic_fingerprint` is BLAKE3 over + `pons-corroboration-v1`, the root-relative path, the 1-based start line and + the evidence class — a *site*, not a finding, so a grid can mark Pons's + finding and a panic-attack or Hypatia finding at the same line as one + corroborated site instead of double-counting the overlap. It deliberately + ignores the rule id, the producer and the severity. The record fingerprint is + unchanged and still does the other job (one finding, rule id plus a + whitespace-folded anchor). `validate-report` recomputes the key rather than + checking its shape, so a producer cannot point a finding at an arbitrary site. + + **`SPECULATIVE` is demoted numerically as well as visually.** The demoted + bucket has an aggregate weight of **zero** at every severity, `counts.actionable` + excludes it, and `validate-report` rejects a document whose + `actionable + demoted_speculative` does not account for `counts.total` — the + demotion is a number in the contract, not a convention a reader might miss. + The human summary gains a `verdict:` line separating the two. Appendix E's + SARIF `rank` of 20 is kept, on purpose and published as a *different* number: + a guess stays visible and adds nothing. + +* Tests for all of it, including the ones that make the claims non-vacuous: a + frozen 0.4.0 report still validates through the binary, a 0.4.0 declaration + carrying a 0.5.0 key is rejected, an unknown version is refused rather than + validated against the nearest shape, a forged corroboration key is rejected, + and the frozen schema digest is recomputed from disk. + +=== Added (verification lane) + +* The estate's Creusot gate is now called by this repository + (`.github/workflows/creusot-gate.yml`) at the same cicd-suite pin as the + estate audit, instead of only arriving inside the audit's `languages` + component. Nothing is proved by this and nothing claims to be: measured with + the gate's own `detect` and `theatre` phases on this tree, `obligations=false + claims=false contract_count=0 trusted_count=0`, and the gate's verdict is its + own words — "nothing was proved and nothing claims to be". What the lane buys + is the negative: a claim that this code is proved, with no contracts behind + it, is now a build failure (T1), a `creusot-std` dependency that annotates nothing + is a failure here because `enforce_hollow` is set (T2), and every `#[trusted]` + axiom must be listed (T3). `PROOF-PROGRESS.adoc` records the state and what a + first real obligation requires. + === Fixed * The shared secret scanner's owner spelling is now recorded where it is diff --git a/PROOF-PROGRESS.adoc b/PROOF-PROGRESS.adoc index 47d98892..70b0c858 100644 --- a/PROOF-PROGRESS.adoc +++ b/PROOF-PROGRESS.adoc @@ -15,6 +15,35 @@ conformance, not source-derived typestate or a proof of all executions. See link:docs/PURPOSE-AND-PROOF.adoc[purpose, comparison, proof status and opportunities] for the operator-facing distinction and bounded next steps. +== The Rust deductive-verification lane (Creusot) + +This repository now calls the estate's Creusot gate as its own check +(`.github/workflows/creusot-gate.yml`), at the same cicd-suite pin the estate +audit uses. Before this, the gate was not run here at all: the pinned estate +audit's `languages` job runs the Idris2, Zig, proof-runner and BoJ checks, not +the Creusot one, so nothing was watching for a Creusot claim in this tree. + +Measured on this tree with the gate's own `detect` and `theatre` phases +(2026-10-28): `obligations=false claims=false contract_count=0 +trusted_count=0`. The gate's verdict is therefore, in its own words, *"no +Creusot obligations and no verification claims in this tree. Nothing was proved +and nothing claims to be — this gate is silent, not green-as-assurance."* + +That silence is the accurate status, not a pass: there is no `creusot-std` +dependency, no `#[requires]`/`#[ensures]` anywhere, and no claim of Creusot +verification in this documentation. What the lane now guarantees is the +negative: a claim of Creusot verification with no contracts behind it (T1) is a +build failure, a `creusot-std` dependency that annotates nothing (T2) is a +failure here because `enforce_hollow` is set, and every `#[trusted]` axiom has +to be listed (T3). + +Adding a first real obligation needs three things together — the `creusot-std` +dependency, the nightly pin Creusot is built against, and a crate whose +contracts `cargo creusot prove` discharges — and honesty about ordering: with +no obligation, the `prove` phase does not run, so nothing about this tree is +proved today. The provenance ledger keeps `Actually-Proven` unreachable +(`proof_artifacts: []`) so that no test can claim it in the meantime. + == Headline [cols="2,1,4",options="header"] diff --git a/README.adoc b/README.adoc index f1391a43..0df1db29 100644 --- a/README.adoc +++ b/README.adoc @@ -30,7 +30,7 @@ and link:docs/adr/0004-companion-to-panic-attack.adoc[ADR-0004]. *Implementation in progress toward v0.1.0.* Milestones M0–M4 are present: the Rust workspace, scanning engine, six native tree-sitter grammars including PHP, -JSON 0.4.0/SARIF reporters, Python CFG/dataflow and a bounded PHP context-flow +JSON 0.5.0 (0.4.0 frozen)/SARIF reporters, Python CFG/dataflow and a bounded PHP context-flow slice. Thirteen source rules are registered (ten T0, three T1). `pons scan `, `pons catalogue`, `pons man`, `pons doctor`, `pons trace`, `pons wharf-trace` and `pons validate-report` exist; diff --git a/ci/json-ratchet.kyaml b/ci/json-ratchet.kyaml index 0c3ca98b..10991f71 100644 --- a/ci/json-ratchet.kyaml +++ b/ci/json-ratchet.kyaml @@ -25,6 +25,7 @@ }, { paths: [ + "crates/pons-core/tests/golden/report-0.4.0.json", "crates/pons-core/tests/golden/report.json", "crates/pons-core/tests/golden/report.sarif.json", ], @@ -144,9 +145,12 @@ { paths: [ "schemas/evidence-1.0.0.json", + "schemas/index.json", "schemas/scan-0.4.0.json", + "schemas/scan-0.5.0.json", + "schemas/verdict-matrix-1.0.0.json", ], - reason: "The published report schemas. They are the human-readable contract for Pons's machine-readable output and are reviewed line by line; a canonical rendering of a 755-line schema is not reviewable.", + reason: "The published contract artifacts: the report schemas, the frozen 0.4.0 schema, the version index that attests each of them by BLAKE3, and the severity x evidence matrix. They are the human-readable contract for Pons's machine-readable output and are reviewed line by line; a canonical rendering of a 755-line schema is not reviewable, and canonicalising a document whose field order is part of its published guarantee would change the contract rather than enforce it.", }, ], } diff --git a/ci/yaml-ratchet.kyaml b/ci/yaml-ratchet.kyaml index 95a1e79a..95ea87fb 100644 --- a/ci/yaml-ratchet.kyaml +++ b/ci/yaml-ratchet.kyaml @@ -28,6 +28,7 @@ { paths: [ ".github/workflows/codeql.yml", + ".github/workflows/creusot-gate.yml", ".github/workflows/deed-conformance.yml", ".github/workflows/format-policy.yml", ".github/workflows/k9-conformance.yml", diff --git a/crates/pons-cli/src/main.rs b/crates/pons-cli/src/main.rs index ef3dbab3..c7e0c6ab 100644 --- a/crates/pons-cli/src/main.rs +++ b/crates/pons-cli/src/main.rs @@ -93,11 +93,11 @@ enum FailOn { } impl FailOn { - fn rank(self) -> u8 { + fn severity(self) -> Severity { match self { - FailOn::Info => 0, - FailOn::Warn => 1, - FailOn::Error => 2, + FailOn::Info => Severity::Info, + FailOn::Warn => Severity::Warn, + FailOn::Error => Severity::Error, } } } @@ -126,14 +126,6 @@ impl Language { } } -fn severity_rank(s: Severity) -> u8 { - match s { - Severity::Info => 0, - Severity::Warn => 1, - Severity::Error => 2, - } -} - /// Run the CLI, treating a propagated broken pipe as success and other /// operational errors as exit status 2. fn main() -> ExitCode { @@ -312,8 +304,11 @@ fn doctor() -> String { /// `no_speculative` removes speculative findings and counts them as suppressed; /// `explain` adds evidence and tier annotations only in human output. /// -/// Returns 2 if `fail_on_skips` is set and any input was skipped, otherwise 1 -/// if a remaining non-speculative finding meets `fail_on`, or 0. +/// Each entry in `required_languages` must have at least one inventory entry, +/// with every matching entry analysed completely. +/// Returns 2 if no files were analysed, a coverage requirement failed, or +/// `fail_on_skips` is set and any input was skipped; otherwise 1 if a remaining +/// non-speculative finding meets `fail_on`, or 0. /// /// Missing paths, a forced grammar for a directory, unknown selected rules, /// configuration and discovery errors, and rendering or stdout errors propagate. @@ -450,8 +445,12 @@ fn run_scan( || (fail_on_skips && !report.skipped.is_empty()); let finding_threshold = fail_on.is_some_and(|threshold| { report.findings.iter().any(|f| { - f.evidence() != EvidenceClass::Speculative - && severity_rank(f.severity()) >= threshold.rank() + pons_core::verdict::counts_toward_verdict(f.evidence()) + // One rank table, shared with the published verdict matrix + // (schemas/verdict-matrix-1.0.0.json), so a consumer reading + // the data and the binary get the same gate. + && pons_core::verdict::fail_on_rank(f.severity()) + >= pons_core::verdict::fail_on_rank(threshold.severity()) }) }); report.exit_code = Some(if coverage_failed { diff --git a/crates/pons-cli/tests/contracts.rs b/crates/pons-cli/tests/contracts.rs new file mode 100644 index 00000000..85f01502 --- /dev/null +++ b/crates/pons-cli/tests/contracts.rs @@ -0,0 +1,271 @@ +// SPDX-License-Identifier: MPL-2.0 +//! The consumer-contract tests: what a pinned verdict grid relies on. +//! +//! The library tests prove the envelope's shape. These prove the three things a +//! *consumer* depends on that a shape test cannot see: +//! +//! * a report pinned at the frozen 0.4.0 version still validates, and the +//! version actually selects the schema, so a 0.4.0 declaration cannot carry +//! 0.5.0 keys; +//! * an unknown version is refused rather than validated against the nearest +//! shape; +//! * the corroboration key is recomputed rather than trusted, and the demoted +//! count cannot be inflated. +//! +//! Each of these is checked through the binary — `pons validate-report` — not +//! through the library, because the binary is what a consumer runs. + +use serde_json::{json, Value}; +use std::{fs, path::PathBuf, process::Command}; + +/// A source directory and a separate output directory. +/// +/// Separate on purpose: writing the report inside the scanned tree would make +/// it part of the next scan's inventory, and `validate-report` would then be +/// validating a document that describes a tree containing itself. +struct Fixture { + src: PathBuf, + out: PathBuf, +} + +impl Fixture { + fn new() -> Self { + let base = std::env::temp_dir().join(format!( + "pons-contract-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + let src = base.join("src"); + let out = base.join("out"); + fs::create_dir_all(&src).unwrap(); + fs::create_dir_all(&out).unwrap(); + // One file that fires a registered rule, so the report carries at least + // one finding to point the fingerprint tests at. + fs::write( + src.join("sample.php"), + " Value { + let output = Command::new(env!("CARGO_BIN_EXE_pons")) + .arg("scan") + .arg(&self.src) + .args(["--format", "json"]) + .output() + .expect("pons scan runs"); + assert_eq!( + output.status.code(), + Some(0), + "the fixture scan must be clean enough to produce a valid report: {}", + String::from_utf8_lossy(&output.stderr) + ); + let report: Value = serde_json::from_slice(&output.stdout).expect("scan emits JSON"); + assert!( + !report["findings"] + .as_array() + .expect("findings array") + .is_empty(), + "the fixture must fire at least one rule; a report with no finding \ + would make the fingerprint assertions vacuous" + ); + report + } + + fn validate(&self, report: &Value) -> std::process::Output { + let path = self.out.join("report.json"); + fs::write(&path, serde_json::to_string_pretty(report).unwrap()).unwrap(); + Command::new(env!("CARGO_BIN_EXE_pons")) + .arg("validate-report") + .arg(&path) + .output() + .expect("pons validate-report runs") + } +} + +impl Drop for Fixture { + fn drop(&mut self) { + if let Some(base) = self.src.parent() { + let _ = fs::remove_dir_all(base); + } + } +} + +/// Everything 0.5.0 added, removed again — the projection a pinned 0.4.0 +/// consumer holds. +fn as_0_4_0(report: &Value) -> Value { + let mut pinned = report.clone(); + pinned["schema_version"] = json!("0.4.0"); + let counts = pinned["counts"].as_object_mut().expect("counts object"); + counts.remove("actionable"); + counts.remove("demoted_speculative"); + for finding in pinned["findings"].as_array_mut().expect("findings array") { + finding + .as_object_mut() + .expect("finding object") + .remove("semantic_fingerprint"); + } + pinned +} + +#[test] +fn a_report_pinned_at_the_frozen_0_4_0_version_still_validates() { + let fixture = Fixture::new(); + let current = fixture.scan(); + assert_eq!(current["schema_version"], "0.5.0"); + + let accepted = fixture.validate(¤t); + assert_eq!( + accepted.status.code(), + Some(0), + "this build's own report must validate: {}", + String::from_utf8_lossy(&accepted.stderr) + ); + + let accepted = fixture.validate(&as_0_4_0(¤t)); + assert_eq!( + accepted.status.code(), + Some(0), + "a consumer pinned to 0.4.0 must still be able to validate the reports \ + it holds; the freeze would be worthless otherwise: {}", + String::from_utf8_lossy(&accepted.stderr) + ); +} + +#[test] +fn a_0_4_0_declaration_cannot_carry_a_0_5_0_key() { + let fixture = Fixture::new(); + let current = fixture.scan(); + + let mut mixed = as_0_4_0(¤t); + // Put one 0.5.0 key back, leaving the 0.4.0 declaration in place. If the + // validator simply used the current schema whatever the document said, this + // document would pass and the version would mean nothing. + let key = current["findings"][0]["semantic_fingerprint"].clone(); + mixed["findings"][0]["semantic_fingerprint"] = key; + let rejected = fixture.validate(&mixed); + assert_eq!( + rejected.status.code(), + Some(2), + "a 0.4.0 document carrying a 0.5.0 key must be rejected: {}", + String::from_utf8_lossy(&rejected.stderr) + ); +} + +#[test] +fn an_unknown_envelope_version_is_refused() { + let fixture = Fixture::new(); + let mut report = fixture.scan(); + report["schema_version"] = json!("0.6.0"); + let rejected = fixture.validate(&report); + assert_eq!( + rejected.status.code(), + Some(2), + "an unknown version must not be validated against the nearest shape: {}", + String::from_utf8_lossy(&rejected.stderr) + ); +} + +#[test] +fn a_forged_corroboration_key_is_rejected() { + let fixture = Fixture::new(); + let mut report = fixture.scan(); + let key = report["findings"][0]["semantic_fingerprint"] + .as_str() + .expect("the report carries a key") + .to_string(); + // Flip one hex digit. A validator that only checked the key's shape would + // pass this — and then a producer could point any finding at any site, + // which is exactly what the grid deduplicates on. + let flipped = format!( + "{}{}{}", + &key[..1], + if &key[1..2] == "0" { "1" } else { "0" }, + &key[2..] + ); + assert_ne!(flipped, key); + report["findings"][0]["semantic_fingerprint"] = json!(flipped); + + let rejected = fixture.validate(&report); + assert_eq!( + rejected.status.code(), + Some(2), + "a corroboration key that does not match its own location must be \ + rejected: {}", + String::from_utf8_lossy(&rejected.stderr) + ); +} + +#[test] +fn a_speculative_finding_cannot_be_promoted_into_the_actionable_count() { + let fixture = Fixture::new(); + let mut report = fixture.scan(); + let counts = &report["counts"]; + assert_eq!( + counts["actionable"].as_u64().unwrap() + counts["demoted_speculative"].as_u64().unwrap(), + counts["total"].as_u64().unwrap(), + "the two counts must account for every finding exactly once" + ); + assert_eq!( + counts["demoted_speculative"], + counts["by_evidence"]["SPECULATIVE"] + ); + + // The demotion is a number, so it is enforced as one: adding a demoted + // finding back into the actionable total is a rejected document, not a + // judgement call for the reader. + let inflated = counts["actionable"].as_u64().unwrap() + 1; + report["counts"]["actionable"] = json!(inflated); + let rejected = fixture.validate(&report); + assert_eq!( + rejected.status.code(), + Some(2), + "a report that counts a demoted finding as actionable must be \ + rejected: {}", + String::from_utf8_lossy(&rejected.stderr) + ); +} + +#[test] +fn a_finding_outside_the_scan_root_is_rejected() { + let fixture = Fixture::new(); + let mut report = fixture.scan(); + let original = report["findings"][0]["location"]["file"] + .as_str() + .expect("the finding names a file") + .to_string(); + let outside = "/elsewhere/outside.php"; + + // Move the finding AND its inventory entry together, so the only thing + // wrong with the document is that the path cannot be expressed relative to + // the scan root. Without the inventory move it would be rejected earlier, + // for a different reason, and this test would pass without exercising the + // check it is named for. + report["findings"][0]["location"]["file"] = json!(outside); + for entry in report["scanned"]["inventory"] + .as_array_mut() + .expect("inventory array") + { + if entry["path"] == json!(original) { + entry["path"] = json!(outside); + } + } + + let rejected = fixture.validate(&report); + assert_eq!( + rejected.status.code(), + Some(2), + "a finding outside the scan root must be rejected: {}", + String::from_utf8_lossy(&rejected.stderr) + ); + assert!( + String::from_utf8_lossy(&rejected.stderr).contains("not inside the scan root"), + "it must be rejected BY the root check, not incidentally by another: {}", + String::from_utf8_lossy(&rejected.stderr) + ); +} diff --git a/crates/pons-cli/tests/coverage.rs b/crates/pons-cli/tests/coverage.rs index 0ef97601..0d884df7 100644 --- a/crates/pons-cli/tests/coverage.rs +++ b/crates/pons-cli/tests/coverage.rs @@ -55,7 +55,7 @@ fn inventory_accounts_for_supported_unsupported_excluded_and_parse_error() { let out = t.scan(&["--format", "json", "--fail-on-skips"]); assert_eq!(out.status.code(), Some(2)); let v: Value = serde_json::from_slice(&out.stdout).unwrap(); - assert_eq!(v["schema_version"], "0.4.0"); + assert_eq!(v["schema_version"], "0.5.0"); let inv = v["scanned"]["inventory"].as_array().unwrap(); assert_eq!(inv.len(), 7); for (name, status) in [ diff --git a/crates/pons-core/src/corroborate.rs b/crates/pons-core/src/corroborate.rs new file mode 100644 index 00000000..5bee3d40 --- /dev/null +++ b/crates/pons-core/src/corroborate.rs @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: MPL-2.0 +//! A per-finding *semantic* fingerprint: the stable key an aggregating verdict +//! grid uses to mark one site once instead of counting it twice. +//! +//! A pons report already carries a fingerprint per record in +//! `evidence_contract.records[].fingerprint`, and that one answers a different +//! question. It hashes the rule id and a whitespace-folded source anchor, so +//! two rules firing on the same line are two records — which is right for +//! provenance and wrong for aggregation. The key here hashes the root-relative +//! path, the 1-based line and the evidence class, and **deliberately ignores +//! the rule id and the producer**, so a `panic-attack` or Hypatia finding at the +//! same path, line and class can be matched to it once rather than added to it. +//! +//! Consequences a consumer must know, all of them intended: +//! +//! * two pons findings at one site in one evidence class share a key; the grid +//! counts that site once and keeps both findings' provenance; +//! * the key carries no severity, so it stays stable when a rule's severity is +//! tuned — severity is advisory, the evidence class is the honesty invariant; +//! * the key is not a proof of semantic equivalence, and a `SPECULATIVE` key +//! corroborates nothing at all. Agreement between producers is corroboration, +//! not independent proof — see `docs/PURPOSE-AND-PROOF.adoc` and the +//! published policy in `schemas/verdict-matrix-1.0.0.json`. +//! +//! The field layout is versioned by [`SCHEME`], which is hashed as the first +//! field: a future change cannot collide with keys minted by this version. + +use crate::finding::{EvidenceClass, Finding}; + +/// Scheme identifier, hashed as the first field of the canonical input. +pub const SCHEME: &str = "pons-corroboration-v1"; + +/// Field separator. NUL is outside the alphabet of paths, decimal line numbers +/// and evidence class names, so the canonical input cannot be read two ways. +const SEPARATOR: char = '\0'; + +/// The exact string that is hashed. +/// +/// Joins [`SCHEME`], `path`, the decimal `line` and the wire name of `class` +/// with NUL separators and no trailing separator. Supply a root-relative, +/// `/`-separated path without a leading `./` and a 1-based start line; +/// these conventions are not normalised or checked here. +/// +/// Public because a consumer must be able to mint the same key without pons — +/// for a panic-attack or Hypatia finding, once the grid has mapped it onto an +/// evidence class using `schemas/verdict-matrix-1.0.0.json`. An implementation +/// that cannot reproduce this string byte for byte has not implemented the +/// contract. +pub fn canonical_input(path: &str, line: usize, class: EvidenceClass) -> String { + format!( + "{scheme}{sep}{path}{sep}{line}{sep}{class}", + scheme = SCHEME, + sep = SEPARATOR, + path = path, + line = line, + class = class.name() + ) +} + +/// BLAKE3 of [`canonical_input`], as 64 lowercase hexadecimal characters. +pub fn fingerprint_of(path: &str, line: usize, class: EvidenceClass) -> String { + blake3::hash(canonical_input(path, line, class).as_bytes()) + .to_hex() + .to_string() +} + +/// The fingerprint of one finding, relative to the scan root. +/// +/// Uses the finding's 1-based start line and evidence class. The path matches +/// the evidence contract's records ([`crate::evidence::relative_path`]): strip +/// `root` when it is a prefix, use the basename when `root` is the file itself, +/// and otherwise retain the path. Backslashes become `/`; no filesystem lookup +/// is performed. +pub fn semantic_fingerprint(finding: &Finding, root: &str) -> String { + let loc = finding.location(); + fingerprint_of( + &crate::evidence::relative_path(&loc.file, root), + loc.line_start, + finding.evidence(), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::finding::{Location, Severity, Tier}; + + fn finding(rule: &str, file: &str, line: usize, tier: Tier) -> Finding { + Finding::new( + rule, + tier, + Severity::Warn, + Location { + file: file.to_string(), + byte_start: 0, + byte_end: 1, + line_start: line, + col_start: 1, + line_end: line, + col_end: 2, + }, + "msg", + "note", + None, + ) + } + + #[test] + fn the_canonical_input_is_four_nul_separated_fields() { + let input = canonical_input("app/svc.py", 41, EvidenceClass::Dataflow); + assert_eq!( + input.split('\0').collect::>(), + vec![SCHEME, "app/svc.py", "41", "DATAFLOW"], + "the field order and the separator are the whole contract: a \ + consumer reproducing this string byte for byte must get the same key" + ); + } + + /// A pinned digest, not a recomputed one. If the scheme name, the field + /// order, the separator or the class spelling changes, this number changes + /// and the diff says so — whereas a self-consistent test (hash, then compare + /// with the same function) would pass through any of those changes and + /// quietly invalidate every key a consumer had stored. + #[test] + fn a_pinned_key_pins_the_scheme() { + assert_eq!( + fingerprint_of("app/svc.py", 41, EvidenceClass::Dataflow), + "a54bb8fa2b7f74ff5c9d1fa745ac22e61abb49847ddbac0bd220fef2442baaf1" + ); + } + + #[test] + fn a_key_is_64_lowercase_hex_characters() { + let key = fingerprint_of("app/svc.py", 41, EvidenceClass::Dataflow); + assert_eq!(key.len(), 64); + assert!(key + .bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b))); + } + + #[test] + fn the_key_changes_with_the_path_the_line_and_the_class() { + let base = fingerprint_of("app/svc.py", 41, EvidenceClass::Dataflow); + assert_ne!( + base, + fingerprint_of("app/svc.ts", 41, EvidenceClass::Dataflow) + ); + assert_ne!( + base, + fingerprint_of("app/svc.py", 42, EvidenceClass::Dataflow) + ); + assert_ne!( + base, + fingerprint_of("app/svc.py", 41, EvidenceClass::Heuristic) + ); + } + + #[test] + fn the_key_ignores_the_rule_id_because_overlap_is_the_point() { + // Two rules naming the same line in the same evidence class are one + // site. If the rule id were hashed, the grid would double-count exactly + // the overlap this key exists to collapse. + let a = finding("dead-store", "app/svc.py", 41, Tier::T1); + let b = finding("read-before-init", "app/svc.py", 41, Tier::T1); + assert_eq!(semantic_fingerprint(&a, "."), semantic_fingerprint(&b, ".")); + // ...but a different class at the same site is a different claim, and + // must not be collapsed into the same corroborated row. + let c = finding("dead-store", "app/svc.py", 41, Tier::T3); + assert_ne!(semantic_fingerprint(&a, "."), semantic_fingerprint(&c, ".")); + } + + #[test] + fn the_key_uses_the_contract_path_spelling_and_survives_a_checkout_move() { + let from_root = semantic_fingerprint(&finding("r", "app/svc.py", 41, Tier::T1), "."); + let from_absolute = semantic_fingerprint( + &finding("r", "/home/dev/checkout/app/svc.py", 41, Tier::T1), + "/home/dev/checkout", + ); + assert_eq!(from_root, from_absolute); + assert_eq!( + from_root, + fingerprint_of("app/svc.py", 41, EvidenceClass::Dataflow) + ); + } + + #[test] + fn a_single_file_scan_keys_the_file_name_not_the_absolute_path() { + // root == file for a one-file scan; the contract's relative_path strips + // to the basename, so the key does not encode the build machine. + let key = semantic_fingerprint( + &finding("r", "/tmp/probe/a.py", 3, Tier::T1), + "/tmp/probe/a.py", + ); + assert_eq!(key, fingerprint_of("a.py", 3, EvidenceClass::Dataflow)); + } +} diff --git a/crates/pons-core/src/evidence.rs b/crates/pons-core/src/evidence.rs index 555d28fb..962a0145 100644 --- a/crates/pons-core/src/evidence.rs +++ b/crates/pons-core/src/evidence.rs @@ -100,13 +100,9 @@ pub fn contract(report: &ScanReport) -> Contract<'_> { rule: f.rule_id().into(), tier: f.tier(), evidence: f.evidence(), - confidence: match f.tier() { - Tier::T0 => "heuristic", - Tier::T1 => "model-dependent", - Tier::T2 => "premise-dependent", - Tier::T3 => "speculative", - } - .into(), + // One table, in verdict.rs, because the same label is + // published to consumers in schemas/verdict-matrix-1.0.0.json. + confidence: crate::verdict::confidence(f.evidence()).into(), path: relative, line: f.location().line_start, trace: f.evidence_note().into(), diff --git a/crates/pons-core/src/finding.rs b/crates/pons-core/src/finding.rs index 72eb264d..2e622f1e 100644 --- a/crates/pons-core/src/finding.rs +++ b/crates/pons-core/src/finding.rs @@ -30,6 +30,24 @@ impl From for EvidenceClass { } } +impl EvidenceClass { + /// The wire spelling, as it appears in every report. + /// + /// A method rather than four string literals wherever a class has to be + /// named outside serde — the corroboration fingerprint hashes this string, + /// so it must be the same spelling the envelope carries. The test below + /// compares the two, so a rename in either place fails the build rather + /// than silently minting a second vocabulary. + pub fn name(self) -> &'static str { + match self { + EvidenceClass::Heuristic => "HEURISTIC", + EvidenceClass::Dataflow => "DATAFLOW", + EvidenceClass::Protocol => "PROTOCOL", + EvidenceClass::Speculative => "SPECULATIVE", + } + } +} + #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "UPPERCASE")] pub enum Severity { @@ -403,6 +421,27 @@ mod tests { } } + #[test] + fn name_matches_the_wire_spelling_and_the_tier_edge() { + // One spelling, three consumers: serde in the envelope, the + // corroboration fingerprint, and the published verdict matrix. This + // asserts the method against serde, so a rename in either place fails + // here rather than silently minting a second vocabulary. + for (class, tier) in [ + (EvidenceClass::Heuristic, Tier::T0), + (EvidenceClass::Dataflow, Tier::T1), + (EvidenceClass::Protocol, Tier::T2), + (EvidenceClass::Speculative, Tier::T3), + ] { + assert_eq!( + serde_json::to_value(class).unwrap(), + serde_json::json!(class.name()) + ); + // ...and the tier edge still points the same way. + assert_eq!(EvidenceClass::from(tier), class); + } + } + #[test] fn evidence_class_is_derived_from_tier() { let f = Finding::new("r", Tier::T3, Severity::Warn, loc(), "msg", "note", None); diff --git a/crates/pons-core/src/lib.rs b/crates/pons-core/src/lib.rs index e6e8aea3..1c69be5f 100644 --- a/crates/pons-core/src/lib.rs +++ b/crates/pons-core/src/lib.rs @@ -2,6 +2,7 @@ pub mod cfg; pub mod config; +pub mod corroborate; pub mod coverage; pub mod dataflow; pub mod engine; @@ -13,3 +14,4 @@ pub mod php; pub mod report; pub mod source; pub mod suppress; +pub mod verdict; diff --git a/crates/pons-core/src/report/human.rs b/crates/pons-core/src/report/human.rs index 83906043..f14a714f 100644 --- a/crates/pons-core/src/report/human.rs +++ b/crates/pons-core/src/report/human.rs @@ -3,8 +3,7 @@ use std::fmt::Write as _; use crate::engine::ScanReport; -use crate::finding::{EvidenceClass, Finding, Severity}; -use crate::report::SPECULATIVE_SUFFIX; +use crate::finding::{Finding, Severity}; /// Deterministic, colour-free rendering of findings for terminal/CI output. pub fn render(findings: &[Finding]) -> String { @@ -14,6 +13,8 @@ pub fn render(findings: &[Finding]) -> String { /// Full terminal report, including the accounting a reader needs to know /// whether a clean result was a complete scan. `--explain` adds the evidence /// class and tier that back (or qualify) each finding. +/// The verdict summary excludes speculative findings from the actionable count +/// and reports their demoted count separately; the scanned finding total includes both. pub fn render_report(report: &ScanReport, explain: bool) -> String { let mut out = render_detailed(&report.findings, explain); let _ = writeln!( @@ -25,6 +26,24 @@ pub fn render_report(report: &ScanReport, explain: bool) -> String { report.skipped.len(), report.transcoded.len() ); + // The numeric half of the demotion. The per-finding suffix above demotes a + // guess visually; this line stops it being added to the count a reader + // takes away, which is the same distinction the JSON envelope publishes as + // counts.actionable / counts.demoted_speculative. + // The predicate and the text both come from verdict.rs, which is where the + // published matrix gets them: a local `== Speculative` test here would be a + // second source of truth the matrix drift test cannot see. + let speculative = report + .findings + .iter() + .filter(|f| !crate::verdict::counts_toward_verdict(f.evidence())) + .count(); + let _ = writeln!( + out, + "verdict: {} actionable finding(s); {} speculative demoted, not counted", + report.findings.len() - speculative, + speculative + ); let _ = writeln!(out, "coverage: {}", report.coverage_status()); for failure in &report.coverage_failures { let _ = writeln!(out, "coverage failure: {failure}"); @@ -62,11 +81,7 @@ fn render_one(f: &Finding, out: &mut String, explain: bool) { Severity::Error => "ERROR", }; - let suffix = if f.evidence() == EvidenceClass::Speculative { - SPECULATIVE_SUFFIX - } else { - "" - }; + let suffix = crate::verdict::demotion_suffix(f.evidence()); let loc = f.location(); let _ = writeln!( @@ -168,6 +183,17 @@ mod golden_tests { ); } + #[test] + fn the_summary_demotes_speculative_numerically() { + // The shared fixture carries one SPECULATIVE finding of four. + let out = render_report(&fixture::report(), false); + assert!( + out.contains("verdict: 3 actionable finding(s); 1 speculative demoted, not counted"), + "the aggregate line must separate guesses from the count a reader \ + takes away; got:\n{out}" + ); + } + #[test] fn the_speculative_finding_is_demoted_in_the_shared_fixture() { let out = render(&fixture::report().findings); diff --git a/crates/pons-core/src/report/json.rs b/crates/pons-core/src/report/json.rs index 4149a4a2..af04f634 100644 --- a/crates/pons-core/src/report/json.rs +++ b/crates/pons-core/src/report/json.rs @@ -19,7 +19,7 @@ use serde::Serialize; use crate::engine::{ScanReport, SkippedFile}; -use crate::finding::{EvidenceClass, Finding}; +use crate::finding::{EvidenceClass, Finding, Location, Severity, Tier}; /// Version of the envelope itself, not of the tool. /// @@ -30,9 +30,41 @@ use crate::finding::{EvidenceClass, Finding}; /// start. `envelope_shape_is_pinned_to_the_schema_version` below is what /// stops the next change skipping this line silently. /// +/// 0.4.0 -> 0.5.0 added the three keys an aggregating consumer needs and could +/// not derive: `findings[].semantic_fingerprint` (the cross-producer +/// corroboration key from [`crate::corroborate`]) and the two demoted-count +/// fields `counts.actionable` / `counts.demoted_speculative`. All three are +/// additive and appended, so a 0.4.0-shaped reading of a 0.5.0 document still +/// works; the bump is still taken, because a consumer must be able to tell +/// whether a report carries the corroboration key at all. +/// +/// 0.4.0 does not disappear. It is **frozen**: `schemas/scan-0.4.0.json` is +/// byte-pinned by [`FROZEN_0_4_0_SCHEMA_BLAKE3`] and by `schemas/index.json`, +/// `validate-report` still accepts a 0.4.0 report against that schema, and +/// `a_report_pinned_at_the_frozen_0_4_0_version_still_validates` in +/// `pons-cli/tests/contracts.rs` proves it. A frozen +/// version is never edited and never re-issued; a change is a new file. +/// /// Not to be confused with the `schema_version` in a protocol TOML /// (ADR-0003), which versions a different file entirely. -pub const SCHEMA_VERSION: &str = "0.4.0"; +pub const SCHEMA_VERSION: &str = "0.5.0"; + +/// Envelope versions this build will still validate, and only validate. +/// +/// Pinned consumers keep their old reports; refusing to read them would make +/// the freeze worthless. Each entry is a schema file on disk, byte-attested by +/// [`FROZEN_0_4_0_SCHEMA_BLAKE3`] and `schemas/index.json`. +pub const FROZEN_SCHEMA_VERSIONS: &[&str] = &["0.4.0"]; + +/// BLAKE3 of `schemas/scan-0.4.0.json`, as committed. +/// +/// The freeze is a claim about bytes, so it is checked against bytes: editing +/// the frozen schema — even to "just add a key" — fails +/// `the_frozen_0_4_0_schema_is_still_the_file_this_build_was_tested_against`. +/// A 0.4.0 consumer pinned to this repository therefore reads a document that +/// cannot change underneath it. +pub const FROZEN_0_4_0_SCHEMA_BLAKE3: &str = + "bdef6c68691ffa0681b56d86adc31450e8c625d7ce5884090efe6fbc8fc46562"; #[derive(Serialize)] struct Envelope<'a> { @@ -41,7 +73,7 @@ struct Envelope<'a> { execution: Execution, scanned: Scanned<'a>, counts: Counts, - findings: &'a [Finding], + findings: Vec>, evidence_contract: crate::evidence::Contract<'a>, } @@ -81,6 +113,30 @@ struct Scanned<'a> { struct Counts { by_evidence: ByEvidence, total: usize, + /// `total` minus the demoted count: the number an aggregating consumer adds + /// up. Published rather than left to subtraction so the demotion is a + /// number in the document, not a convention a consumer might miss. + actionable: usize, + /// Findings that are demoted precisely because they are guesses — the + /// numeric half of the demotion the human and SARIF reporters apply + /// visually. Equal to `by_evidence.SPECULATIVE`, and published twice on + /// purpose: a consumer reading only the aggregate must see it. + demoted_speculative: usize, +} + +impl Counts { + /// Count all findings by evidence class, including zero counts, and exclude + /// speculative findings from the actionable total regardless of severity. + fn tally(findings: &[Finding]) -> Self { + let by_evidence = ByEvidence::tally(findings); + let demoted = by_evidence.speculative; + Counts { + by_evidence, + total: findings.len(), + actionable: findings.len() - demoted, + demoted_speculative: demoted, + } + } } /// A fixed four-field struct rather than a map: every evidence class must @@ -119,6 +175,45 @@ impl ByEvidence { } } +/// One finding as the envelope carries it: the finding itself, plus the +/// corroboration key the verdict grid matches other producers against. +/// +/// A view rather than a field on [`Finding`] because the key needs the scan +/// root, which no rule and no engine step knows — the reporter does. Field +/// order is the contract: the first eight fields are exactly 0.4.0's, in +/// 0.4.0's order, and `semantic_fingerprint` is appended, so the 0.5.0 finding +/// really is a 0.4.0 finding plus one key rather than a reshuffle. +#[derive(Serialize)] +struct FindingView<'a> { + rule_id: &'a str, + tier: Tier, + evidence: EvidenceClass, + severity: Severity, + location: &'a Location, + message: &'a str, + evidence_note: &'a str, + counter_condition: Option<&'a str>, + semantic_fingerprint: String, +} + +impl<'a> FindingView<'a> { + /// Borrow a finding's fields and add its corroboration key relative to the + /// scan `root`, using the finding's start line and evidence class. + fn of(finding: &'a Finding, root: &str) -> Self { + Self { + rule_id: finding.rule_id(), + tier: finding.tier(), + evidence: finding.evidence(), + severity: finding.severity(), + location: finding.location(), + message: finding.message(), + evidence_note: finding.evidence_note(), + counter_condition: finding.counter_condition(), + semantic_fingerprint: crate::corroborate::semantic_fingerprint(finding, root), + } + } +} + /// Render a [`ScanReport`] as the Appendix G envelope, with a trailing /// newline so the output is a well-formed text stream. pub fn render(report: &ScanReport) -> anyhow::Result { @@ -145,11 +240,12 @@ pub fn render(report: &ScanReport) -> anyhow::Result { transcoded: report.transcoded.iter().map(String::as_str).collect(), skipped: &report.skipped, }, - counts: Counts { - by_evidence: ByEvidence::tally(&report.findings), - total: report.findings.len(), - }, - findings: &report.findings, + counts: Counts::tally(&report.findings), + findings: report + .findings + .iter() + .map(|f| FindingView::of(f, &report.root)) + .collect(), evidence_contract: crate::evidence::contract(report), }; @@ -316,7 +412,10 @@ mod tests { "transcoded", ], ), - ("counts", &["by_evidence", "total"]), + ( + "counts", + &["actionable", "by_evidence", "demoted_speculative", "total"], + ), ( "counts.by_evidence", &["DATAFLOW", "HEURISTIC", "PROTOCOL", "SPECULATIVE"], @@ -362,4 +461,223 @@ mod tests { constant — one of the two was bumped without the other" ); } + + /// 0.5.0 promises that a finding gained exactly one key and gained it last. + /// A key set cannot show order, so the order is asserted on the rendered + /// text against the *frozen 0.4.0 schema's own* required list — the frozen + /// file is the specification of "what a 0.4.0 finding was", so this cannot + /// drift into agreeing with a mistake in this module. + #[test] + fn a_0_5_0_finding_is_a_0_4_0_finding_plus_one_appended_key() { + let frozen: serde_json::Value = + serde_json::from_str(include_str!("../../../../schemas/scan-0.4.0.json")).unwrap(); + let four_oh: Vec<&str> = frozen["properties"]["findings"]["items"]["required"] + .as_array() + .expect("the frozen schema lists required finding keys") + .iter() + .map(|v| v.as_str().expect("required keys are strings")) + .collect(); + assert_eq!( + four_oh, + vec![ + "rule_id", + "tier", + "evidence", + "severity", + "location", + "message", + "evidence_note", + "counter_condition", + ], + "the 0.4.0 finding shape is the baseline this test is anchored to" + ); + + let out = render(&fixture::report()).unwrap(); + let tail = &out[out.find("\"findings\": [").expect("findings array")..]; + let positions: Vec = four_oh + .iter() + .map(|key| { + tail.find(&format!("\"{key}\": ")) + .unwrap_or_else(|| panic!("0.4.0 key {key} vanished from a finding")) + }) + .collect(); + let mut sorted = positions.clone(); + sorted.sort_unstable(); + assert_eq!( + positions, sorted, + "the 0.4.0 keys are no longer a prefix of the 0.5.0 finding, in order" + ); + + let fingerprint = tail + .find("\"semantic_fingerprint\": ") + .expect("0.5.0 findings carry the corroboration key"); + assert!( + fingerprint > *positions.last().expect("non-empty baseline"), + "semantic_fingerprint must be appended, not interleaved: a 0.4.0 \ + reader that stops at counter_condition must still have read a \ + complete finding" + ); + } + + #[test] + fn every_finding_carries_its_corroboration_key() { + let out = render(&fixture::report()).unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&out).unwrap(); + for finding in parsed["findings"].as_array().unwrap() { + let key = finding["semantic_fingerprint"] + .as_str() + .unwrap_or_else(|| panic!("missing key on {finding}")); + assert_eq!(key.len(), 64, "{key}"); + assert!( + key.bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)), + "{key} is not lowercase hex" + ); + } + // Tied to the implementation, not just to a shape: recompute the first + // one from the fields the same document publishes. + let first = &parsed["findings"][0]; + let class: EvidenceClass = + serde_json::from_value(first["evidence"].clone()).expect("evidence class"); + let expected = crate::corroborate::fingerprint_of( + "app/svc.py", + first["location"]["line_start"].as_u64().unwrap() as usize, + class, + ); + assert_eq!(first["semantic_fingerprint"], expected); + } + + /// The numeric half of the demotion. A guess is reported, never added up: + /// `actionable` is what a verdict grid sums, and `total` still equals the + /// array length so no consumer loses sight of a finding. + #[test] + fn the_counts_demote_speculative_numerically() { + let out = render(&fixture::report()).unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&out).unwrap(); + let counts = &parsed["counts"]; + assert_eq!(counts["total"], 4); + assert_eq!(counts["demoted_speculative"], 1); + assert_eq!(counts["actionable"], 3); + let actionable = counts["actionable"].as_u64().unwrap(); + let demoted = counts["demoted_speculative"].as_u64().unwrap(); + assert_eq!( + actionable + demoted, + counts["total"].as_u64().unwrap(), + "actionable + demoted must account for every finding exactly once" + ); + assert_eq!( + counts["demoted_speculative"], + counts["by_evidence"]["SPECULATIVE"] + ); + } + + /// The freeze, checked against the bytes rather than asserted in prose. + #[test] + fn the_frozen_0_4_0_schema_is_still_the_file_this_build_was_tested_against() { + let bytes = std::fs::read( + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../schemas/scan-0.4.0.json"), + ) + .expect("the frozen 0.4.0 schema must be committed"); + let actual = blake3::hash(&bytes).to_hex().to_string(); + assert_eq!( + actual, FROZEN_0_4_0_SCHEMA_BLAKE3, + "schemas/scan-0.4.0.json changed. A frozen envelope version is never \ + edited: revert the edit, or publish a new version and add it to \ + FROZEN_SCHEMA_VERSIONS with its own schema file." + ); + } + + /// A 0.4.0 report remains readable by this build's validator, and the + /// frozen schema is genuinely closed — so this is not a test that passes + /// against any JSON at all. + #[test] + fn a_frozen_0_4_0_report_still_satisfies_the_frozen_schema() { + let schema: serde_json::Value = + serde_json::from_str(include_str!("../../../../schemas/scan-0.4.0.json")).unwrap(); + let validator = jsonschema::validator_for(&schema).expect("the frozen schema compiles"); + + // Both samples carry the golden tests' `TEST-DIRTY` placeholder, which + // is deliberately not one of the three wire values (`true`, `false`, + // `unknown`) — an artefact of how the goldens are made, not of the + // envelope. Normalising that one field keeps this test about the shape; + // the assertion below proves the placeholder really was there, so the + // normalisation cannot be hiding an invalid document. + fn with_unknown_dirty(mut v: serde_json::Value) -> serde_json::Value { + v["tool"]["source_dirty"] = serde_json::json!("unknown"); + v + } + + let frozen: serde_json::Value = + serde_json::from_str(include_str!("../../tests/golden/report-0.4.0.json")).unwrap(); + assert_eq!(frozen["schema_version"], "0.4.0"); + assert_eq!( + frozen["tool"]["source_dirty"], "TEST-DIRTY", + "precondition: the frozen sample carries the golden placeholder" + ); + assert!( + validator.is_valid(&with_unknown_dirty(frozen)), + "the committed 0.4.0 sample no longer validates against its own schema" + ); + + let current = + with_unknown_dirty(serde_json::from_str(&render(&fixture::report()).unwrap()).unwrap()); + assert!( + !validator.is_valid(¤t), + "the frozen 0.4.0 schema accepted a {} document — it is not closed, \ + and the version dispatch in validate.rs is doing nothing", + SCHEMA_VERSION + ); + } + + /// `schemas/index.json` is how a consumer at a pin discovers the artifacts + /// and checks it has the bytes pons was tested with. + #[test] + fn the_schema_index_attests_every_published_contract_file() { + let index: serde_json::Value = + serde_json::from_str(include_str!("../../../../schemas/index.json")).unwrap(); + let artifacts = index["artifacts"] + .as_array() + .expect("the index lists artifacts"); + assert!(!artifacts.is_empty(), "an empty index attests nothing"); + + let schemas = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../schemas"); + for artifact in artifacts { + let file = artifact["file"].as_str().expect("file"); + let version = artifact["version"].as_str().expect("version"); + assert!( + file.contains(version), + "schemas/{file} does not carry its version {version} in its name" + ); + let bytes = std::fs::read(schemas.join(file)) + .unwrap_or_else(|e| panic!("cannot read schemas/{file}: {e}")); + let actual = blake3::hash(&bytes).to_hex().to_string(); + assert_eq!( + artifact["blake3"].as_str().expect("blake3"), + actual.as_str(), + "schemas/{file} changed: update its blake3 entry in schemas/index.json" + ); + } + + let envelopes: Vec<&serde_json::Value> = artifacts + .iter() + .filter(|a| a["id"] == "scan-envelope") + .collect(); + let current: Vec<&&serde_json::Value> = envelopes + .iter() + .filter(|a| a["status"] == "current") + .collect(); + assert_eq!( + current.len(), + 1, + "exactly one scan envelope version is the one this build emits" + ); + assert_eq!(current[0]["version"], SCHEMA_VERSION); + for frozen in envelopes.iter().filter(|a| a["status"] == "frozen") { + assert!( + FROZEN_SCHEMA_VERSIONS.contains(&frozen["version"].as_str().expect("version")), + "schemas/index.json calls {} frozen, FROZEN_SCHEMA_VERSIONS does not", + frozen["version"] + ); + } + } } diff --git a/crates/pons-core/src/report/sarif.rs b/crates/pons-core/src/report/sarif.rs index a2ce6fe3..2742c667 100644 --- a/crates/pons-core/src/report/sarif.rs +++ b/crates/pons-core/src/report/sarif.rs @@ -10,8 +10,7 @@ use std::collections::{BTreeMap, BTreeSet}; use serde::Serialize; use crate::engine::ScanReport; -use crate::finding::{EvidenceClass, Finding, Severity, Tier}; -use crate::report::SPECULATIVE_SUFFIX; +use crate::finding::{EvidenceClass, Finding, Tier}; const SARIF_SCHEMA: &str = "https://json.schemastore.org/sarif-2.1.0.json"; const INFORMATION_URI: &str = "https://github.com/hyperpolymath/pons-asinorum"; @@ -130,30 +129,22 @@ fn artifact_uri(file: &str, root: &str) -> String { /// SARIF severity for a finding. /// -/// The `SPECULATIVE` check comes **first** and unconditionally: Appendix E -/// requires T3 to render as `"note"` whatever its own severity says. A -/// speculative finding must never arrive in a code-scanning UI wearing the -/// colour of a verdict. +/// Delegates to [`crate::verdict::sarif_level`], which owns the mapping because +/// the same table is published as data for consumers. That function keeps the +/// `SPECULATIVE` check **first** and unconditional: Appendix E requires T3 to +/// render as `"note"` whatever its own severity says. A speculative finding +/// must never arrive in a code-scanning UI wearing the colour of a verdict. fn level_of(f: &Finding) -> &'static str { - if f.evidence() == EvidenceClass::Speculative { - return "note"; - } - match f.severity() { - Severity::Error => "error", - Severity::Warn => "warning", - Severity::Info => "note", - } + crate::verdict::sarif_level(f.evidence(), f.severity()) } /// Appendix E's fixed rank table: how much attention the evidence class has -/// earned, independent of severity. +/// earned, independent of severity. Owned by [`crate::verdict::sarif_rank`] so +/// the published matrix cannot state a different number than this reporter +/// emits; note that this rank is *not* an aggregation weight — see +/// [`crate::verdict::evidence_weight`]. fn rank_of(evidence: EvidenceClass) -> f64 { - match evidence { - EvidenceClass::Heuristic => 50.0, - EvidenceClass::Dataflow => 80.0, - EvidenceClass::Protocol => 70.0, - EvidenceClass::Speculative => 20.0, - } + f64::from(crate::verdict::sarif_rank(evidence)) } /// `div-by-literal-zero` -> `DivByLiteralZero`. SARIF wants a readable @@ -173,11 +164,13 @@ fn descriptor_name(id: &str) -> String { } fn message_text(f: &Finding) -> String { - if f.evidence() == EvidenceClass::Speculative { - format!("{}{}", f.message(), SPECULATIVE_SUFFIX) - } else { - f.message().to_string() - } + // The suffix is the published `demotion_suffix`, not a copy of it: a second + // literal here could drift from what the matrix promises. + format!( + "{}{}", + f.message(), + crate::verdict::demotion_suffix(f.evidence()) + ) } #[derive(Serialize)] @@ -486,6 +479,7 @@ pub fn render(report: &ScanReport) -> anyhow::Result { #[cfg(test)] mod tests { use super::*; + use crate::finding::Severity; use crate::report::{assert_golden, fixture}; /// The SARIF 2.1.0 schema, bundled rather than fetched: a validator that @@ -797,7 +791,7 @@ mod tests { assert_eq!(spec.severity(), Severity::Error); assert_eq!(level_of(spec), "note"); assert_eq!(rank_of(spec.evidence()), 20.0); - assert!(message_text(spec).ends_with(SPECULATIVE_SUFFIX)); + assert!(message_text(spec).ends_with(crate::verdict::demotion_suffix(spec.evidence()))); } #[test] diff --git a/crates/pons-core/src/report/validate.rs b/crates/pons-core/src/report/validate.rs index adeca2d1..2c6063f9 100644 --- a/crates/pons-core/src/report/validate.rs +++ b/crates/pons-core/src/report/validate.rs @@ -1,5 +1,13 @@ // SPDX-License-Identifier: MPL-2.0 -//! Fail-closed validation for the 0.4.0 scan exchange envelope. +//! Fail-closed validation for the scan exchange envelope. +//! +//! Two envelope versions are readable: the one this build emits +//! ([`super::json::SCHEMA_VERSION`]) and every version listed in +//! [`super::json::FROZEN_SCHEMA_VERSIONS`]. The version declared in the document +//! selects the schema it is validated against, so a consumer pinned to 0.4.0 +//! keeps a validator for its own artifacts, and a 0.4.0 declaration cannot +//! smuggle in a 0.5.0 key. Checks that only exist in the newer shape are gated +//! on the newer version rather than skipped silently. use crate::{ coverage::{InventoryEntry, Outcome}, evidence::Record, @@ -22,23 +30,68 @@ pub fn validate_reader(reader: impl std::io::Read) -> Result { validate_value(serde_json::from_reader(reader)?) } +/// Compile a bundled JSON schema. +/// +/// # Panics +/// +/// Panics if `text` is invalid JSON or the schema cannot be compiled. +fn compiled(text: &str) -> jsonschema::Validator { + let schema: Value = serde_json::from_str(text).expect("bundled scan schema must be JSON"); + jsonschema::validator_for(&schema).expect("bundled scan schema must compile") +} + +/// The schema that a declared envelope version is validated against. +/// +/// Deliberately a closed list and not "whatever parses": an unknown version +/// returns `None`, because validating an unknown shape against the nearest +/// schema is how a consumer ends up trusting a document nobody specified. +fn validator_for_version(version: &str) -> Option<&'static jsonschema::Validator> { + static CURRENT: std::sync::OnceLock = std::sync::OnceLock::new(); + static FROZEN_0_4_0: std::sync::OnceLock = std::sync::OnceLock::new(); + + if version == super::json::SCHEMA_VERSION { + return Some( + CURRENT.get_or_init(|| compiled(include_str!("../../../../schemas/scan-0.5.0.json"))), + ); + } + // One frozen generation so far. A second one adds its own schema file and + // its own OnceLock here; the dispatch stays explicit on purpose. + match version { + "0.4.0" => Some( + FROZEN_0_4_0 + .get_or_init(|| compiled(include_str!("../../../../schemas/scan-0.4.0.json"))), + ), + _ => None, + } +} + +/// Validate a report against its declared schema and check its accounting, +/// coverage and evidence records. Version 0.5.0 also requires matching +/// corroboration keys and actionable/demoted counts. +/// +/// Returns the verified coverage status: `no-applicable-input`, `partial` or +/// `complete-within-policy`. +/// +/// # Errors +/// +/// Returns an error for a missing or unsupported version, a schema violation +/// or inconsistent report data. Field deserialisation and evidence-record +/// validation errors propagate. fn validate_value(v: Value) -> Result { // Structural validation precedes semantic checks: hand-picked fields cannot // enforce required properties, enums or the closed envelope. - static SCHEMA: std::sync::OnceLock = std::sync::OnceLock::new(); - let validator = SCHEMA.get_or_init(|| { - let schema: Value = - serde_json::from_str(include_str!("../../../../schemas/scan-0.4.0.json")) - .expect("bundled scan schema must be JSON"); - jsonschema::validator_for(&schema).expect("bundled scan schema must compile") - }); + let version = v["schema_version"] + .as_str() + .context("missing schema_version")? + .to_string(); + let validator = validator_for_version(&version).context("unsupported report schema")?; validator .validate(&v) .map_err(|e| anyhow::anyhow!("report schema: {e} at {}", e.instance_path()))?; - ensure!( - v["schema_version"] == super::json::SCHEMA_VERSION, - "unsupported report schema" - ); + // The corroboration key and the demoted-count fields exist only from 0.5.0; + // their checks are gated rather than dropped so a 0.4.0 artifact stays + // valid while a 0.5.0 one cannot skip them. + let has_corroboration = version == super::json::SCHEMA_VERSION; let tool = required(&v, "tool")?; for k in ["name", "version", "source_commit", "source_dirty"] { ensure!( @@ -134,6 +187,7 @@ fn validate_value(v: Value) -> Result { v["counts"]["total"].as_u64() == Some(findings.len() as u64), "finding count mismatch" ); + let root = s["root"].as_str().context("missing scan root")?; for f in findings { let tier: Tier = serde_json::from_value(required(f, "tier")?.clone())?; let evidence: EvidenceClass = serde_json::from_value(required(f, "evidence")?.clone())?; @@ -160,6 +214,41 @@ fn validate_value(v: Value) -> Result { .any(|e| e.outcome == Outcome::Analysed && location["file"] == e.path), "finding outside analysed inventory" ); + // The corroboration key is minted from the root-relative path, so a + // path that cannot be made relative — an absolute file outside the + // scan root, or one climbing out with `..` — would mint a key encoding + // the build machine and matching nothing a consumer can derive. + // `relative_path` falls back to the path unchanged in that case, which + // is right for rendering and wrong for a join key, so it is refused. + if has_corroboration { + let file = location["file"].as_str().context("missing finding file")?; + let relative = crate::evidence::relative_path(file, root); + ensure!( + !std::path::Path::new(&relative).is_absolute() + && !relative.starts_with("../") + && relative != "..", + "finding path is not inside the scan root" + ); + } + if has_corroboration { + // Recomputed, not merely present: a key the grid deduplicates on is + // worthless if a producer can write an arbitrary one, and a keyed + // site that does not match its own location would silently fail to + // corroborate anything. + let file = location["file"].as_str().context("missing finding file")?; + let line = location["line_start"] + .as_u64() + .context("missing finding line")? as usize; + let expected = crate::corroborate::fingerprint_of( + &crate::evidence::relative_path(file, root), + line, + evidence, + ); + ensure!( + f["semantic_fingerprint"] == expected.as_str(), + "semantic fingerprint mismatch" + ); + } } for class in ["HEURISTIC", "DATAFLOW", "PROTOCOL", "SPECULATIVE"] { ensure!( @@ -168,6 +257,23 @@ fn validate_value(v: Value) -> Result { "evidence count mismatch" ); } + let demoted = findings + .iter() + .filter(|f| f["evidence"] == "SPECULATIVE") + .count() as u64; + if has_corroboration { + // The demotion has to hold numerically where a consumer sums: a + // document that counted a guess as actionable is rejected rather than + // silently re-added by the reader. + ensure!( + v["counts"]["demoted_speculative"].as_u64() == Some(demoted), + "demoted speculative count mismatch" + ); + ensure!( + v["counts"]["actionable"].as_u64() == Some(findings.len() as u64 - demoted), + "actionable count mismatch" + ); + } let contract = required(&v, "evidence_contract")?; ensure!( contract["schema_version"] == "1.0.0" && contract["execution_outcome"] == status, @@ -184,7 +290,6 @@ fn validate_value(v: Value) -> Result { let file = f["location"]["file"] .as_str() .context("missing finding file")?; - let root = s["root"].as_str().context("missing scan root")?; let notes = &inventory .iter() .find(|e| e.path == file) @@ -209,13 +314,42 @@ fn validate_value(v: Value) -> Result { #[cfg(test)] mod tests { + /// Every version the build claims to read has its own schema. + /// + /// The frozen list and the dispatch are two facts that must be one; this + /// makes a forgotten dispatch arm a failing test rather than a silent + /// fallback to the nearest shape. + #[test] + fn every_frozen_version_has_its_own_schema() { + for version in crate::report::json::FROZEN_SCHEMA_VERSIONS { + assert!( + super::validator_for_version(version).is_some(), + "{version} is listed as frozen but validator_for_version has no \ + arm for it" + ); + } + assert!( + super::validator_for_version(crate::report::json::SCHEMA_VERSION).is_some(), + "the current version has no schema" + ); + assert!( + super::validator_for_version("9.9.9").is_none(), + "an unknown version must not resolve to any schema" + ); + } + + /// The rendered report is validated against the schema its own + /// `schema_version` selects — through the same dispatch the validator + /// uses, so this cannot drift into checking the wrong generation. #[test] fn published_json_schema_accepts_native_report_and_rejects_tier_promotion() { - let schema: serde_json::Value = - serde_json::from_str(include_str!("../../../../schemas/scan-0.4.0.json")).unwrap(); - let validator = jsonschema::validator_for(&schema).unwrap(); let text = crate::report::json::render(&crate::report::fixture::report()).unwrap(); let mut value: serde_json::Value = serde_json::from_str(&text).unwrap(); + let version = value["schema_version"] + .as_str() + .expect("the report declares its version"); + let validator = super::validator_for_version(version) + .expect("this build's own version must have a schema"); assert!(validator.is_valid(&value)); value["findings"][0]["evidence"] = serde_json::json!("PROTOCOL"); assert!(!validator.is_valid(&value)); diff --git a/crates/pons-core/src/verdict.rs b/crates/pons-core/src/verdict.rs new file mode 100644 index 00000000..e848bcb4 --- /dev/null +++ b/crates/pons-core/src/verdict.rs @@ -0,0 +1,480 @@ +// SPDX-License-Identifier: MPL-2.0 +//! The severity × evidence matrix, published as data. +//! +//! An aggregating consumer — the estate's verdict grid — needs three things it +//! must not re-invent: which evidence class a tier implies, what a severity +//! plus a class is *worth* when findings are added up, and how a `SPECULATIVE` +//! finding is kept out of that sum. Appendix B of `docs/PLAN.adoc` states the +//! policy in prose; this module is the same policy as data, generated from the +//! constants the reporters themselves use: +//! +//! * [`sarif_level`] and [`sarif_rank`] are what `report/sarif.rs` emits; +//! * [`confidence`] is what `evidence.rs` writes into every record; +//! * [`fail_on_rank`] is what `--fail-on` compares; +//! * [`SPECULATIVE_SUFFIX`] is the demotion text the human and SARIF reporters +//! append. +//! +//! The document is committed at [`MATRIX_PATH`] and +//! `the_published_matrix_matches_the_code` fails if the two ever disagree, so a +//! consumer can read the file at a pinned commit and trust that it describes the +//! binary at that commit. +//! +//! Two numbers look similar and mean different things, which is why both are +//! published under distinct names. `sarif_rank` is Appendix E's *attention* +//! rank (50/80/70/20): it tells a viewer which class to look at first, and +//! `SPECULATIVE` keeps its 20 so it stays visible. `weight` is what a class +//! contributes to an aggregate total, and for `SPECULATIVE` it is **zero** — a +//! guess adds nothing to a sum, however visible it is in a table. + +use serde::Serialize; + +use crate::finding::{EvidenceClass, Severity, Tier}; +use crate::report::SPECULATIVE_SUFFIX; + +/// Version of the published matrix document, not of the tool. +pub const MATRIX_VERSION: &str = "1.0.0"; + +/// Where the generated document lives, relative to the repository root. +pub const MATRIX_PATH: &str = "schemas/verdict-matrix-1.0.0.json"; + +static ALL_SEVERITIES: [Severity; 3] = [Severity::Error, Severity::Warn, Severity::Info]; +static ALL_EVIDENCE: [EvidenceClass; 4] = [ + EvidenceClass::Heuristic, + EvidenceClass::Dataflow, + EvidenceClass::Protocol, + EvidenceClass::Speculative, +]; + +/// What `--fail-on` compares: `ERROR` 2 > `WARN` 1 > `INFO` 0. +/// +/// Kept separate from [`severity_weight`] on purpose. `INFO` is rank 0 here +/// because "meets or exceeds INFO" must select everything; it is weight 1 there +/// because an informational finding still counts once when findings are summed. +pub fn fail_on_rank(severity: Severity) -> u8 { + match severity { + Severity::Info => 0, + Severity::Warn => 1, + Severity::Error => 2, + } +} + +/// Ordinal weight of a severity in an aggregate total: `INFO` 1, `WARN` 2, +/// `ERROR` 3. Never zero — a finding that arrived is a finding. +pub fn severity_weight(severity: Severity) -> u32 { + match severity { + Severity::Info => 1, + Severity::Warn => 2, + Severity::Error => 3, + } +} + +/// Appendix E's fixed SARIF `rank` for an evidence class: how much attention +/// the class has earned, independent of severity. +pub fn sarif_rank(evidence: EvidenceClass) -> u32 { + match evidence { + EvidenceClass::Heuristic => 50, + EvidenceClass::Dataflow => 80, + EvidenceClass::Protocol => 70, + EvidenceClass::Speculative => 20, + } +} + +/// What a class contributes to an aggregate count. +/// +/// Equal to [`sarif_rank`] for the three verdict classes, and **zero** for +/// `SPECULATIVE`: the demotion is numeric as well as visual, so no summing +/// consumer can turn guesses into weight by adding them up. +pub fn evidence_weight(evidence: EvidenceClass) -> u32 { + if counts_toward_verdict(evidence) { + sarif_rank(evidence) + } else { + 0 + } +} + +/// Whether a finding in this class may contribute to a verdict at all. +pub fn counts_toward_verdict(evidence: EvidenceClass) -> bool { + evidence != EvidenceClass::Speculative +} + +/// The single aggregate weight for a (severity, evidence) pair. Zero for every +/// `SPECULATIVE` cell, whatever its severity claims. +pub fn aggregate_weight(evidence: EvidenceClass, severity: Severity) -> u32 { + severity_weight(severity) * evidence_weight(evidence) +} + +/// The SARIF level for a finding, which is also the visual demotion. +/// +/// The `SPECULATIVE` check comes **first** and unconditionally (Appendix E): a +/// guess must never arrive in a code-scanning UI wearing the colour of a +/// verdict, so an `ERROR`-severity speculative finding still renders as `note`. +pub fn sarif_level(evidence: EvidenceClass, severity: Severity) -> &'static str { + if evidence == EvidenceClass::Speculative { + return "note"; + } + level_for_severity(severity) +} + +/// The SARIF level a severity earns when its class is not demoted. +fn level_for_severity(severity: Severity) -> &'static str { + match severity { + Severity::Error => "error", + Severity::Warn => "warning", + Severity::Info => "note", + } +} + +/// The qualitative confidence label the evidence contract publishes per record. +pub fn confidence(evidence: EvidenceClass) -> &'static str { + match evidence { + EvidenceClass::Heuristic => "heuristic", + EvidenceClass::Dataflow => "model-dependent", + EvidenceClass::Protocol => "premise-dependent", + EvidenceClass::Speculative => "speculative", + } +} + +/// Which aggregate bucket a class feeds: the actionable count, or the demoted +/// one that is reported separately and never added to it. +pub fn count_bucket(evidence: EvidenceClass) -> &'static str { + if counts_toward_verdict(evidence) { + "actionable" + } else { + "demoted" + } +} + +/// The tier that fixes this class. The inverse of [`EvidenceClass::from`], kept +/// beside it so the published `tier` column cannot be typed by hand. +pub fn tier_of(evidence: EvidenceClass) -> Tier { + match evidence { + EvidenceClass::Heuristic => Tier::T0, + EvidenceClass::Dataflow => Tier::T1, + EvidenceClass::Protocol => Tier::T2, + EvidenceClass::Speculative => Tier::T3, + } +} + +/// The demotion text for a class: the exact suffix the human and SARIF +/// reporters append, and the empty string for every class that is a verdict. +pub fn demotion_suffix(evidence: EvidenceClass) -> &'static str { + if counts_toward_verdict(evidence) { + "" + } else { + SPECULATIVE_SUFFIX + } +} + +#[derive(Debug, Serialize)] +pub struct Matrix { + pub schema_version: &'static str, + pub producer: &'static str, + pub generated_from: &'static str, + pub severity_scale: Vec, + pub evidence_scale: Vec, + pub corroboration: Corroboration, + pub grid: Vec, + pub aggregation: Aggregation, +} + +#[derive(Debug, Serialize)] +pub struct SeverityRow { + pub severity: Severity, + pub weight: u32, + pub fail_on_rank: u8, + pub sarif_level_when_not_demoted: &'static str, +} + +#[derive(Debug, Serialize)] +pub struct EvidenceRow { + pub evidence: EvidenceClass, + pub tier: Tier, + pub sarif_rank: u32, + pub weight: u32, + pub confidence: &'static str, + pub counts_toward_verdict: bool, +} + +/// How a consumer mints the corroboration key for a finding, including a +/// finding from another producer. Mirrors [`crate::corroborate`]. +#[derive(Debug, Serialize)] +pub struct Corroboration { + pub scheme: &'static str, + pub hash: &'static str, + pub encoding: &'static str, + pub input_template: &'static str, + pub class_source: &'static str, + pub path_convention: &'static str, + pub line_convention: &'static str, + pub caution: &'static str, +} + +#[derive(Debug, Serialize)] +pub struct Cell { + pub severity: Severity, + pub evidence: EvidenceClass, + pub sarif_level: &'static str, + pub aggregate_weight: u32, + pub counts_toward_verdict: bool, + pub gate_eligible: bool, + pub count_bucket: &'static str, + pub demotion_suffix: &'static str, +} + +/// The counting rules, stated where a consumer will actually read them. +#[derive(Debug, Serialize)] +pub struct Aggregation { + pub demoted_evidence: Vec<&'static str>, + pub demoted_sarif_level: &'static str, + pub demotion_suffix: &'static str, + pub demoted_count_field: &'static str, + pub actionable_count_field: &'static str, + pub total_count_field: &'static str, + pub total_includes_demoted: bool, + pub actionable_definition: &'static str, + pub rules: Vec<&'static str>, +} + +/// Build the published matrix from the constants the reporters use. +pub fn matrix() -> Matrix { + let mut grid = Vec::new(); + for severity in ALL_SEVERITIES.iter().copied() { + for evidence in ALL_EVIDENCE.iter().copied() { + grid.push(Cell { + severity, + evidence, + sarif_level: sarif_level(evidence, severity), + aggregate_weight: aggregate_weight(evidence, severity), + counts_toward_verdict: counts_toward_verdict(evidence), + // A finding is gate-eligible when `--fail-on` may act on it, + // which is exactly when it counts toward a verdict: the CLI + // skips SPECULATIVE before comparing severity. + gate_eligible: counts_toward_verdict(evidence), + count_bucket: count_bucket(evidence), + demotion_suffix: demotion_suffix(evidence), + }); + } + } + + Matrix { + schema_version: MATRIX_VERSION, + producer: "pons", + generated_from: "crates/pons-core/src/verdict.rs", + severity_scale: ALL_SEVERITIES + .iter() + .map(|&severity| SeverityRow { + severity, + weight: severity_weight(severity), + fail_on_rank: fail_on_rank(severity), + sarif_level_when_not_demoted: level_for_severity(severity), + }) + .collect(), + evidence_scale: ALL_EVIDENCE + .iter() + .map(|&evidence| EvidenceRow { + evidence, + tier: tier_of(evidence), + sarif_rank: sarif_rank(evidence), + weight: evidence_weight(evidence), + confidence: confidence(evidence), + counts_toward_verdict: counts_toward_verdict(evidence), + }) + .collect(), + corroboration: Corroboration { + scheme: crate::corroborate::SCHEME, + hash: "blake3", + encoding: "64 lowercase hexadecimal characters", + input_template: "scheme, then one NUL, then the root-relative path, one NUL, \ + the 1-based start line in decimal, one NUL, then the evidence \ + class name; the whole string is hashed", + class_source: "The producer's own evidence class, or the class the consuming grid \ + has explicitly mapped a foreign producer onto. Never a raw foreign \ + severity, and never an upgrade of one.", + path_convention: "Root-relative, '/' separated, no leading './' — the same spelling \ + as evidence_contract.records[].path, so a key survives a checkout \ + move.", + line_convention: "1-based line of the finding's start; a finding spanning lines is \ + keyed on the line it starts on.", + caution: "Equal keys assert the same path, line and evidence class and nothing more. \ + They are not proof of semantic equivalence, and a demoted class \ + corroborates no verdict.", + }, + grid, + aggregation: Aggregation { + demoted_evidence: vec!["SPECULATIVE"], + demoted_sarif_level: "note", + demotion_suffix: SPECULATIVE_SUFFIX, + demoted_count_field: "counts.demoted_speculative", + actionable_count_field: "counts.actionable", + total_count_field: "counts.total", + total_includes_demoted: true, + actionable_definition: "counts.total - counts.demoted_speculative", + rules: vec![ + "A demoted finding contributes zero aggregate_weight and is counted only in \ + counts.demoted_speculative; it is never part of counts.actionable.", + "Demotion is unconditional and overrides the finding's own severity: ERROR + \ + SPECULATIVE is still not a verdict, and its SARIF level is 'note'.", + "An aggregating consumer reports counts.actionable as its finding total, or \ + reports both numbers; a total that includes demoted findings must say so.", + "Corroboration collapses a site ACROSS producers, not findings within one \ + report: two producers minting the same semantic_fingerprint are one \ + corroborated site, while counts.* count findings, so two Pons rules at one \ + site in one class are two findings sharing one key.", + ], + }, + } +} + +/// The published document, with a trailing newline so it is a well-formed text +/// stream. Pretty-printed for review, like the scan envelope. +pub fn matrix_json() -> String { + let mut out = + serde_json::to_string_pretty(&matrix()).expect("the verdict matrix is plain data"); + out.push('\n'); + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn published() -> String { + std::fs::read_to_string( + std::path::Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../..") + .join(MATRIX_PATH), + ) + .expect("the committed verdict matrix must be readable") + } + + /// The point of the file: a consumer reads it at a pin, so it must be the + /// document the code would emit at that pin, byte for byte. + #[test] + fn the_published_matrix_matches_the_code() { + let generated = matrix_json(); + if std::env::var_os("PONS_BLESS").is_some() { + std::fs::write( + std::path::Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../..") + .join(MATRIX_PATH), + &generated, + ) + .expect("could not write the verdict matrix"); + return; + } + assert_eq!( + published(), + generated, + "{MATRIX_PATH} drifted from the code — inspect the diff, then re-run \ + with PONS_BLESS=1 only if the new document is correct" + ); + } + + #[test] + fn every_severity_and_class_pair_appears_exactly_once() { + let grid = matrix().grid; + assert_eq!(grid.len(), 12); + for severity in ALL_SEVERITIES.iter().copied() { + for evidence in ALL_EVIDENCE.iter().copied() { + let hits = grid + .iter() + .filter(|c| c.severity == severity && c.evidence == evidence) + .count(); + assert_eq!( + hits, 1, + "grid cell {severity:?} x {evidence:?} is not unique" + ); + } + } + } + + #[test] + fn a_speculative_cell_weighs_nothing_at_any_severity() { + for cell in matrix().grid { + if cell.evidence != EvidenceClass::Speculative { + assert!(cell.counts_toward_verdict && cell.gate_eligible); + assert_eq!(cell.count_bucket, "actionable"); + assert_eq!(cell.demotion_suffix, ""); + assert!(cell.aggregate_weight > 0, "{cell:?} weighs nothing"); + continue; + } + assert_eq!(cell.aggregate_weight, 0, "{cell:?}"); + assert!(!cell.counts_toward_verdict, "{cell:?}"); + assert!(!cell.gate_eligible, "{cell:?}"); + assert_eq!(cell.count_bucket, "demoted"); + assert_eq!(cell.demotion_suffix, SPECULATIVE_SUFFIX); + assert_eq!( + cell.sarif_level, "note", + "an ERROR-severity speculation must still render as a note" + ); + } + } + + #[test] + fn speculative_stays_visible_in_sarif_even_though_it_weighs_nothing() { + // The two numbers are different jobs; if someone "fixes" the weight by + // setting the rank to 0 too, the demotion stops being visible in a + // SARIF viewer and the finding silently disappears instead. + assert_eq!(sarif_rank(EvidenceClass::Speculative), 20); + assert_eq!(evidence_weight(EvidenceClass::Speculative), 0); + } + + #[test] + fn a_published_level_is_appendix_e_pinned() { + // Pinned literals, because the SARIF reporter now delegates here: a + // "fix" that made every level "warning" would otherwise be invisible. + assert_eq!( + sarif_level(EvidenceClass::Dataflow, Severity::Error), + "error" + ); + assert_eq!( + sarif_level(EvidenceClass::Dataflow, Severity::Warn), + "warning" + ); + assert_eq!(sarif_level(EvidenceClass::Dataflow, Severity::Info), "note"); + assert_eq!( + sarif_level(EvidenceClass::Protocol, Severity::Error), + "error" + ); + assert_eq!( + sarif_level(EvidenceClass::Speculative, Severity::Error), + "note" + ); + assert_eq!( + sarif_level(EvidenceClass::Speculative, Severity::Info), + "note" + ); + } + + #[test] + fn the_two_ordinal_scales_stay_distinct() { + // INFO is fail-on rank 0 but weight 1. Collapsing them would make every + // informational finding weigh nothing, which is a different bug from + // the demotion above and must not be "fixed" the same way. + assert_eq!(fail_on_rank(Severity::Info), 0); + assert_eq!(severity_weight(Severity::Info), 1); + assert_eq!(fail_on_rank(Severity::Error), 2); + assert_eq!(severity_weight(Severity::Error), 3); + } + + #[test] + fn the_tier_and_class_columns_are_inverses() { + for evidence in ALL_EVIDENCE { + assert_eq!(EvidenceClass::from(tier_of(evidence)), evidence); + } + } + + #[test] + fn corroboration_fields_match_the_implementation() { + // The document tells a consumer how to mint the key without pons. If + // that prose stops describing what corroborate.rs actually hashes, the + // grid starts minting keys that never match a pons finding. + let c = matrix().corroboration; + assert_eq!(c.scheme, crate::corroborate::SCHEME); + let sample = crate::corroborate::canonical_input("a.py", 7, EvidenceClass::Dataflow); + assert_eq!( + sample, "pons-corroboration-v1\u{0}a.py\u{0}7\u{0}DATAFLOW", + "the template in the published document must describe this string" + ); + } +} diff --git a/crates/pons-core/tests/golden/report-0.4.0.json b/crates/pons-core/tests/golden/report-0.4.0.json new file mode 100644 index 00000000..c2db8cf5 --- /dev/null +++ b/crates/pons-core/tests/golden/report-0.4.0.json @@ -0,0 +1,198 @@ +{ + "schema_version": "0.4.0", + "tool": { + "name": "pons", + "version": "0.1.0", + "source_commit": "TEST-COMMIT", + "source_dirty": "TEST-DIRTY" + }, + "execution": { + "status": "completed", + "exit_code": null + }, + "scanned": { + "root": ".", + "coverage_status": "partial", + "inventory": [], + "coverage_failures": [], + "files": 7, + "suppressed_findings": 0, + "languages": [ + "python", + "typescript" + ], + "transcoded": [], + "skipped": [ + { + "path": "app/broken.py", + "reason": "could not read: No such file or directory (os error 2)" + } + ] + }, + "counts": { + "by_evidence": { + "HEURISTIC": 1, + "DATAFLOW": 1, + "PROTOCOL": 1, + "SPECULATIVE": 1 + }, + "total": 4 + }, + "findings": [ + { + "rule_id": "dead-store", + "tier": "T1", + "evidence": "DATAFLOW", + "severity": "ERROR", + "location": { + "file": "app/svc.py", + "byte_start": 812, + "byte_end": 819, + "line_start": 41, + "col_start": 5, + "line_end": 41, + "col_end": 12 + }, + "message": "value assigned to `total` is never read before it is overwritten", + "evidence_note": "live-variable analysis: `total` not in LiveOut after this store", + "counter_condition": null + }, + { + "rule_id": "div-by-literal-zero", + "tier": "T0", + "evidence": "HEURISTIC", + "severity": "WARN", + "location": { + "file": "app/svc.py", + "byte_start": 904, + "byte_end": 911, + "line_start": 47, + "col_start": 5, + "line_end": 47, + "col_end": 12 + }, + "message": "division by a literal zero", + "evidence_note": "constant zero on the right-hand side", + "counter_condition": "the surrounding branch is provably dead" + }, + { + "rule_id": "suppress-then-emit", + "tier": "T2", + "evidence": "PROTOCOL", + "severity": "WARN", + "location": { + "file": "lib/util.ts", + "byte_start": 120, + "byte_end": 127, + "line_start": 9, + "col_start": 5, + "line_end": 9, + "col_end": 12 + }, + "message": "channel enters `suppressed` and reaches `prompt` with no `set_loud`", + "evidence_note": "enters `suppressed` at line 4; reaches `prompt` at line 9", + "counter_condition": "a different channel instance reaches the prompt" + }, + { + "rule_id": "possible-superlinear", + "tier": "T3", + "evidence": "SPECULATIVE", + "severity": "ERROR", + "location": { + "file": "lib/util.ts", + "byte_start": 300, + "byte_end": 307, + "line_start": 22, + "col_start": 5, + "line_end": 22, + "col_end": 12 + }, + "message": "nested iteration may be superlinear in the input size", + "evidence_note": "two loops over collections derived from the same source", + "counter_condition": null + } + ], + "evidence_contract": { + "schema_version": "1.0.0", + "target": { + "commit": null, + "dirty": null + }, + "execution_outcome": "partial", + "parser_inventory": [], + "records": [ + { + "fingerprint": "074af7e827a6865eb8319f120dccf1308227aa3be85895acc899e883969f0e16", + "producer": "pons", + "correlation_group": "pons:T1", + "rule": "dead-store", + "tier": "T1", + "evidence": "DATAFLOW", + "confidence": "model-dependent", + "path": "app/svc.py", + "line": 41, + "trace": "live-variable analysis: `total` not in LiveOut after this store", + "assumptions": [ + "Only registered rule semantics and declared parser/model coverage; no runtime safety certification" + ], + "unknown_edges": [], + "counterconditions": [] + }, + { + "fingerprint": "1b592d3e020037ca9be8ed8f95b4dd78679e41f67d858ba2e6dbe08b749efa38", + "producer": "pons", + "correlation_group": "pons:T0", + "rule": "div-by-literal-zero", + "tier": "T0", + "evidence": "HEURISTIC", + "confidence": "heuristic", + "path": "app/svc.py", + "line": 47, + "trace": "constant zero on the right-hand side", + "assumptions": [ + "Only registered rule semantics and declared parser/model coverage; no runtime safety certification" + ], + "unknown_edges": [], + "counterconditions": [ + "the surrounding branch is provably dead" + ] + }, + { + "fingerprint": "d84849476219f034f56f7324f010a1ac40f17df3f9442e8b9ecdf7f83b4a4bf4", + "producer": "pons", + "correlation_group": "pons:T2", + "rule": "suppress-then-emit", + "tier": "T2", + "evidence": "PROTOCOL", + "confidence": "premise-dependent", + "path": "lib/util.ts", + "line": 9, + "trace": "enters `suppressed` at line 4; reaches `prompt` at line 9", + "assumptions": [ + "Only registered rule semantics and declared parser/model coverage; no runtime safety certification" + ], + "unknown_edges": [], + "counterconditions": [ + "a different channel instance reaches the prompt" + ] + }, + { + "fingerprint": "03c613c8d0a7ea9849732b22ee2756c5b0cb6f75cdf7e519dea94fdf7abb265c", + "producer": "pons", + "correlation_group": "pons:T3", + "rule": "possible-superlinear", + "tier": "T3", + "evidence": "SPECULATIVE", + "confidence": "speculative", + "path": "lib/util.ts", + "line": 22, + "trace": "two loops over collections derived from the same source", + "assumptions": [ + "Only registered rule semantics and declared parser/model coverage; no runtime safety certification" + ], + "unknown_edges": [], + "counterconditions": [] + } + ] + } +} diff --git a/crates/pons-core/tests/golden/report.json b/crates/pons-core/tests/golden/report.json index c2db8cf5..1f891a7a 100644 --- a/crates/pons-core/tests/golden/report.json +++ b/crates/pons-core/tests/golden/report.json @@ -1,5 +1,5 @@ { - "schema_version": "0.4.0", + "schema_version": "0.5.0", "tool": { "name": "pons", "version": "0.1.0", @@ -36,7 +36,9 @@ "PROTOCOL": 1, "SPECULATIVE": 1 }, - "total": 4 + "total": 4, + "actionable": 3, + "demoted_speculative": 1 }, "findings": [ { @@ -55,7 +57,8 @@ }, "message": "value assigned to `total` is never read before it is overwritten", "evidence_note": "live-variable analysis: `total` not in LiveOut after this store", - "counter_condition": null + "counter_condition": null, + "semantic_fingerprint": "a54bb8fa2b7f74ff5c9d1fa745ac22e61abb49847ddbac0bd220fef2442baaf1" }, { "rule_id": "div-by-literal-zero", @@ -73,7 +76,8 @@ }, "message": "division by a literal zero", "evidence_note": "constant zero on the right-hand side", - "counter_condition": "the surrounding branch is provably dead" + "counter_condition": "the surrounding branch is provably dead", + "semantic_fingerprint": "59709a1c67d882050dcb5d14464e720154430b85a860c89a3bede92bc6615f31" }, { "rule_id": "suppress-then-emit", @@ -91,7 +95,8 @@ }, "message": "channel enters `suppressed` and reaches `prompt` with no `set_loud`", "evidence_note": "enters `suppressed` at line 4; reaches `prompt` at line 9", - "counter_condition": "a different channel instance reaches the prompt" + "counter_condition": "a different channel instance reaches the prompt", + "semantic_fingerprint": "0ca3019e7ced8cb406e68d08411f62c00e7b100369bc96b3aed3afd02bff7b86" }, { "rule_id": "possible-superlinear", @@ -109,7 +114,8 @@ }, "message": "nested iteration may be superlinear in the input size", "evidence_note": "two loops over collections derived from the same source", - "counter_condition": null + "counter_condition": null, + "semantic_fingerprint": "ebf407bdf66ad80a109f8ecd833f1af4c163d3ac09f5fbecaef4841fa63f10ed" } ], "evidence_contract": { diff --git a/docs/COVERAGE-MIGRATION.adoc b/docs/COVERAGE-MIGRATION.adoc index f3799458..258943c2 100644 --- a/docs/COVERAGE-MIGRATION.adoc +++ b/docs/COVERAGE-MIGRATION.adoc @@ -38,8 +38,10 @@ an externally time/memory-bounded worker. Latin-1 fallback is a byte transcoder, not a language adapter; byte offsets remain decoded-string offsets and SARIF omits artifact byte offsets. -Consumers should first require schema 0.4.0, inspect execution/coverage outcomes, -and only then apply finding severity policy. Do not translate an empty findings +Consumers should first require schema 0.4.0 or later (0.5.0 is current; 0.4.0 is +frozen and still accepted by `pons validate-report`, so a pinned artifact keeps +validating — see `docs/EVIDENCE-CONTRACT.adoc`), inspect execution/coverage +outcomes, and only then apply finding severity policy. Do not translate an empty findings array into a pass. Haskell and Elixir remain unsupported. Configuration files are inventory only; JSON/TOML syntax recognition is not security analysis. diff --git a/docs/EVIDENCE-CONTRACT.adoc b/docs/EVIDENCE-CONTRACT.adoc index 8ae1ca6e..ee6f82cd 100644 --- a/docs/EVIDENCE-CONTRACT.adoc +++ b/docs/EVIDENCE-CONTRACT.adoc @@ -1,5 +1,5 @@ // SPDX-License-Identifier: CC-BY-SA-4.0 -= Exchange contract 1.0.0 / scan envelope 0.4.0 += Scan envelope 0.5.0 (0.4.0 frozen) / exchange contract 1.0.0 `evidence_contract` carries the observed target Git commit (null outside Git), execution outcome, native parser inventory and independently identifiable @@ -14,10 +14,120 @@ state as well as package version. Git provenance is observational, not signed. Each record includes producer, correlation group, rule, root-relative path, line, tier, evidence class, qualitative confidence, trace, assumptions, counterconditions and unknown edges. Pons-native T0 records share an AST -heuristic correlation group. Agreement with panic-attack or Sanctify must not -be counted as independent proof merely because producer names differ: shared +heuristic correlation group. Agreement with panic-attack or Sanctify must not be +counted as independent proof merely because producer names differ: shared rules, models, parser sources and fixtures can correlate their evidence. +== Versions, and what "frozen" means + +`schemas/index.json` is the machine-readable inventory: per artifact, its +version, file, whether that version is `frozen` or `current`, and the BLAKE3 +digest of the bytes Pons was tested against. Three rules hold it together: + +* A **frozen** version is never edited and never re-issued. `scan-0.4.0.json` is + byte-pinned by `FROZEN_0_4_0_SCHEMA_BLAKE3` in + `crates/pons-core/src/report/json.rs`, so an edit fails a test rather than + changing what a pinned consumer reads. +* A change is a **new version in a new file**, listed beside the frozen ones. +* `pons validate-report` accepts every frozen version as well as the current one, + and the version declared *inside the document* selects the schema it is + validated against. A 0.4.0 declaration carrying a 0.5.0 key is rejected, and a + report pinned at 0.4.0 still validates. + +`0.5.0` is the envelope emitted by this build. Against `0.4.0` it adds three +keys, all appended and all additive — no key was renamed, moved or retyped: + +[cols="1,2",options="header"] +|=== +|Key |Why it exists + +|`findings[].semantic_fingerprint` +|The cross-producer corroboration key. A verdict grid can mark one site once +instead of adding Pons's finding to panic-attack's or Hypatia's at the same line. + +|`counts.actionable` +|`counts.total` minus the demoted count: the number an aggregating consumer +sums. Published rather than left to subtraction, so the demotion is a number in +the document rather than a convention a consumer might miss. + +|`counts.demoted_speculative` +|How many findings are demoted as guesses. Equal to `by_evidence.SPECULATIVE`, +published twice on purpose: a consumer reading only the aggregate must see it. +`validate-report` rejects a document whose `actionable + demoted_speculative` +does not account for `counts.total`, or whose demoted count disagrees with +`by_evidence`. +|=== + +== Corroborating a Pons finding against another producer + +Each finding carries `semantic_fingerprint`: 64 lowercase hexadecimal characters, +BLAKE3 over + +.... +pons-corroboration-v1 NUL root-relative-path NUL 1-based-start-line NUL evidence-class +.... + +with NUL (`0x00`) as the field separator and the evidence class spelled exactly +as the envelope spells it (`HEURISTIC`, `DATAFLOW`, `PROTOCOL`, `SPECULATIVE`). +The path is the same root-relative, `/`-separated spelling records use, so a key +survives a checkout move. A consumer can mint the same key for a foreign +producer's finding without running Pons; `schemas/verdict-matrix-1.0.0.json` +publishes the recipe and the token that identifies it. + +What the key is for, and what it is not: + +* Same key means **same path, same line, same evidence class** — a site, not a + defect. Two Pons rules firing on one line in one class share a key by design, + and so do two producers. The grid counts that site once and keeps both + findings' provenance. +* It deliberately **ignores the rule id, the producer and the severity**, so it + stays stable when a rule's severity is tuned. Severity is advisory; the + evidence class is the honesty invariant. +* It is **not** the record fingerprint. `evidence_contract.records[].fingerprint` + identifies one finding (rule id plus a whitespace-folded anchor) and survives + blank-line insertion; this key identifies one site and survives a rule rename. + Both are identifiers, neither is a semantic-equivalence proof. +* Every finding's `location.file` must be expressible **inside** the scan root. + A path that cannot be made relative — an absolute file elsewhere on the + machine, or one climbing out with `..` — is rejected rather than substituted + unchanged, because a key minted from it would encode the build machine and + match nothing a consumer can derive. +* The `counts.*` figures count **findings**, while corroboration collapses + **sites across producers**. Two Pons rules firing on one line in one class are + two findings sharing one key; a Pons finding and another producer's finding at + the same site are one corroborated site. Both directions are published in + `schemas/verdict-matrix-1.0.0.json` (`aggregation.rules`). +* A `SPECULATIVE` key **corroborates nothing**. Agreement between two guesses is + two guesses; the demoted bucket is excluded from every verdict. +* Agreement is corroboration, not independent proof. Shared rules, models, + parser sources and fixtures correlate evidence across producers, and + `validate-report` cannot detect that — only the consumer, holding both + provenance records, can. + +== Severity × evidence, published as data + +`schemas/verdict-matrix-1.0.0.json` states, for all twelve +(severity, evidence class) pairs, what an aggregating consumer must do with a +finding: + +* `aggregate_weight` — `severity weight (INFO 1, WARN 2, ERROR 3) × evidence + weight (HEURISTIC 50, DATAFLOW 80, PROTOCOL 70, SPECULATIVE 0)`. **Zero for + every speculative cell**, whatever severity the finding claims: a guess adds + nothing to a sum. +* `sarif_level`, `counts_toward_verdict`, `gate_eligible`, `count_bucket` + (`actionable` or `demoted`) and `demotion_suffix`. + +`sarif_rank` (50/80/70/20) is also published and is a *different number* from +the weight: it is how much attention Appendix E's `rank` asks a viewer to pay, +and speculative findings keep their 20 so they stay visible. Visible, not +counted — collapsing the two would either hide guesses or inflate totals. + +The document is generated from `crates/pons-core/src/verdict.rs`, which is also +where the SARIF reporter, the SARIF rank table, the confidence labels and +`--fail-on` get their values; `the_published_matrix_matches_the_code` fails if +the committed file and the code disagree. + + Fingerprints use BLAKE3 over rule ID, root-relative path and a whitespace-folded source anchor plus occurrence ordinal. Checkout relocation and added blank lines preserve them. Renames, changed anchors and insertion of an identical diff --git a/docs/INTEROPERABILITY.adoc b/docs/INTEROPERABILITY.adoc index 2778c9f9..6f6ecbd9 100644 --- a/docs/INTEROPERABILITY.adoc +++ b/docs/INTEROPERABILITY.adoc @@ -7,7 +7,7 @@ Companion repositories are not modified by this development branch. [cols="1,2,2",options="header"] |=== |Companion |Current boundary |Next contract -|panic-attack |Separate executable; both expose SARIF runs. Pons does not import its crate or relabel its findings. |A report combiner may concatenate runs while preserving tool identity, version, rules and evidence; deduplication must retain provenance. +|panic-attack |Separate executable; both expose SARIF runs. Pons does not import its crate or relabel its findings. |A report combiner may concatenate runs while preserving tool identity, version, rules and evidence; deduplication must retain provenance. Pons supplies the key for that deduplication: `findings[].semantic_fingerprint` (path, line, evidence class) and the recipe in `schemas/verdict-matrix-1.0.0.json`. A foreign finding may be mapped onto the published severity × evidence matrix, but the mapping is the consumer's claim, and a demoted class corroborates no verdict. |Invariant Path |`pons challenge FILE` is **planned**, not present in the current binary. The deferred 2026-09-09 fork invoked `invariant-path scan --file ABSOLUTE --json` (or `invariant-path-cli`) and returned unreviewed JSON without `--write`; that adapter was not merged. |Version negotiation and a validated annotation schema before stronger mapping. Any future adapter must validate envelope fields, not merely array shape. |Authority Watch (metadatastician) |Inspected offline/manual precision core; automated monitoring and production signing remain gated. No Pons ingestion adapter or automatic update job is implemented. |Candidate bundle: source URI, document revision/digest, observed change, proposed failure type, rationale, fixtures, review status. Separate observation, interpretation, approval and publication. Pin an approved bundle; never hot-load specification changes as unquestionable rules. |LOL |The current binary does not implement localisation checks. The deferred fork accepted local JSON records shaped as an array of code/name objects; LOL's inspected catalogue uses iso639_3/iso639_1 and a separate locale ABI, and no compatible production export was verified. |Versioned export/conversion with ISO identifiers, BCP47 tags, aliases and fallback metadata. Keep language identity distinct from spelling dictionaries. diff --git a/docs/PLAN.adoc b/docs/PLAN.adoc index d5332bbc..c77ad3b3 100644 --- a/docs/PLAN.adoc +++ b/docs/PLAN.adoc @@ -233,6 +233,20 @@ mixed fixture directory (Python + JS/TS + Rust). All boxes green ⇒ tag * A rule that fires on its own negative corpus is *demoted* (tier down and/or severity down) or removed — never merged as-is (kickoff falsifier discipline). +*Published as data.* Consumers must not re-derive any of this. +`schemas/verdict-matrix-1.0.0.json` carries the twelve (severity, evidence) +cells with `sarif_level`, `aggregate_weight`, `counts_toward_verdict`, +`gate_eligible`, `count_bucket` and `demotion_suffix`, plus the two scales and +the corroboration-key recipe. It is generated from +`crates/pons-core/src/verdict.rs` — the same module the SARIF reporter, +`--fail-on` and the evidence-contract confidence labels read — and a drift test +fails if the committed file and the code disagree. + +Two numbers that look alike are deliberately distinct: `sarif_rank` +(50/80/70/20) is Appendix E's *attention* rank, and `weight` is what a class +contributes to an aggregate total, which is **zero** for `SPECULATIVE`. A guess +stays visible and adds nothing. + == Appendix C — The falsifier gate (CI mechanics) Implement as a single Rust integration test `tests/falsifier.rs` (plus a @@ -385,16 +399,22 @@ unless `--lang` forces one. [source,json] ---- { - "schema_version": "0.3.0", - "tool": { "name": "pons", "version": "" }, + "schema_version": "0.5.0", + "tool": { "name": "pons", "version": "", + "source_commit": "", "source_dirty": "true|false|unknown" }, + "execution": { "status": "completed", "exit_code": 0 }, "scanned": { "root": "", + "coverage_status": "complete-within-policy", + "inventory": [], "coverage_failures": [], "files": 123, "suppressed_findings": 2, "languages": ["python","rust"], "transcoded": [], "skipped": [ { "path": "app/broken.py", "reason": "could not read: No such file or directory (os error 2)" } ] }, "counts": { "by_evidence": { "HEURISTIC": 4, "DATAFLOW": 2, "PROTOCOL": 1, "SPECULATIVE": 3 }, - "total": 10 }, + "total": 10, + "actionable": 7, + "demoted_speculative": 3 }, "findings": [ { "rule_id": "dead-store", @@ -407,15 +427,22 @@ unless `--lang` forces one. "line_end": 41, "col_end": 12 }, "message": "value assigned to `total` is never read before it is overwritten", "evidence_note": "live-variable analysis: `total` not in LiveOut after this store", - "counter_condition": null + "counter_condition": null, + "semantic_fingerprint": "<64 lowercase hex: blake3 of scheme, path, line, evidence class>" } - ] + ], + "evidence_contract": { "schema_version": "1.0.0", "...": "see EVIDENCE-CONTRACT.adoc" } } ---- Field order stable (golden-file tested). `counter_condition` is `null` when -absent, never omitted. This envelope is compatible in shape with panic-attack's -JSON habit (`schema_version` + typed arrays), easing the ADR-0004 records sink. +absent, never omitted. `semantic_fingerprint` is appended *after* +`counter_condition`: the first eight keys of a 0.5.0 finding are exactly the +0.4.0 keys, in 0.4.0's order, so a reader written against 0.4.0 reads a complete +finding before it meets the new key. `evidence_contract` is abbreviated above; +the keys it must carry are in `schemas/evidence-1.0.0.json`, unchanged by this +bump. This envelope is compatible in shape with panic-attack's JSON habit +(`schema_version` + typed arrays), easing the ADR-0004 records sink. `scanned.skipped` names every part of the tree that was found but not analysed: a subdirectory the walker could not enumerate, a file that could not be read, a @@ -443,6 +470,21 @@ is version-visible from its first release. It moved `0.2.0 -> 0.3.0` when `scanned.suppressed_findings` and `scanned.transcoded` were added, making filtration and encoding uncertainty visible to consumers. Purely additive, but version-visible for the same reason. +It moved `0.3.0 -> 0.4.0` for the coverage inventory, execution outcome and +nested evidence contract. +It moved `0.4.0 -> 0.5.0` when `findings[].semantic_fingerprint`, +`counts.actionable` and `counts.demoted_speculative` were appended — the three +things an aggregating verdict grid needs and cannot derive. Additive in both +directions: a 0.4.0-shaped reading of a 0.5.0 document still works, and the +bump is still taken, because a consumer must be able to tell whether a report +carries the corroboration key at all. + +`0.4.0` is **frozen**, not superseded-in-place: `schemas/scan-0.4.0.json` is +byte-pinned (a test recomputes its digest), `pons validate-report` still accepts +a 0.4.0 report against that schema, and the version declared inside the document +selects the schema — so a 0.4.0 declaration cannot carry a 0.5.0 key. +`schemas/index.json` lists every published version, its file and its digest; +`schemas/README.adoc` states the freeze rule and how to add a version. A version constant that nothing checks is a comment, not a contract, so the key set above is asserted against `SCHEMA_VERSION` by @@ -457,7 +499,8 @@ the whole suite passing at an unchanged `schema_version`. The PHP/WordPress upgrade authorises the implementation and explicit limits in COVERAGE-MIGRATION.adoc, PHP-ADAPTER.adoc, PROTOCOL-MODELS.adoc and EVIDENCE-CONTRACT.adoc. These supersede older envelope/exit/suppression wording: -scan schema is 0.4.0; no-applicable-input exits 2; native PHP is a bounded adapter; +scan schema is 0.5.0 with 0.4.0 frozen; no-applicable-input exits 2; native PHP +is a bounded adapter; unsupported languages are accounted for; source-derived universal protocol verification remains unimplemented. Measured evidence is kept separately from these design commitments. No release tag or claim of completion of the entire diff --git a/docs/PURPOSE-AND-PROOF.adoc b/docs/PURPOSE-AND-PROOF.adoc index ea95f6a2..ad58b969 100644 --- a/docs/PURPOSE-AND-PROOF.adoc +++ b/docs/PURPOSE-AND-PROOF.adoc @@ -50,7 +50,13 @@ describes its scope; this repository does not audit its advertised guarantees. The interface is link:EVIDENCE-CONTRACT.adoc[versioned evidence], not a shared scanner engine. Preserve producer, rule, parser and target versions, premises, correlation group and execution outcome. Agreement between correlated analyses -is corroboration, not independent proof. Consumer integrations remain unfinished. +is corroboration, not independent proof. When two producers agree, the agreement +is expressed by a published key rather than by a merged count: pons emits +`findings[].semantic_fingerprint` (root-relative path, line, evidence class), the +same recipe is published in `schemas/verdict-matrix-1.0.0.json`, and a site both +producers name is counted once. A `SPECULATIVE` key corroborates nothing, and a +corroborated finding is still corroboration — the consumer keeps both producers' +provenance. Consumer integrations remain unfinished. == How far the solver and proof work has got diff --git a/docs/SCOPE.adoc b/docs/SCOPE.adoc index 324d2b77..e300a345 100644 --- a/docs/SCOPE.adoc +++ b/docs/SCOPE.adoc @@ -36,7 +36,7 @@ a diagnosis. No philosophy qualification is expected. models. These check supplied premises/events, not source-derived typestate. * Scoped configuration and suppression: `pons.toml`, `--config`, rule/path allowances, inline comment allowances, and suppression accounting. -* Human, JSON 0.4.0 and SARIF 2.1.0 output. SARIF carries invocation +* Human, JSON 0.5.0 (0.4.0 frozen and still validatable) and SARIF 2.1.0 output. SARIF carries invocation notifications for skipped paths; JSON carries skip, transcode and suppression accounting; a transcoded file omits misleading artifact byte offsets. * Generated help/man/catalogue, `doctor`, and a CLI/rule/doc drift test. diff --git a/schemas/README.adoc b/schemas/README.adoc new file mode 100644 index 00000000..110cf0d4 --- /dev/null +++ b/schemas/README.adoc @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 += Published contracts + +Everything in this directory is a contract a consumer may depend on at a pinned +commit. `index.json` is the machine-readable inventory of it; this page is the +human one. The prose contract is link:../docs/EVIDENCE-CONTRACT.adoc[]. + +[cols="2,1,1,1,3",options="header"] +|=== +|Artifact |Version |File |Status |What it is + +|Scan envelope +|0.4.0 +|`scan-0.4.0.json` +|frozen +|The previous JSON envelope. Never emitted by this build, still accepted by +`pons validate-report`. + +|Scan envelope +|0.5.0 +|`scan-0.5.0.json` +|current +|What `pons scan --format json` emits. `0.4.0` plus three appended keys. + +|Evidence contract +|1.0.0 +|`evidence-1.0.0.json` +|current +|The `evidence_contract` object nested inside every scan envelope. + +|Verdict matrix +|1.0.0 +|`verdict-matrix-1.0.0.json` +|current +|Severity × evidence dispositions, the demoted-count fields and the recipe for +the corroboration key. Generated from `crates/pons-core/src/verdict.rs`. +|=== + +== The freeze rule + +A **frozen** version is a promise about bytes, not about intent: + +. its file is never edited and never re-issued — a change is a new version in a + new file, listed beside the frozen one; +. its BLAKE3 digest is recorded in `index.json` *and* pinned in the crate, so an + edit fails a test (`the_frozen_0_4_0_schema_is_still_the_file_this_build_was_tested_against`) + rather than quietly changing what a pinned consumer reads; +. it stays readable. `pons validate-report` accepts every frozen version, and the + version declared *inside* the document selects the schema it is validated + against — so an old report cannot be validated against a new shape, and a new + key cannot hide behind an old version number. + +`index.json` is itself checked: every listed file must exist, must carry its +version in its name, and must match its recorded digest +(`the_schema_index_attests_every_published_contract_file`). Exactly one scan +envelope is marked `current`, and it is the version the binary emits. + +== Adding a version + +. Write the new file; never touch an existing version's file. +. Add it to `index.json` with `"status": "current"`, and mark the previous one + `"frozen"`. +. If it is the scan envelope, bump `SCHEMA_VERSION` in + `crates/pons-core/src/report/json.rs`, update `FROZEN_SCHEMA_VERSIONS`, add the + frozen digest constant, and add an arm to `validator_for_version`. +. Update the shape test, Appendix G of `docs/PLAN.adoc`, and the version history + there and in `CHANGELOG.adoc`. +. Re-bless the verdict matrix only if the policy itself changed + (`PONS_BLESS=1 cargo test -p pons-core`), and say why in the changelog. + +A version constant nothing checks is a comment, not a contract: every step above +that can be checked automatically is. diff --git a/schemas/index.json b/schemas/index.json new file mode 100644 index 00000000..7156bfa9 --- /dev/null +++ b/schemas/index.json @@ -0,0 +1,43 @@ +{ + "schema_version": "1.0.0", + "producer": "pons", + "policy": "A frozen version is never edited and never re-issued: its file stays byte-identical and its blake3 digest is pinned here and in the crate's tests. A change is a new version with a new file, listed alongside the frozen ones, so a consumer at a pin can read the version it needs and verify it holds the bytes pons was tested against.", + "artifacts": [ + { + "id": "scan-envelope", + "version": "0.4.0", + "file": "scan-0.4.0.json", + "kind": "json-schema", + "status": "frozen", + "blake3": "bdef6c68691ffa0681b56d86adc31450e8c625d7ce5884090efe6fbc8fc46562", + "note": "Still accepted by `pons validate-report`; never emitted by this build." + }, + { + "id": "scan-envelope", + "version": "0.5.0", + "file": "scan-0.5.0.json", + "kind": "json-schema", + "status": "current", + "blake3": "1370aa9f168ae316cda9a13a364a03773c8aab2ad37465c043cf17d7295fe56a", + "note": "The envelope `pons scan --format json` emits." + }, + { + "id": "evidence-contract", + "version": "1.0.0", + "file": "evidence-1.0.0.json", + "kind": "json-schema", + "status": "current", + "blake3": "0dd397f09a31884614ad0e1727b4af595013aadb672701be1ff485810aa55fd8", + "note": "The `evidence_contract` object nested in every scan envelope." + }, + { + "id": "verdict-matrix", + "version": "1.0.0", + "file": "verdict-matrix-1.0.0.json", + "kind": "contract-data", + "status": "current", + "blake3": "b04cfb1b96808366251f0950d81a8d848277a6173de54fa56e4b81c65f11124d", + "note": "Severity x evidence dispositions, the demoted-count fields and the corroboration key recipe. Generated from crates/pons-core/src/verdict.rs." + } + ] +} diff --git a/schemas/scan-0.5.0.json b/schemas/scan-0.5.0.json new file mode 100644 index 00000000..3cffc07e --- /dev/null +++ b/schemas/scan-0.5.0.json @@ -0,0 +1,770 @@ +{ + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "tool", + "execution", + "scanned", + "counts", + "findings", + "evidence_contract" + ], + "properties": { + "schema_version": { + "const": "0.5.0" + }, + "tool": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "version", + "source_commit", + "source_dirty" + ], + "properties": { + "name": { + "const": "pons" + }, + "version": { + "type": "string", + "minLength": 1 + }, + "source_commit": { + "type": "string", + "minLength": 1 + }, + "source_dirty": { + "enum": [ + "true", + "false", + "unknown" + ] + } + } + }, + "execution": { + "type": "object", + "additionalProperties": false, + "required": [ + "status", + "exit_code" + ], + "properties": { + "status": { + "const": "completed" + }, + "exit_code": { + "anyOf": [ + { + "enum": [ + 0, + 1, + 2 + ] + }, + { + "type": "null" + } + ] + } + } + }, + "scanned": { + "type": "object", + "additionalProperties": false, + "required": [ + "root", + "coverage_status", + "inventory", + "coverage_failures", + "files", + "suppressed_findings", + "languages", + "transcoded", + "skipped" + ], + "properties": { + "root": { + "type": "string", + "minLength": 1 + }, + "coverage_status": { + "enum": [ + "complete-within-policy", + "partial", + "no-applicable-input" + ] + }, + "inventory": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "path", + "kind", + "language", + "outcome", + "reason", + "parser", + "artifact_digest", + "transcoded", + "analysis_notes", + "analysis_complete" + ], + "properties": { + "path": { + "type": "string", + "minLength": 1 + }, + "kind": { + "enum": [ + "file", + "directory", + "unknown" + ] + }, + "language": { + "type": "string", + "minLength": 1 + }, + "outcome": { + "enum": [ + "analysed", + "unsupported", + "excluded-by-policy", + "parse-error", + "timeout", + "unreadable" + ] + }, + "reason": { + "type": "string", + "minLength": 1 + }, + "parser": { + "anyOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "null" + } + ] + }, + "artifact_digest": { + "anyOf": [ + { + "type": "string", + "pattern": "^blake3:[0-9a-f]{64}$" + }, + { + "type": "null" + } + ] + }, + "transcoded": { + "type": "boolean" + }, + "analysis_notes": { + "type": "array", + "items": { + "type": "string" + } + }, + "analysis_complete": { + "type": "boolean" + } + } + } + }, + "coverage_failures": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "files": { + "type": "integer", + "minimum": 0 + }, + "suppressed_findings": { + "type": "integer", + "minimum": 0 + }, + "languages": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "transcoded": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "skipped": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "path", + "reason" + ], + "properties": { + "path": { + "type": "string", + "minLength": 1 + }, + "reason": { + "type": "string", + "minLength": 1 + } + } + } + } + } + }, + "counts": { + "type": "object", + "additionalProperties": false, + "required": [ + "by_evidence", + "total", + "actionable", + "demoted_speculative" + ], + "properties": { + "by_evidence": { + "type": "object", + "additionalProperties": false, + "required": [ + "HEURISTIC", + "DATAFLOW", + "PROTOCOL", + "SPECULATIVE" + ], + "properties": { + "HEURISTIC": { + "type": "integer", + "minimum": 0 + }, + "DATAFLOW": { + "type": "integer", + "minimum": 0 + }, + "PROTOCOL": { + "type": "integer", + "minimum": 0 + }, + "SPECULATIVE": { + "type": "integer", + "minimum": 0 + } + } + }, + "total": { + "type": "integer", + "minimum": 0 + }, + "actionable": { + "type": "integer", + "minimum": 0 + }, + "demoted_speculative": { + "type": "integer", + "minimum": 0 + } + } + }, + "findings": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "rule_id", + "tier", + "evidence", + "severity", + "location", + "message", + "evidence_note", + "counter_condition", + "semantic_fingerprint" + ], + "properties": { + "rule_id": { + "type": "string", + "minLength": 1 + }, + "tier": { + "enum": [ + "T0", + "T1", + "T2", + "T3" + ] + }, + "evidence": { + "enum": [ + "HEURISTIC", + "DATAFLOW", + "PROTOCOL", + "SPECULATIVE" + ] + }, + "severity": { + "enum": [ + "INFO", + "WARN", + "ERROR" + ] + }, + "location": { + "type": "object", + "additionalProperties": false, + "required": [ + "file", + "byte_start", + "byte_end", + "line_start", + "col_start", + "line_end", + "col_end" + ], + "properties": { + "file": { + "type": "string", + "minLength": 1 + }, + "byte_start": { + "type": "integer", + "minimum": 0 + }, + "byte_end": { + "type": "integer", + "minimum": 0 + }, + "line_start": { + "type": "integer", + "minimum": 1 + }, + "col_start": { + "type": "integer", + "minimum": 1 + }, + "line_end": { + "type": "integer", + "minimum": 1 + }, + "col_end": { + "type": "integer", + "minimum": 1 + } + } + }, + "message": { + "type": "string", + "minLength": 1 + }, + "evidence_note": { + "type": "string", + "minLength": 1 + }, + "counter_condition": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "semantic_fingerprint": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + } + }, + "allOf": [ + { + "if": { + "properties": { + "tier": { + "const": "T0" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "HEURISTIC" + } + } + } + }, + { + "if": { + "properties": { + "tier": { + "const": "T1" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "DATAFLOW" + } + } + } + }, + { + "if": { + "properties": { + "tier": { + "const": "T2" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "PROTOCOL" + } + } + } + }, + { + "if": { + "properties": { + "tier": { + "const": "T3" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "SPECULATIVE" + } + } + } + } + ] + } + }, + "evidence_contract": { + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "target", + "execution_outcome", + "parser_inventory", + "records" + ], + "properties": { + "schema_version": { + "const": "1.0.0" + }, + "target": { + "type": "object", + "additionalProperties": false, + "required": [ + "commit", + "dirty" + ], + "properties": { + "commit": { + "anyOf": [ + { + "type": "string", + "pattern": "^[0-9a-f]{40}$" + }, + { + "type": "null" + } + ] + }, + "dirty": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ] + } + } + }, + "execution_outcome": { + "enum": [ + "complete-within-policy", + "partial", + "no-applicable-input" + ] + }, + "parser_inventory": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "path", + "kind", + "language", + "outcome", + "reason", + "parser", + "artifact_digest", + "transcoded", + "analysis_notes", + "analysis_complete" + ], + "properties": { + "path": { + "type": "string", + "minLength": 1 + }, + "kind": { + "enum": [ + "file", + "directory", + "unknown" + ] + }, + "language": { + "type": "string", + "minLength": 1 + }, + "outcome": { + "enum": [ + "analysed", + "unsupported", + "excluded-by-policy", + "parse-error", + "timeout", + "unreadable" + ] + }, + "reason": { + "type": "string", + "minLength": 1 + }, + "parser": { + "anyOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "null" + } + ] + }, + "artifact_digest": { + "anyOf": [ + { + "type": "string", + "pattern": "^blake3:[0-9a-f]{64}$" + }, + { + "type": "null" + } + ] + }, + "transcoded": { + "type": "boolean" + }, + "analysis_notes": { + "type": "array", + "items": { + "type": "string" + } + }, + "analysis_complete": { + "type": "boolean" + } + } + } + }, + "records": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "fingerprint", + "producer", + "correlation_group", + "rule", + "tier", + "evidence", + "confidence", + "path", + "line", + "trace", + "assumptions", + "unknown_edges", + "counterconditions" + ], + "properties": { + "fingerprint": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, + "producer": { + "type": "string", + "minLength": 1 + }, + "correlation_group": { + "type": "string", + "minLength": 1 + }, + "rule": { + "type": "string", + "minLength": 1 + }, + "tier": { + "enum": [ + "T0", + "T1", + "T2", + "T3" + ] + }, + "evidence": { + "enum": [ + "HEURISTIC", + "DATAFLOW", + "PROTOCOL", + "SPECULATIVE" + ] + }, + "confidence": { + "type": "string", + "minLength": 1 + }, + "path": { + "type": "string", + "minLength": 1 + }, + "line": { + "type": "integer", + "minimum": 1 + }, + "trace": { + "type": "string", + "minLength": 1 + }, + "assumptions": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "unknown_edges": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "counterconditions": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "tier": { + "const": "T0" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "HEURISTIC" + } + } + } + }, + { + "if": { + "properties": { + "tier": { + "const": "T1" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "DATAFLOW" + } + } + } + }, + { + "if": { + "properties": { + "tier": { + "const": "T2" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "PROTOCOL" + } + } + } + }, + { + "if": { + "properties": { + "tier": { + "const": "T3" + } + } + }, + "then": { + "properties": { + "evidence": { + "const": "SPECULATIVE" + } + } + } + } + ] + } + } + }, + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Pons evidence exchange 1.0.0", + "$comment": "SPDX-License-Identifier: MPL-2.0; shape validation must be followed by pons validate-report semantic checks." + } + }, + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$comment": "SPDX-License-Identifier: MPL-2.0; envelope shape, not a security verdict.", + "title": "Pons scan envelope 0.5.0" +} diff --git a/schemas/verdict-matrix-1.0.0.json b/schemas/verdict-matrix-1.0.0.json new file mode 100644 index 00000000..814f70ed --- /dev/null +++ b/schemas/verdict-matrix-1.0.0.json @@ -0,0 +1,209 @@ +{ + "schema_version": "1.0.0", + "producer": "pons", + "generated_from": "crates/pons-core/src/verdict.rs", + "severity_scale": [ + { + "severity": "ERROR", + "weight": 3, + "fail_on_rank": 2, + "sarif_level_when_not_demoted": "error" + }, + { + "severity": "WARN", + "weight": 2, + "fail_on_rank": 1, + "sarif_level_when_not_demoted": "warning" + }, + { + "severity": "INFO", + "weight": 1, + "fail_on_rank": 0, + "sarif_level_when_not_demoted": "note" + } + ], + "evidence_scale": [ + { + "evidence": "HEURISTIC", + "tier": "T0", + "sarif_rank": 50, + "weight": 50, + "confidence": "heuristic", + "counts_toward_verdict": true + }, + { + "evidence": "DATAFLOW", + "tier": "T1", + "sarif_rank": 80, + "weight": 80, + "confidence": "model-dependent", + "counts_toward_verdict": true + }, + { + "evidence": "PROTOCOL", + "tier": "T2", + "sarif_rank": 70, + "weight": 70, + "confidence": "premise-dependent", + "counts_toward_verdict": true + }, + { + "evidence": "SPECULATIVE", + "tier": "T3", + "sarif_rank": 20, + "weight": 0, + "confidence": "speculative", + "counts_toward_verdict": false + } + ], + "corroboration": { + "scheme": "pons-corroboration-v1", + "hash": "blake3", + "encoding": "64 lowercase hexadecimal characters", + "input_template": "scheme, then one NUL, then the root-relative path, one NUL, the 1-based start line in decimal, one NUL, then the evidence class name; the whole string is hashed", + "class_source": "The producer's own evidence class, or the class the consuming grid has explicitly mapped a foreign producer onto. Never a raw foreign severity, and never an upgrade of one.", + "path_convention": "Root-relative, '/' separated, no leading './' — the same spelling as evidence_contract.records[].path, so a key survives a checkout move.", + "line_convention": "1-based line of the finding's start; a finding spanning lines is keyed on the line it starts on.", + "caution": "Equal keys assert the same path, line and evidence class and nothing more. They are not proof of semantic equivalence, and a demoted class corroborates no verdict." + }, + "grid": [ + { + "severity": "ERROR", + "evidence": "HEURISTIC", + "sarif_level": "error", + "aggregate_weight": 150, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "ERROR", + "evidence": "DATAFLOW", + "sarif_level": "error", + "aggregate_weight": 240, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "ERROR", + "evidence": "PROTOCOL", + "sarif_level": "error", + "aggregate_weight": 210, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "ERROR", + "evidence": "SPECULATIVE", + "sarif_level": "note", + "aggregate_weight": 0, + "counts_toward_verdict": false, + "gate_eligible": false, + "count_bucket": "demoted", + "demotion_suffix": " (heuristic — not a verdict)" + }, + { + "severity": "WARN", + "evidence": "HEURISTIC", + "sarif_level": "warning", + "aggregate_weight": 100, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "WARN", + "evidence": "DATAFLOW", + "sarif_level": "warning", + "aggregate_weight": 160, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "WARN", + "evidence": "PROTOCOL", + "sarif_level": "warning", + "aggregate_weight": 140, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "WARN", + "evidence": "SPECULATIVE", + "sarif_level": "note", + "aggregate_weight": 0, + "counts_toward_verdict": false, + "gate_eligible": false, + "count_bucket": "demoted", + "demotion_suffix": " (heuristic — not a verdict)" + }, + { + "severity": "INFO", + "evidence": "HEURISTIC", + "sarif_level": "note", + "aggregate_weight": 50, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "INFO", + "evidence": "DATAFLOW", + "sarif_level": "note", + "aggregate_weight": 80, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "INFO", + "evidence": "PROTOCOL", + "sarif_level": "note", + "aggregate_weight": 70, + "counts_toward_verdict": true, + "gate_eligible": true, + "count_bucket": "actionable", + "demotion_suffix": "" + }, + { + "severity": "INFO", + "evidence": "SPECULATIVE", + "sarif_level": "note", + "aggregate_weight": 0, + "counts_toward_verdict": false, + "gate_eligible": false, + "count_bucket": "demoted", + "demotion_suffix": " (heuristic — not a verdict)" + } + ], + "aggregation": { + "demoted_evidence": [ + "SPECULATIVE" + ], + "demoted_sarif_level": "note", + "demotion_suffix": " (heuristic — not a verdict)", + "demoted_count_field": "counts.demoted_speculative", + "actionable_count_field": "counts.actionable", + "total_count_field": "counts.total", + "total_includes_demoted": true, + "actionable_definition": "counts.total - counts.demoted_speculative", + "rules": [ + "A demoted finding contributes zero aggregate_weight and is counted only in counts.demoted_speculative; it is never part of counts.actionable.", + "Demotion is unconditional and overrides the finding's own severity: ERROR + SPECULATIVE is still not a verdict, and its SARIF level is 'note'.", + "An aggregating consumer reports counts.actionable as its finding total, or reports both numbers; a total that includes demoted findings must say so.", + "Corroboration collapses a site ACROSS producers, not findings within one report: two producers minting the same semantic_fingerprint are one corroborated site, while counts.* count findings, so two Pons rules at one site in one class are two findings sharing one key." + ] + } +} diff --git a/wiki/Evidence-Tiers.md b/wiki/Evidence-Tiers.md index 2c9950f2..4e8bccc5 100644 --- a/wiki/Evidence-Tiers.md +++ b/wiki/Evidence-Tiers.md @@ -43,10 +43,19 @@ trust label that travels with the finding into every report. - `HEURISTIC` — a syntactic match. Almost always right for its narrow pattern (e.g. `x / 0` with a **literal** zero), but it did not reason about reachability. - `SPECULATIVE` — a guess. pons thinks this **might** be worth a look and is - telling you so **without pretending to know**. In every format it is demoted: - in the terminal it is set apart and suffixed "`(heuristic — not a verdict)`"; - in SARIF it is forced to `level: note` with a low `rank`; in JSON it carries - `"evidence": "SPECULATIVE"`. You can hide all of them with `--no-speculative`. + telling you so **without pretending to know**. In every format it is demoted, + visually *and* numerically: in the terminal it is set apart and suffixed + "`(heuristic — not a verdict)`", and the summary line separates actionable + findings from demoted ones; in SARIF it is forced to `level: note` with a low + `rank` (visible, deliberately); in JSON it carries + `"evidence": "SPECULATIVE"`, and `counts.actionable` excludes it while + `counts.demoted_speculative` counts it — so a dashboard that sums findings + cannot turn a guess into weight. You can hide all of them with + `--no-speculative`. + + The same table consumers need is published as data in + `schemas/verdict-matrix-1.0.0.json`: all twelve severity × evidence cells with + their SARIF level, aggregate weight, count bucket and demotion suffix. ## The rule that makes this real: lossy labelling diff --git a/wiki/For-Platform-Maintainers.md b/wiki/For-Platform-Maintainers.md index c616148e..038d1c83 100644 --- a/wiki/For-Platform-Maintainers.md +++ b/wiki/For-Platform-Maintainers.md @@ -34,9 +34,11 @@ differently from proofs: - **`human`** (default) — terminal, `SPECULATIVE` visually demoted. - **`json`** — a stable envelope (Appendix G of the plan): `schema_version`, - `tool`, `scanned` counts, `counts.by_evidence`, and a `findings[]` array with - fixed field order. Compatible in shape with panic-attack's JSON habit, easing a - shared records sink. + `tool`, `scanned` counts, `counts.by_evidence`, `counts.actionable`, + `counts.demoted_speculative`, and a `findings[]` array with fixed field order. + Compatible in shape with panic-attack's JSON habit, easing a shared records + sink, and each finding carries a `semantic_fingerprint` so an aggregating grid + can mark one site once instead of counting overlapping producers twice. - **`sarif`** — SARIF 2.1.0, for GitHub code scanning, IDEs, and any SARIF viewer. ## SARIF: the demotion survives (this is the important bit) @@ -111,7 +113,18 @@ it does not invent them. surface, so a grammar change that would alter results breaks fixtures loudly rather than silently drifting your findings. - The JSON envelope and SARIF mapping carry `schema_version`; treat a bump as a - potential breaking change for your parser. + potential breaking change for your parser. The envelope is at **0.5.0**; **0.4.0 + is frozen** — its schema file is byte-pinned, `pons validate-report` still + accepts 0.4.0 reports, and the version inside a document selects the schema it + is checked against. `schemas/index.json` lists every published version, file + and BLAKE3 digest; read it at your pin. +- Do not re-derive the severity/evidence policy. It is published as data in + `schemas/verdict-matrix-1.0.0.json` (generated from the code, drift-tested), + including the demoted-count fields and the corroboration-key recipe. +- `SPECULATIVE` findings are demoted visually (message suffix, SARIF `note`) and + numerically (`aggregate_weight` 0, `counts.actionable` excludes them). If your + dashboard sums findings, sum `counts.actionable`, or show both numbers — never + add a guess into a verdict total. ## Estate integration (AmbientOps)