diff --git a/CHANGELOG.md b/CHANGELOG.md index 9bd09ee..b0d980a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,23 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **Loop closure (P5 of the substrate-v2 plan): doom-loop diagnosis, imagination, CUSUM + drift, checkpoint cadence.** `forge diagnose ""` hashes each failure into a + signature (line numbers, addresses, timestamps, and absolute paths normalized out) and + counts recurrences in a 50-entry ring; the 3rd identical hit is thrash — it mints a + content-addressed `diagnosis` claim into the team ledger and tells the agent to STOP + retrying and escalate ONE model tier with the diagnosis as the prompt's head (the same + loop becomes a one-per-team event, not one-per-session). `forge imagine ""` is the + static half of the consequence simulator (paper Eq. 4): entities → blast radius → + predicted breaks with confidence, plus the minimal dry-run test suite via weighted greedy + set cover (weight = file size as a duration proxy; classic ln-n approximation) and + `riskScore = Σ confidence` — the sandboxed worktree runner that executes the suite is the + P5 follow-up. `anchor.cusum()` adds the M4 one-sided CUSUM control chart (k = 0.35, + h = 1.0): sustained small drift alarms, a single exploratory spike drains back to zero. + `verify.checkpointCadence()` prices M6's "when to check?" as the optimal-stopping + threshold rule `n* = ⌈checkCost / (pErr·tokensPerStep·costPerToken)⌉`, clamped to + [1, 50] — every input measured or priced, no magic constants. + - **Context assembly + completeness gate (P4 of the substrate-v2 plan).** `forge context ""` makes what goes into the window a budgeted optimization and makes *sufficiency* a computed set. The required-knowledge set `R(edit)` — the target's @@ -20,6 +37,43 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). implements it?"), shown in `forge substrate` and — under `FORGE_ENFORCE=1` — blocking: acting on missing context is acting on a guess. Incomplete context stops being a feeling and starts being a set difference. +- **Generated-UI quality gate (P6 of the substrate-v2 plan).** Taste becomes measurable: + `src/uifingerprint.js` extracts a deterministic design fingerprint from CSS/JSX/Tailwind + classes — pure static parsing, no LLM, no screenshots — covering palette (HSL + 12-bin hue + histogram), spacing (base unit by residual-minimization approximate GCD, on-scale + fraction), font families, radius and shadow levels. Two distances gate generated UI: + `slopDistance` to a shipped, rationale-documented generic-template signature set + (default-Tailwind blue/indigo, stock Bootstrap, the AI-landing gradient) must stay HIGH, + and `conformance` to the project's own fingerprint — stored as a shared `fingerprint` + ledger claim via `mintProjectFingerprint` — must stay LOW; `uiGate` failures are + actionable per-feature edits, never a bare score. Scale-conformance checks + (spacing-on-base, radius/shadow level caps, palette bound) join `ASSERTABLE_CHECKS`. + `forge uicheck` gains `fingerprint [--mint]` and `design ` (exit 1 on + fail) alongside the unchanged contrast math. +- **Local dashboard (P7 of the substrate-v2 plan).** `forge dash [--port N]` serves a + read-only lens on the substrate's state: a `node:http` stdlib server (localhost-only, + zero runtime deps) with ONE self-contained HTML page — inline CSS/JS, no CDN, no + framework, no build step. Panels: Ledger (claims with val bars, kind filter, contested + claims — val ∈ [0.4, 0.6] with ≥1 contradiction — and per-author trust), Cost/Cache + (stage counters + measured saved-token estimates from `.forge/metrics.jsonl`), and + Impact (atlas blast-radius explorer via `/api/impact?target=X`). Every claim row shows + its `forge ledger blame ` command — no unexplained scores anywhere in the UI. Data + is separated from serving (`dashData()` vs `serve()` in `src/dash.js`) so the payload + is tested without sockets, and corrupt/missing stores degrade to empty sections instead + of taking down the lens. The ratify/retract POSTs are a follow-up; this phase never + writes. +- **Measured cost report (P8 of the substrate-v2 plan).** `forge cost --stages [--json]` + computes per-stage cost factors as pure arithmetic over `.forge/metrics.jsonl` + (`src/cost_report.js`): gate halt rate, tier-weighted cache hit rate (exact 1.0 / near + 0.85 / adapt 0.5), route saving priced against the always-premium baseline, and context + assembly — then composes `C = C₀ · Π(1 − fᵢ)` over ONLY the measured stages. A stage with + no events reports "no data", never a default; the composed figure is a lower bound whose + caveats name every unmeasured stage; the paper's 62 % routing figure is cited as context, + and ~90 % appears only as a labeled target. `substrateCheck` now meters the assumption + gate on the explicit path (one `gate` halt/pass line per decision; ambient hooks stay + write-free), `recordGate`/`recordRoute` give future stage wiring one obvious call each, + and `reports/cost-eval.md` scaffolds the paired-run harness report with a truthful + empty state. - **Proof-carrying reuse cache (P3 of the substrate-v2 plan).** `forge reuse` turns "reuse already-generated code" from prose into a deterministic system: verified code diff --git a/reports/cost-eval.md b/reports/cost-eval.md new file mode 100644 index 0000000..9bfe203 --- /dev/null +++ b/reports/cost-eval.md @@ -0,0 +1,66 @@ +# Cost evaluation — measured stage factors (P8) + +> Status: **no data yet.** This document is the artifact the P8 harness +> ([docs/plans/substrate-v2/05-cost-model.md](../docs/plans/substrate-v2/05-cost-model.md) §3) +> fills with measurements. Until a cell below holds a measured number, the only claimable +> figures are the paper's: 62 % routing saving on live tokens (paper §9). The plan's ~90 % +> composed figure is a **target**, not a result, and does not appear in this table. + +## Methodology + +The cost model is multiplicative — `C = C₀ · Π(1 − fᵢ)` over independent stages — so each +stage factor is measured separately and composed arithmetically, never asserted: + +1. **Instrumentation.** Every substrate stage appends one line to `.forge/metrics.jsonl` + (`{t, stage, outcome, tokensIn, tokensOut, tier, savedEstimate, ref}` — `src/metrics.js`). + `forge cost --stages` computes the per-stage factors from those lines (`src/cost_report.js`); + a stage with no events reports **no data**, never a default. +2. **Paired runs.** Baseline (always-premium, read-everything, no cache) vs. substrate over + the same replay corpus (N ≥ 100 real tasks, stratified repeat-heavy / mixed / cold), the + paper §9 methodology: identical tokens repriced, so every saving is arithmetic on measured + tokens. +3. **Correctness guard (spec §3).** A saving counts only if the external verifier passes the + output. A routed-down answer that fails is not a saving; a cache hit that gets reverted is + recorded as a *negative* entry. + +## Measured factors + +| stage | factor | events | measured? | +|---|---|---|---| +| gate (M2 halt rate) | — | 0 | no data yet — run with metrics enabled | +| cache (reuse, tier-weighted) | — | 0 | no data yet — run with metrics enabled | +| route (vs always-premium) | — | 0 | no data yet — run with metrics enabled | +| context (assembly ρ) | — | 0 | no data yet — run with metrics enabled | +| **composed (measured stages only)** | — | 0 | nothing to compose yet | + +Secondary counters (doom-loop halts avoided, M5 lean, avoided rework) are reported alongside +when populated — they are deliberately excluded from the multiplication (spec §1). + +## How to populate this table + +Metrics accrue as the substrate is actually used — each command below appends stage-tagged +lines to `.forge/metrics.jsonl`: + +```sh +# gate + cache: every explicit pre-action check meters both stages +forge substrate "" + +# cache: explicit reuse queries and mints +forge reuse query "" + +# then read the measured factors (and paste them here): +forge cost --stages # human table +forge cost --stages --json # machine-readable, for this report +``` + +Route and context events are emitted via `recordRoute` / a future context-assembly hook +(`src/cost_report.js`) as those stages gain live wiring. + +## Caveats that ship with any number placed here + +- Stage rates are **workload-dependent**: factors describe the recorded traffic of one repo, + not a general claim (spec §2 — repeat-heavy warm-ledger workloads differ from cold starts). +- The composed reduction is a **lower bound from measured stages only**; unmeasured stages + contribute nothing rather than a target. +- Until the paired-run harness with the correctness guard has run, per-stage factors from + live metrics are unguarded observational numbers, not eval results. diff --git a/src/anchor.js b/src/anchor.js index 94e784b..10bd8c6 100644 --- a/src/anchor.js +++ b/src/anchor.js @@ -165,6 +165,33 @@ export function goalDrift(root, goal, opts = {}) { }; } +/** + * M4 — one-sided CUSUM control chart over a drift-signal series (spec §5: + * docs/plans/substrate-v2/06-faculties-and-mechanisms.md). A raw threshold on a + * single checkpoint's drift Dₜ is noisy — one exploratory step legitimately wanders. + * CUSUM accumulates only the excess over the allowance k (Cₜ = max(0, Cₜ₋₁ + Dₜ − k)) + * and alarms at Cₜ > h, which detects SUSTAINED small drift with provably minimal + * detection delay for a given false-alarm rate (classical sequential analysis), + * while a single within-tolerance spike drains back to zero instead of alarming. + * Defaults k = 0.35, h = 1.0 per the spec (calibration lands in P8). Pure. + * @param {number[]} signals drift per checkpoint, Dₜ ∈ [0, 1] (non-numeric → 0) + * @param {{k?: number, h?: number}} [opts] + * @returns {{alarm: boolean, C: number[], firstAlarm: number}} firstAlarm = index of + * the first checkpoint whose statistic crossed h, or -1 if none did. + */ +export function cusum(signals, { k = 0.35, h = 1.0 } = {}) { + const C = []; + let c = 0; + let firstAlarm = -1; + for (let i = 0; i < signals.length; i++) { + const d = Number(signals[i]); + c = Math.max(0, c + (Number.isFinite(d) ? d : 0) - k); + C.push(c); + if (firstAlarm < 0 && c > h) firstAlarm = i; + } + return { alarm: firstAlarm >= 0, C, firstAlarm }; +} + export function renderAnchor(r) { const lines = ["Forge anchor — goal-drift check", ""]; if (!r.changed.length) diff --git a/src/cli.js b/src/cli.js index e90fb0d..34bf381 100755 --- a/src/cli.js +++ b/src/cli.js @@ -16,7 +16,7 @@ const COMMANDS = { harden: "wire security controls — gitleaks pre-commit + sandbox settings", remember: "add a durable fact to this repo's portable memory (forge brain)", brain: "show / rebuild the portable project memory index", - cost: "real per-day spend via ccusage + the cost ceiling", + cost: "real per-day spend via ccusage + measured stage factors (--stages)", spec: "spec-as-contract — init (OpenSpec) / lock / check drift", cortex: "self-correcting project memory — status / why ", ledger: "proof-carrying memory — stats / verify / show / blame / query / merge / import", @@ -28,8 +28,13 @@ const COMMANDS = { substrate: "one pre-action gate: assumptions, route, impact, scope, memory, verify", scope: "decompose files into independent clusters (+ coupled files you didn't name)", anchor: "goal-drift check — are your actual (git) changes still on the stated goal?", + diagnose: + "doom-loop check — record a failure; 3× the same signature mints a diagnosis + escalation", + imagine: "consequence simulation — predicted breaks + the minimal dry-run test suite for a task", lean: "scope-minimality (M5) — measure the diff's footprint vs what the task asked for", - uicheck: "deterministic UI check — WCAG contrast (assertable, no guessing)", + uicheck: + "deterministic UI checks — contrast · fingerprint · design ", + dash: "local dashboard over the ledger, metrics, and blast radius", brand: "print the active brand token map", }; @@ -585,6 +590,14 @@ async function run(argv) { return; } if (cmd === "cost") { + // `--stages` is the P8 measured report (per-stage factors from .forge/metrics.jsonl); + // the default path stays the ccusage per-day spend view, untouched. + if (argv.includes("--stages")) { + const { renderCostReport, report } = await import("./cost_report.js"); + const r = report(process.cwd()); + console.log(argv.includes("--json") ? JSON.stringify(r, null, 2) : renderCostReport(r)); + return; + } const { execFileSync } = await import("node:child_process"); const run = (bin, args) => execFileSync(bin, args, { encoding: "utf8", stdio: "pipe" }); console.log(`${BRAND.brand} cost — real per-day spend (ccusage)\n`); @@ -808,6 +821,59 @@ async function run(argv) { console.log(json ? JSON.stringify(r, null, 2) : renderAnchor(r)); return; // advisory — never fails the process } + if (cmd === "diagnose") { + const { diagnose, THRASH_K } = await import("./diagnose.js"); + const json = argv.includes("--json"); + const flagVal = (name) => { + const i = argv.indexOf(name); + return i >= 0 ? argv[i + 1] : undefined; + }; + const args = argv.filter( + (a, i) => !a.startsWith("--") && argv[i - 1] !== "--file" && argv[i - 1] !== "--symbol", + ); + const errorText = args.slice(1).join(" "); + if (!errorText) { + console.error('usage: forge diagnose "" [--file f] [--symbol s] [--json]'); + process.exitCode = 1; + return; + } + const r = diagnose(process.cwd(), { + errorText, + file: flagVal("--file"), + symbol: flagVal("--symbol"), + }); + if (json) return console.log(JSON.stringify(r, null, 2)); + console.log(`${BRAND.brand} diagnose — doom-loop check\n`); + console.log( + ` signature: ${r.signature.slice(0, 12)} · seen ${r.count}× in the recent failure window`, + ); + if (r.thrash) { + if (r.claimId) + console.log( + ` diagnosis claim: ${r.claimId.slice(0, 12)} (\`forge ledger show ${r.claimId.slice(0, 8)}\`)`, + ); + console.log(`\n ${r.escalate ?? r.reason}`); + } else { + console.log(` below the thrash threshold (${THRASH_K}) — recorded; keep going.`); + } + return; // advisory — halting the retry loop is the AGENT's move, not an exit code + } + if (cmd === "imagine") { + const { imagineTask, renderImagine } = await import("./imagine.js"); + const json = argv.includes("--json"); + const task = argv + .slice(1) + .filter((a) => a !== "--json") + .join(" "); + if (!task) { + console.error('usage: forge imagine "" [--json]'); + process.exitCode = 1; + return; + } + const r = imagineTask(process.cwd(), task); + console.log(json ? JSON.stringify(r, null, 2) : renderImagine(r)); + return; + } if (cmd === "lean") { const { leanRepo, renderLean } = await import("./lean.js"); const json = argv.includes("--json"); @@ -859,10 +925,88 @@ async function run(argv) { return; } if (cmd === "uicheck") { + const sub = argv[1]; + if (sub === "fingerprint" || sub === "design") { + const ui = await import("./uifingerprint.js"); + const json = argv.includes("--json"); + const files = argv.slice(2).filter((a) => !a.startsWith("--")); + if (!files.length) { + console.error( + `usage: ${BRAND.cli} uicheck ${sub} [--json]${sub === "fingerprint" ? " [--mint]" : ""}`, + ); + process.exitCode = 1; + return; + } + const fp = ui.fingerprintFiles(process.cwd(), files); + if (sub === "fingerprint") { + let minted = null; + if (argv.includes("--mint")) { + const { epochDay } = await import("./util.js"); + minted = ui.mintProjectFingerprint(process.cwd(), files, { t: epochDay() }); + } + if (json) { + console.log(JSON.stringify(minted ? { fingerprint: fp, minted } : fp, null, 2)); + } else { + console.log(`${BRAND.brand} uicheck fingerprint — the design feature vector\n`); + console.log( + ` palette: ${fp.paletteSize} color(s), hue bins [${fp.hueBuckets.join(" ")}]`, + ); + console.log( + ` spacing: ${fp.spacing.join(", ") || "(none)"} px — base ${fp.spacingBase ?? "(none)"}, ${Math.round(fp.spacingOnScale * 100)}% on-scale`, + ); + console.log(` type: ${fp.fontFamilies.join(", ") || "(none)"}`); + console.log( + ` shape: radii ${fp.radii.join(", ") || "(none)"} (${fp.radiusLevels} level(s)) · ${fp.shadowLevels} shadow level(s)`, + ); + if (minted) { + if (minted.ok) + console.log( + `\n minted fingerprint claim ${minted.id.slice(0, 12)}${minted.existed ? " (already in ledger)" : ""} — the gate's "home"`, + ); + else console.error(`\n mint failed: ${"reason" in minted ? minted.reason : ""}`); + } + } + if (minted && !minted.ok) process.exitCode = 1; + return; + } + // design — the two-sided gate: fail when too close to generic OR (when the + // project has minted its fingerprint) too far from the project's own system. + const projectFp = ui.loadProjectFingerprint(process.cwd()); + const gate = ui.uiGate(fp, { projectFp }); + const checks = ui.scaleChecks(fp); + const fail = !gate.pass || checks.some((c) => !c.pass); + if (json) { + console.log( + JSON.stringify({ ...gate, checks, hasProjectFingerprint: !!projectFp }, null, 2), + ); + } else { + const { tauSlop, tauConform } = ui.UI_GATE_DEFAULTS; + console.log(`${BRAND.brand} uicheck design — slop distance + project conformance\n`); + console.log( + ` slop distance: ${gate.slop} (need ≥ ${tauSlop} — farther from generic is better)`, + ); + console.log( + projectFp + ? ` conformance: ${gate.conform} (need ≤ ${tauConform} — closer to the project system is better)` + : ` conformance: (no project fingerprint claim — slop-only; mint one: \`${BRAND.cli} uicheck fingerprint --mint\`)`, + ); + for (const v of gate.violations) console.log(`\n ✗ ${v.detail}\n fix: ${v.hint}`); + console.log(""); + for (const c of checks) + console.log( + ` ${c.pass ? "✓" : "✗"} ${c.id}: ${c.detail}${c.pass || !c.hint ? "" : `\n fix: ${c.hint}`}`, + ); + console.log(`\n ${fail ? "✗ FAIL" : "✓ PASS"}`); + } + if (fail) process.exitCode = 1; + return; + } const { contrastRatio, wcagLevel, ASSERTABLE_CHECKS, ADVISORY_ONLY } = await import( "./uicheck.js" ); - const [fg, bg] = [argv[1], argv[2]]; + // `uicheck contrast ` is the named form; bare `uicheck ` stays + // supported (it predates the subcommands and hooks already call it). + const [fg, bg] = sub === "contrast" ? [argv[2], argv[3]] : [argv[1], argv[2]]; console.log(`${BRAND.brand} uicheck — deterministic UI review\n`); if (fg && bg) { try { @@ -880,6 +1024,27 @@ async function run(argv) { console.log(` ADVISE (subjective, human-only): ${ADVISORY_ONLY.slice(0, 4).join(", ")} …`); return; } + if (cmd === "dash") { + const { serve } = await import("./dash.js"); + const i = argv.indexOf("--port"); + const port = i >= 0 ? Number(argv[i + 1]) : 4242; + if (!Number.isInteger(port) || port < 0 || port > 65535) { + console.error("usage: forge dash [--port N]"); + process.exitCode = 1; + return; + } + const server = serve(process.cwd(), { port }); + server.on("listening", () => { + const addr = /** @type {import("node:net").AddressInfo} */ (server.address()); + console.log(`${BRAND.brand} dash — read-only lens on .forge/\n`); + console.log(` http://127.0.0.1:${addr.port} (localhost-only · Ctrl-C to stop)`); + }); + server.on("error", (err) => { + console.error(` ${err.message}`); + process.exitCode = 1; + }); + return; // the process stays alive serving — that's the command + } if (!(cmd in COMMANDS)) { console.error(`Unknown command: ${cmd}\nRun \`${BRAND.cli} --help\` to see commands.`); process.exitCode = 1; diff --git a/src/cost_report.js b/src/cost_report.js new file mode 100644 index 0000000..8bf021d --- /dev/null +++ b/src/cost_report.js @@ -0,0 +1,207 @@ +// forge cost report — the P8 measured-stage report (docs/plans/substrate-v2/05-cost-model.md). +// The cost model is multiplicative: C = C₀ · Π(1 − fᵢ) over independent stages. The discipline +// this module enforces is the paper's (§4, C6): a number is an assumption until measured. Every +// factor here is ARITHMETIC over .forge/metrics.jsonl lines that stages actually emitted; a +// stage with no events reports measured:false and value:null — it is never guessed, defaulted, +// or backfilled from a target. The ~90 % figure in the plan stays a TARGET everywhere in this +// module's output; the only measured external figure (62 % routing, paper §9) is cited as +// context, clearly labeled as not-local. +import { read, record } from "./metrics.js"; +import { MODELS } from "./model_tiers.js"; + +/** Saving weight per cache-hit tier — must stay consistent with reuse.js savedEstimate + * (exact = full regeneration avoided; near/adapt still spend adaptation tokens). */ +export const CACHE_TIER_SAVINGS = { hit_exact: 1.0, hit_near: 0.85, hit_adapt: 0.5 }; + +// The route factor's baseline is "always-premium": the tier an unrouted agent defaults to. +// That is the complex tier (opus), NOT the extreme tier — pricing the baseline at the +// rarely-justified top model would flatter the savings, and honesty is the point. +const ROUTE_BASELINE_KEY = "opus"; + +/** Resolve a metrics `tier` field to a pricing row — accepts the model key ("haiku") + * or the tier name ("simple"), since both appear in route results. */ +const modelForTier = (tier) => + MODELS[tier] ?? Object.values(MODELS).find((m) => m.tier === tier) ?? null; + +const tokenCost = (m, tokensIn, tokensOut) => tokensIn * m.inCost + tokensOut * m.outCost; + +const unmeasured = () => ({ measured: false, value: null, events: 0 }); + +/** + * Per-stage measured factors from .forge/metrics.jsonl. Each factor is + * {measured:boolean, value:number|null, events:number} — `events` counts only the lines + * usable for that stage's arithmetic, and a stage with none reports measured:false / + * value:null. NEVER invents a number. + * + * - gate: fraction of "gate" events with outcome "halt" — requests where spend was avoided + * entirely. (This measures h_gate; the plan's g ≈ per-halt spend share is taken as 1 — + * i.e. a halted task is assumed to have cost an average task, the simplest honest reading + * until the paired harness prices halts individually.) + * - cache: hit rate weighted by tier savings (exact 1.0 / near 0.85 / adapt 0.5); a miss + * contributes 0 to the numerator but counts in the denominator. + * - route: 1 − (actual token cost / always-premium cost) over "route" events that carry a + * resolvable tier AND real token counts — events without tokens can't be priced and are + * excluded rather than estimated. + * - context: fraction of would-have-been input tokens avoided, Σsaved / (Σsaved + Σactual), + * over "context" events carrying both savedEstimate and tokensIn. + * @param {string} root + */ +export function stageFactors(root) { + const gateEvents = read(root, { stage: "gate" }); + const gate = gateEvents.length + ? { + measured: true, + value: gateEvents.filter((e) => e.outcome === "halt").length / gateEvents.length, + events: gateEvents.length, + } + : unmeasured(); + + const cacheEvents = read(root, { stage: "cache" }); + const cache = cacheEvents.length + ? { + measured: true, + value: + cacheEvents.reduce((s, e) => s + (CACHE_TIER_SAVINGS[e.outcome] ?? 0), 0) / + cacheEvents.length, + events: cacheEvents.length, + } + : unmeasured(); + + const baseline = MODELS[ROUTE_BASELINE_KEY]; + const routeEvents = read(root, { stage: "route" }).filter( + (e) => + modelForTier(e.tier) && + Number.isFinite(e.tokensIn) && + Number.isFinite(e.tokensOut) && + e.tokensIn + e.tokensOut > 0, + ); + const route = routeEvents.length + ? { + measured: true, + value: + 1 - + routeEvents.reduce( + (s, e) => s + tokenCost(modelForTier(e.tier), e.tokensIn, e.tokensOut), + 0, + ) / + routeEvents.reduce((s, e) => s + tokenCost(baseline, e.tokensIn, e.tokensOut), 0), + events: routeEvents.length, + } + : unmeasured(); + + const ctxEvents = read(root, { stage: "context" }).filter( + (e) => Number.isFinite(e.savedEstimate) && Number.isFinite(e.tokensIn), + ); + const ctxSaved = ctxEvents.reduce((s, e) => s + e.savedEstimate, 0); + const ctxActual = ctxEvents.reduce((s, e) => s + e.tokensIn, 0); + const context = + ctxEvents.length && ctxSaved + ctxActual > 0 + ? { measured: true, value: ctxSaved / (ctxSaved + ctxActual), events: ctxEvents.length } + : unmeasured(); + + return { gate, cache, route, context }; +} + +/** + * The multiplicative composition C = C₀ · Π(1 − fᵢ) over ONLY the measured factors. + * Honest framing: because unmeasured stages contribute exactly nothing (factor 0, not a + * target), the result is a LOWER BOUND built from measured stages only — it can only grow + * as more stages start emitting metrics, and it is never the plan's ~90 % target restated. + * @param {ReturnType} factors + * @returns {{measuredReduction:number, stagesIncluded:string[], stagesMissing:string[]}} + */ +export function composedReduction(factors) { + const stagesIncluded = []; + const stagesMissing = []; + let remaining = 1; + for (const [name, f] of Object.entries(factors)) { + if (f.measured && typeof f.value === "number") { + stagesIncluded.push(name); + remaining *= 1 - f.value; + } else stagesMissing.push(name); + } + return { measuredReduction: 1 - remaining, stagesIncluded, stagesMissing }; +} + +/** + * Assemble the full report: factors, composition, raw totals, and a caveat per unmeasured + * stage plus the workload-dependence caveat — the caveats ship WITH the numbers so no + * consumer can quote the reduction without its conditions. + * @param {string} root + */ +export function report(root) { + const factors = stageFactors(root); + const composed = composedReduction(factors); + const all = read(root); + const totals = { + events: all.length, + savedEstimateTokens: all.reduce( + (s, e) => s + (Number.isFinite(e.savedEstimate) ? e.savedEstimate : 0), + 0, + ), + }; + const caveats = composed.stagesMissing.map( + (s) => + `stage "${s}" has no recorded events — unmeasured, contributes nothing to the composition`, + ); + caveats.push( + "stage rates are workload-dependent: these factors describe this repo's recorded traffic, not a general claim (05-cost-model.md §2)", + "the composed figure is a lower bound from measured stages only; savings are not correctness-guarded until the P8 paired harness runs (05-cost-model.md §3)", + ); + return { factors, composed, totals, caveats }; +} + +const pct = (v) => `${(v * 100).toFixed(1)}%`; + +/** + * Human rendering. Register matters as much as arithmetic: measured numbers print as + * measurements, the paper's 62 % routing figure prints as CONTEXT (a citation, not a local + * result), and the ~90 % figure appears only with the word "target" in front of it — this + * report never claims it as achieved. + * @param {ReturnType} r + */ +export function renderCostReport(r) { + const lines = ["Forge cost — measured stage factors (.forge/metrics.jsonl)", ""]; + lines.push(` ${"stage".padEnd(9)} ${"factor".padEnd(10)} events`); + for (const [name, f] of Object.entries(r.factors)) { + const shown = f.measured && typeof f.value === "number" ? pct(f.value) : "no data"; + lines.push(` ${name.padEnd(9)} ${shown.padEnd(10)} ${f.events}`); + } + lines.push(""); + lines.push( + r.composed.stagesIncluded.length + ? ` composed measured reduction: ${pct(r.composed.measuredReduction)} (from: ${r.composed.stagesIncluded.join(", ")}) — lower bound, measured stages only` + : " composed measured reduction: 0.0% — no stage has recorded events yet", + ); + lines.push( + ` totals: ${r.totals.events} metric event(s) · ~${r.totals.savedEstimateTokens} tokens saved (stage self-estimates)`, + ); + lines.push( + "", + " context (not a local measurement): the paper measured a 62% routing saving on live tokens (paper §9)", + " target (unmet until measured): the plan's composed target is ~90% (docs/plans/substrate-v2/05-cost-model.md)", + ); + lines.push("", " caveats:"); + for (const c of r.caveats) lines.push(` - ${c}`); + return lines.join("\n"); +} + +// --------------------------------------------------------------------------- +// Emit-side helpers — one obvious call per stage, so future wiring (context assembly, +// real route execution) adds a single line instead of re-deriving the schema. Thin +// wrappers over metrics.record, which is already best-effort (never throws). +// --------------------------------------------------------------------------- + +/** Record one assumption-gate decision: halted = spend avoided. + * @param {string} root + * @param {{halted?: boolean, ref?: string}} [opts] */ +export function recordGate(root, { halted, ref } = {}) { + return record(root, { stage: "gate", outcome: halted ? "halt" : "pass", ref }); +} + +/** Record one routed generation with its tier and real token counts. + * @param {string} root + * @param {{tier?: string, tokensIn?: number, tokensOut?: number, ref?: string}} [opts] */ +export function recordRoute(root, { tier, tokensIn, tokensOut, ref } = {}) { + return record(root, { stage: "route", tier, tokensIn, tokensOut, ref }); +} diff --git a/src/dash.html b/src/dash.html new file mode 100644 index 0000000..4d09d65 --- /dev/null +++ b/src/dash.html @@ -0,0 +1,339 @@ + + + + + +forge dash + + + +
+ + forge dash + + +
+ +
+
+

Ledger — proof-carrying memory .forge/ledger/

+
+
+ + + + +
+
+

Contested — val ∈ [0.40, 0.60] with ≥1 contradiction

+
+

Author trust — earned from oracle outcomes, floored at 0.50

+
+
+ +
+

Cost / Cache — measured, not asserted .forge/metrics.jsonl

+
+
+

Recent events

+
    +
    + +
    +

    Impact — blast radius .forge/atlas.json

    +
    + + +
    +
    pick a symbol to see what a change would touch
    +
    +
    + +
    read-only lens on .forge/ — every number traces to forge ledger blame <id>. ratify/retract land in a later phase.
    + + + + diff --git a/src/dash.js b/src/dash.js new file mode 100644 index 0000000..c69b399 --- /dev/null +++ b/src/dash.js @@ -0,0 +1,146 @@ +// forge dash — the local dashboard (docs/plans/substrate-v2/08-dashboard-ux.md, P7): +// a read-only lens on the .forge/ stores. DATA (dashData → one JSON payload) is +// separated from SERVING (serve → a node:http stdlib server) so the payload is +// testable without sockets. One self-contained HTML page (src/dash.html — inline +// CSS/JS, no CDN, no framework, no build step), localhost-only by default, zero +// runtime deps. Read-mostly by design: the two writes the spec names (ratify / +// retract POSTs) are a follow-up — this phase never writes anything. +import { readFileSync } from "node:fs"; +import { createServer } from "node:http"; +import { basename, dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { impact, load as loadAtlas } from "./atlas.js"; +import { authorTrust, claimText, val, validOutcome } from "./ledger.js"; +import { loadClaims, repoLedger, stats } from "./ledger_store.js"; +import { read as readMetrics, summarize } from "./metrics.js"; +import { epochDay } from "./util.js"; + +/** A claim is contested when its val sits in this band AND it carries ≥1 oracle + * contradiction — genuinely disputed, not merely fresh (a fresh claim is 0.5 with + * no evidence at all). Spec: 08-dashboard-ux.md §1, Ledger panel. */ +export const CONTESTED_BAND = [0.4, 0.6]; + +/** Payload caps — the dashboard is a lens, not an export format. */ +const CLAIM_CAP = 200; +const RECENT_CAP = 20; + +const emptyLedger = () => ({ + stats: { total: 0, tombstoned: 0, byKind: {}, val: { dormant: 0, uncertain: 0, trusted: 0 } }, + claims: [], + contested: [], + trust: {}, +}); + +/** The row shape every ledger table in the UI renders — id8 is the handle the + * provenance affordance (`forge ledger blame `) is built from. */ +const claimRow = (c, nowDay) => ({ + id8: c.id.slice(0, 8), + kind: c.kind, + val: Number(val(c, nowDay).toFixed(3)), + evidenceCount: (c.evidence ?? []).filter(validOutcome).length, + author: c.provenance?.author ?? "", + tombstoned: Boolean(c.tombstone), + text: claimText(c).slice(0, 140), +}); + +function ledgerSection(root, nowDay) { + const dir = repoLedger(root); + const all = loadClaims(dir); + const claims = all + .map((c) => claimRow(c, nowDay)) + .sort((a, b) => b.val - a.val || (a.id8 < b.id8 ? -1 : 1)) + .slice(0, CLAIM_CAP); + const contested = all + .filter((c) => { + if (c.tombstone) return false; + const v = val(c, nowDay); + if (v < CONTESTED_BAND[0] || v > CONTESTED_BAND[1]) return false; + return (c.evidence ?? []).some((e) => validOutcome(e) && e.result === "contradict"); + }) + .map((c) => claimRow(c, nowDay)); + return { stats: stats(dir, nowDay), claims, contested, trust: authorTrust(all) }; +} + +function metricsSection(root) { + const recent = readMetrics(root) + .slice(-RECENT_CAP) + .map((e) => ({ + t: e.t ?? 0, + stage: e.stage ?? "", + outcome: e.outcome ?? "", + savedEstimate: Number.isFinite(e.savedEstimate) ? e.savedEstimate : 0, + })); + return { stages: summarize(root), recent }; +} + +/** + * One JSON payload for the whole page — everything /api/data serves. Corrupt or + * missing stores degrade to empty sections; this NEVER throws (a broken .forge/ + * must not take down the lens that would let you see it's broken). + * @param {string} root + * @param {{nowDay?: number}} [opts] + */ +export function dashData(root, { nowDay = epochDay() } = {}) { + let ledger = emptyLedger(); + try { + ledger = ledgerSection(root, nowDay); + } catch {} + let metrics = { stages: {}, recent: [] }; + try { + metrics = metricsSection(root); + } catch {} + let atlas = { built: false, symbols: 0, files: 0 }; + try { + const a = loadAtlas(root); + if (a) atlas = { built: true, symbols: a.symbols?.length ?? 0, files: a.files ?? 0 }; + } catch {} + return { repo: basename(root), nowDay, ledger, metrics, atlas }; +} + +const HTML_PATH = join(dirname(fileURLToPath(import.meta.url)), "dash.html"); + +const sendJson = (res, code, body) => { + res.writeHead(code, { + "content-type": "application/json; charset=utf-8", + "cache-control": "no-store", + }); + res.end(JSON.stringify(body)); +}; + +/** + * The dashboard server: GET / → the page, GET /api/data → dashData, GET + * /api/impact?target=X → blast radius (when an atlas exists). Everything else 404. + * Localhost-only by default — pass a host explicitly to expose it, on your own head. + * @param {string} root + * @param {{port?: number, host?: string}} [opts] + * @returns {import("node:http").Server} + */ +export function serve(root, { port = 4242, host = "127.0.0.1" } = {}) { + const html = readFileSync(HTML_PATH, "utf8"); // read once at startup, self-contained + const server = createServer((req, res) => { + const url = new URL(req.url ?? "/", "http://localhost"); + if (req.method !== "GET") + return sendJson(res, 404, { error: "read-only in this phase — GET only" }); + if (url.pathname === "/") { + res.writeHead(200, { + "content-type": "text/html; charset=utf-8", + "cache-control": "no-store", + }); + return res.end(html); + } + if (url.pathname === "/api/data") return sendJson(res, 200, dashData(root)); + if (url.pathname === "/api/impact") { + const target = url.searchParams.get("target"); + if (!target) return sendJson(res, 400, { error: "usage: /api/impact?target=" }); + let atlas = null; + try { + atlas = loadAtlas(root); + } catch {} + if (!atlas) return sendJson(res, 404, { error: "no atlas — run `forge atlas build` first" }); + return sendJson(res, 200, impact(atlas, target)); + } + return sendJson(res, 404, { error: "not found" }); + }); + server.listen(port, host); + return server; +} diff --git a/src/diagnose.js b/src/diagnose.js new file mode 100644 index 0000000..b774a45 Binary files /dev/null and b/src/diagnose.js differ diff --git a/src/imagine.js b/src/imagine.js new file mode 100644 index 0000000..ac38b0d --- /dev/null +++ b/src/imagine.js @@ -0,0 +1,145 @@ +// forge imagine — the consequence simulator ĉ = g(a, C) (paper Eq. 4; +// docs/plans/substrate-v2/06-faculties-and-mechanisms.md §2). The atlas gives the +// exact-structure half: entities → blast radius → predicted breaks with confidence. +// This module adds impacted-test SELECTION — the minimal dry-run suite that makes +// pre-action simulation affordable at all (minutes → seconds vs "run everything"). +// The sandboxed worktree runner that EXECUTES the suite against a proposed diff is +// the P5 follow-up (spec §2.2) — selection had to exist first, and it is useful on +// its own as "run these, in this order, before you touch anything". +import { statSync } from "node:fs"; +import { join } from "node:path"; +import { build as buildAtlas, impact, load as loadAtlas } from "./atlas.js"; +import { referencedEntities } from "./preflight.js"; +import { isTestFile, predictFailingTests } from "./substrate.js"; + +/** + * Weighted greedy set cover over the covers(test, source) relation: candidates are + * substrate's predicted-failing tests plus any test files among the impacted set + * (those cover themselves — they are predicted to break, so the suite must include + * them). Weight = file size, a duration proxy until measured runtimes exist (P8). + * Each round picks the test minimizing weight / newly-covered — the classical + * greedy H(n) ≈ ln n approximation to weighted set cover (Chvátal 1979), and exact + * on the tiny instances a single edit produces. Impacted sources with no covering + * test simply can't constrain the suite; the gaps surface via selectTestsReport. + * @param {string} root + * @param {string[]} impactedFiles + * @returns {string[]} minimal ordered dry-run suite (best value first) + */ +export function selectTests(root, impactedFiles) { + return selectTestsReport(root, impactedFiles).tests; +} + +/** selectTests plus the sources no known test covers (the honest gap in the cover). */ +export function selectTestsReport(root, impactedFiles) { + const files = [...new Set(impactedFiles.map(String))]; + const covers = new Map(); // test → Set of covered elements (sources + itself) + const addCover = (test, el) => { + if (!covers.has(test)) covers.set(test, new Set()); + covers.get(test).add(el); + }; + for (const f of files) { + if (isTestFile(f)) addCover(f, f); + else for (const t of predictFailingTests(root, [f])) addCover(t, f); + } + const universe = new Set(); + for (const els of covers.values()) for (const el of els) universe.add(el); + const weights = new Map( + [...covers.keys()].map((t) => { + // Unreadable candidate → weight 1, never a throw: selection must not die on a + // just-deleted test file; running it later fails honestly instead. + let w = 1; + try { + w = Math.max(1, statSync(join(root, t)).size); + } catch {} + return [t, w]; + }), + ); + const tests = []; + const covered = new Set(); + while (covered.size < universe.size) { + let best = null; + let bestRatio = Infinity; + for (const [t, els] of covers) { + let gain = 0; + for (const el of els) if (!covered.has(el)) gain++; + if (!gain) continue; // also skips already-chosen tests (their gain is 0) + const ratio = weights.get(t) / gain; + // Deterministic tie-break by path so the suite order is stable across runs. + if (ratio < bestRatio || (ratio === bestRatio && best !== null && t < best)) { + best = t; + bestRatio = ratio; + } + } + if (best === null) break; + tests.push(best); + for (const el of covers.get(best)) covered.add(el); + } + const uncovered = files.filter((f) => !isTestFile(f) && !covered.has(f)).sort(); + return { tests, uncovered }; +} + +/** + * Imagine the consequences of a task before acting: entities → impact() blast + * radius → predicted breaks (per-file max confidence across targets), the minimal + * dry-run suite, and riskScore = Σ confidence — the number spec §2.3 thresholds to + * decide whether the (follow-up) sandboxed dry-run is worth paying for. + * @param {string} root + * @param {string} task + * @param {{atlas?: object, threshold?: number}} [opts] inject `atlas` to skip the build. + */ +export function imagineTask(root, task, { atlas, threshold = 0.1 } = {}) { + const graph = atlas || loadAtlas(root) || buildAtlas({ root }); + const entities = referencedEntities(task); + const targets = [...new Set([...entities.symbols, ...entities.files])].slice(0, 8); + const reports = targets.map((t) => impact(graph, t, { threshold })); + const byFile = new Map(); + for (const r of reports) { + for (const x of r.impacted) { + const f = x.node?.file; + if (f) byFile.set(f, Math.max(byFile.get(f) ?? 0, x.confidence)); + } + } + const predictedBreaks = [...byFile] + .map(([file, confidence]) => ({ file, confidence })) + .sort((a, b) => b.confidence - a.confidence || (a.file < b.file ? -1 : 1)); + const { tests, uncovered } = selectTestsReport( + root, + predictedBreaks.map((b) => b.file), + ); + return { + task: String(task), + targets, + found: reports.some((r) => r.found), + predictedBreaks, + tests, + uncovered, + riskScore: Number(predictedBreaks.reduce((s, b) => s + b.confidence, 0).toFixed(4)), + }; +} + +export function renderImagine(r) { + const lines = ["Forge imagine — consequence simulation (pre-action)", ""]; + lines.push(` targets: ${r.targets.length ? r.targets.join(", ") : "(none named)"}`); + if (!r.found) { + lines.push("", " nothing in the code graph matches this task — no consequences to predict."); + return lines.join("\n"); + } + lines.push(` risk score: ${r.riskScore} (Σ confidence over predicted breaks)`); + lines.push("", ` predicted breaks (${r.predictedBreaks.length}):`); + for (const b of r.predictedBreaks.slice(0, 12)) + lines.push(` ${b.confidence.toFixed(2)} ${b.file}`); + if (r.predictedBreaks.length > 12) lines.push(` … ${r.predictedBreaks.length - 12} more`); + if (r.tests.length) { + lines.push("", ` minimal dry-run suite (${r.tests.length}) — run these, in this order:`); + for (const t of r.tests) lines.push(` - ${t}`); + } else { + lines.push("", " no covering tests found for the predicted breaks."); + } + if (r.uncovered.length) + lines.push( + "", + ` ! no test covers: ${r.uncovered.slice(0, 6).join(", ")}${r.uncovered.length > 6 ? " …" : ""}`, + ); + lines.push("", " (sandboxed worktree dry-run of this suite lands as the P5 follow-up)"); + return lines.join("\n"); +} diff --git a/src/substrate.js b/src/substrate.js index 6d864e9..d510954 100644 --- a/src/substrate.js +++ b/src/substrate.js @@ -10,6 +10,7 @@ import { goalDrift } from "./anchor.js"; import { build as buildAtlas, impact as impactGraph, load as loadAtlas } from "./atlas.js"; import { assemble as assembleContext } from "./context.js"; import { matchingLessons } from "./cortex.js"; +import { recordGate } from "./cost_report.js"; import { leanRepo } from "./lean.js"; import { load as loadLessons } from "./lessons_store.js"; import { clarifyBlock, preflightRepo, referencedEntities } from "./preflight.js"; @@ -193,6 +194,14 @@ export function substrateCheck( }; const entities = referencedEntities(text); const preflight = preflightRepo(root, text, { askThreshold, allowBuild, ...llmOpts }); + // P8 gate metering: one metrics line per explicit gate decision (halt = spend avoided). + // Same write contract as reuseQuery vs reusePeek below — the ambient hook path + // (allowBuild:false) never appends. Best-effort: measurement must never block the gate. + if (allowBuild) { + try { + recordGate(root, { halted: preflight.assumption.shouldAsk }); + } catch {} + } // Reuse the gap preflight already computed — routeTask would otherwise recompute it (and, with // FORGE_LLM on, fire a second, redundant assumption model call whose result it discards). const route = routeTask(root, text, { ...llmOpts, ambiguity: preflight.gap }); diff --git a/src/uicheck.js b/src/uicheck.js index 246712b..e73de25 100644 --- a/src/uicheck.js +++ b/src/uicheck.js @@ -72,6 +72,18 @@ export const ASSERTABLE_CHECKS = [ id: "reduced-motion", how: "animations ≥200ms are wrapped in @media (prefers-reduced-motion)", }, + // Scale-conformance checks (P6): executable versions live in uifingerprint.js + // scaleChecks() under the SAME ids — a test pins the two lists together. + { + id: "spacing-scale", + how: "≥90% of spacing values are multiples of the declared base within ε=0.5px", + }, + { + id: "radius-levels", + how: "≤3 distinct border-radius levels (a design system uses few, deliberately)", + }, + { id: "shadow-levels", how: "≤3 distinct box-shadow levels (deliberate elevation steps)" }, + { id: "palette-size", how: "≤8 distinct colors after HSL normalization" }, ]; export const ADVISORY_ONLY = [ diff --git a/src/uifingerprint.js b/src/uifingerprint.js new file mode 100644 index 0000000..1e5914b --- /dev/null +++ b/src/uifingerprint.js @@ -0,0 +1,682 @@ +// forge uifingerprint — the generated-UI quality gate (P6, +// docs/plans/substrate-v2/07-ui-quality-gate.md). AI-generated UI converges on the +// max-likelihood template — the same statistical failure as M5 over-engineering, here +// favoring the *median* design. The taste layer (global/taste/*.md) is prose, and +// prose loses to gradients; this module makes taste MEASURABLE: a deterministic +// design fingerprint (pure static CSS/Tailwind-class parsing — no LLM, no +// screenshots, same discipline as uicheck's WCAG math) and two distances over it: +// slop(v) — too CLOSE to a shipped generic-template signature → fail +// conform(v) — too FAR from the project's own stored design system → fail +// Good output is far from generic and close to home; both are geometry once UI is a +// feature vector. The subjective residue (beauty) stays with the human reviewer — +// the gate's job is to stop the template from ever reaching them. +import { readFileSync } from "node:fs"; +import { isAbsolute, join } from "node:path"; +import { mintClaim } from "./ledger.js"; +import { loadClaims, putClaim, reindex, repoLedger } from "./ledger_store.js"; +import { gitAuthor } from "./util.js"; + +// --------------------------------------------------------------------------- +// Color parsing — everything normalizes to integer {h,s,l} so hue geometry (the +// "looks like the default framework palette" signal) is comparable across syntaxes. +// --------------------------------------------------------------------------- + +/** sRGB (0..255) → {h,s,l} with h in 0..359 degrees, s/l in 0..100 percent. */ +export function rgbToHsl(r, g, b) { + const rn = r / 255; + const gn = g / 255; + const bn = b / 255; + const max = Math.max(rn, gn, bn); + const min = Math.min(rn, gn, bn); + const d = max - min; + const l = (max + min) / 2; + let h = 0; + let s = 0; + if (d > 0) { + s = d / (1 - Math.abs(2 * l - 1)); + if (max === rn) h = ((gn - bn) / d + 6) % 6; + else if (max === gn) h = (bn - rn) / d + 2; + else h = (rn - gn) / d + 4; + h *= 60; + } + return { h: Math.round(h) % 360, s: Math.round(s * 100), l: Math.round(l * 100) }; +} + +// Representative [hue, saturation] per Tailwind color family (the 500 shade — the one +// every starter reaches for). Lightness comes from the shade number. Neutral families +// get s≤5 by fiat so an all-slate UI doesn't register as "chromatic blue": their tint +// is a background choice, not an accent. +const TW_FAMILY_HS = { + red: [0, 84], + orange: [25, 95], + amber: [38, 92], + yellow: [45, 93], + lime: [84, 81], + green: [142, 71], + emerald: [160, 84], + teal: [173, 80], + cyan: [189, 94], + sky: [199, 89], + blue: [217, 91], + indigo: [239, 84], + violet: [258, 90], + purple: [271, 91], + fuchsia: [292, 84], + pink: [330, 81], + rose: [350, 89], + slate: [215, 5], + gray: [220, 5], + zinc: [240, 4], + neutral: [0, 0], + stone: [25, 5], +}; + +const HEX_RE = /#([0-9a-f]{8}|[0-9a-f]{6}|[0-9a-f]{4}|[0-9a-f]{3})\b/gi; +const RGB_RE = /rgba?\(\s*(\d{1,3})[,\s]+(\d{1,3})[,\s]+(\d{1,3})/gi; +const HSL_RE = /hsla?\(\s*([\d.]+)(?:deg)?[,\s]+([\d.]+)%[,\s]+([\d.]+)%/gi; +const TW_COLOR_RE = new RegExp( + `\\b(?:bg|text|border|from|via|to|ring|outline|fill|stroke|accent|caret|decoration|divide|shadow)-(${Object.keys(TW_FAMILY_HS).join("|")})-(50|100|200|300|400|500|600|700|800|900|950)\\b`, + "g", +); +const TW_BW_RE = /\b(?:bg|text|border|from|via|to|ring|fill|stroke)-(white|black)\b/g; + +function parseColors(text) { + /** @type {{h:number,s:number,l:number}[]} */ + const out = []; + for (const [, hex] of text.matchAll(HEX_RE)) { + // 3/4-digit shorthand expands per CSS; a trailing alpha channel is ignored — the + // fingerprint cares about hue identity, not opacity. + const full = + hex.length <= 4 ? [...hex.slice(0, 3)].map((c) => c + c).join("") : hex.slice(0, 6); + out.push( + rgbToHsl( + parseInt(full.slice(0, 2), 16), + parseInt(full.slice(2, 4), 16), + parseInt(full.slice(4, 6), 16), + ), + ); + } + for (const [, r, g, b] of text.matchAll(RGB_RE)) out.push(rgbToHsl(+r, +g, +b)); + for (const [, h, s, l] of text.matchAll(HSL_RE)) + out.push({ h: Math.round(+h) % 360, s: Math.round(+s), l: Math.round(+l) }); + for (const [, family, shade] of text.matchAll(TW_COLOR_RE)) { + const [h, s] = TW_FAMILY_HS[family]; + // Lightness from the shade number: 50→95, 500→50, 950→5 — coarse but monotone, + // and hue (what the slop signatures key on) is exact. + out.push({ h, s, l: Math.min(96, Math.max(4, Math.round(100 - +shade / 10))) }); + } + for (const [, bw] of text.matchAll(TW_BW_RE)) + out.push({ h: 0, s: 0, l: bw === "white" ? 100 : 0 }); + return out; +} + +// --------------------------------------------------------------------------- +// Length parsing — spacing, radii. Everything lands in px (rem/em at the 16px root) +// so the base-unit inference sees one scale. +// --------------------------------------------------------------------------- + +const LEN_RE = /(-?\d*\.?\d+)(px|rem|em)\b/g; + +/** Absolute px lengths inside one CSS value string (zero and non-lengths skipped). */ +function parseLengths(value) { + const out = []; + for (const [, n, unit] of String(value).matchAll(LEN_RE)) { + const px = Math.abs(+n) * (unit === "px" ? 1 : 16); + if (px > 0) out.push(Math.round(px * 100) / 100); + } + return out; +} + +const cssValues = (text, propRe) => [...text.matchAll(propRe)].map((m) => m[1]); + +// The leading class keeps `scroll-padding`, `--m-4` etc. from matching. +const SPACING_PROP_RE = /(?:^|[;{\s"'])(?:margin|padding)(?:-[a-z-]+)?\s*:\s*([^;}]+)/gi; +const GAP_PROP_RE = /(?:^|[;{\s"'])(?:row-gap|column-gap|gap)\s*:\s*([^;}]+)/gi; +const RADIUS_PROP_RE = + /(?:^|[;{\s"'])border(?:-(?:top|bottom|start|end)-(?:left|right)?)?-radius\s*:\s*([^;}]+)/gi; +const SHADOW_PROP_RE = /(?:^|[;{\s"'])box-shadow\s*:\s*([^;}]+)/gi; +const FONT_PROP_RE = /(?:^|[;{\s"'])font-family\s*:\s*([^;}]+)/gi; + +// Tailwind spacing utilities: p-4/mx-2/gap-6/space-y-8 → n×4 px (p-px → 1px). The +// lookbehind stops `top-4` matching as `p-4`. +const TW_SPACE_RE = + /(? Math.abs(s - Math.round(s / u) * u); + +/** + * Infer the spacing base unit: residual(u) = Σ distance-to-nearest-multiple-of-u. + * Residuals are monotone in divisibility (every multiple of 8 is one of 4 and 2), so + * a bare argmin always degenerates to the smallest candidate — instead take the + * LARGEST base that explains the data within half a pixel per value, falling back to + * the true argmin when nothing fits cleanly. + * @param {number[]} values @param {number[]} [candidates] + * @returns {number|null} null when there are no values to infer from + */ +export function inferSpacingBase(values, candidates = [2, 4, 8]) { + if (!values.length) return null; + const residual = (u) => values.reduce((sum, s) => sum + offBase(s, u), 0); + let best = null; + for (const u of [...candidates].sort((a, b) => a - b)) + if (residual(u) <= 0.5 * values.length) best = u; + if (best !== null) return best; + return candidates.reduce((arg, u) => (residual(u) < residual(arg) ? u : arg), candidates[0]); +} + +/** Fraction of values sitting on multiples of `base` within ε (vacuously 1). */ +export function onScaleFraction(values, base, epsilon = 0.5) { + if (!values.length || !base) return 1; + return values.filter((s) => offBase(s, base) <= epsilon).length / values.length; +} + +// --------------------------------------------------------------------------- +// The fingerprint. +// --------------------------------------------------------------------------- + +/** + * @typedef {{h:number,s:number,l:number}} Hsl + * @typedef {{palette:Hsl[], paletteSize:number, hueBuckets:number[], spacing:number[], + * spacingBase:number|null, spacingOnScale:number, fontFamilies:string[], + * radii:number[], radiusLevels:number, shadowLevels:number}} Fingerprint + */ + +// Below this saturation a color is a neutral: it carries no hue identity, so it must +// not vote in the hue histogram (an ink-and-paper UI is not "red" at h=0). +const NEUTRAL_S = 10; + +const sortNum = (a, b) => a - b; +const uniqSorted = (arr) => [...new Set(arr)].sort(sortNum); + +/** + * Extract the design fingerprint from raw CSS / JSX / Tailwind-class text. Pure and + * deterministic — the same text always yields the same vector (it becomes a + * content-addressed ledger claim, so this is a protocol requirement, not a nicety). + * @param {string} text + * @returns {Fingerprint} + */ +export function fingerprintText(text) { + const t = String(text); + + const seen = new Set(); + /** @type {Hsl[]} */ + const palette = []; + for (const c of parseColors(t)) { + const key = `${c.h},${c.s},${c.l}`; + if (!seen.has(key)) { + seen.add(key); + palette.push(c); + } + } + palette.sort((a, b) => a.h - b.h || a.s - b.s || a.l - b.l); + const hueBuckets = new Array(12).fill(0); + for (const c of palette) if (c.s >= NEUTRAL_S) hueBuckets[Math.floor((c.h % 360) / 30)]++; + + const spacingRaw = [ + ...cssValues(t, SPACING_PROP_RE).flatMap(parseLengths), + ...cssValues(t, GAP_PROP_RE).flatMap(parseLengths), + ...[...t.matchAll(TW_SPACE_RE)].map(([, n]) => (n === "px" ? 1 : +n * 4)).filter(Boolean), + ]; + const spacing = uniqSorted(spacingRaw); + const spacingBase = inferSpacingBase(spacing); + const spacingOnScale = onScaleFraction(spacing, spacingBase); + + const fontFamilies = [ + ...new Set([ + // Only the FIRST family in a stack — the intended face; the rest are fallbacks. + ...cssValues(t, FONT_PROP_RE) + .map((v) => + String(v.split(",")[0]) + .trim() + .replace(/^["']|["']$/g, "") + .toLowerCase(), + ) + .filter((f) => f && !f.includes("(")), // var()/env() indirections carry no face + ...[...t.matchAll(TW_FONT_RE)].map(([, k]) => TW_FONT_STACK[k]), + ]), + ].sort(); + + const radii = uniqSorted([ + // 999+px pill radii normalize to one "full" level — 9999 vs 99999 is not a choice. + ...cssValues(t, RADIUS_PROP_RE) + .flatMap(parseLengths) + .map((r) => (r >= 999 ? 9999 : r)), + ...[...t.matchAll(TW_ROUNDED_RE)] + .map(([, size]) => (size === undefined ? 4 : TW_ROUNDED_PX[size])) + .filter((r) => r > 0), + ]); + + const shadows = new Set([ + ...cssValues(t, SHADOW_PROP_RE) + .map((v) => v.trim().replace(/\s+/g, " ")) + .filter((v) => v !== "none"), + ...[...t.matchAll(TW_SHADOW_RE)] + .map(([, size]) => `tw:${size ?? "base"}`) + .filter((s) => s !== "tw:none"), + ]); + + return { + palette, + paletteSize: palette.length, + hueBuckets, + spacing, + spacingBase, + spacingOnScale, + fontFamilies, + radii, + radiusLevels: radii.length, + shadowLevels: shadows.size, + }; +} + +/** + * Fingerprint a set of files as ONE vector (a design system is a property of the + * whole surface, not any single file). Unreadable files are skipped; the file list + * is sorted first so argument order can never change the vector. + * @param {string} root @param {string[]} files + * @returns {Fingerprint} + */ +export function fingerprintFiles(root, files) { + const texts = []; + for (const f of [...files].sort()) { + try { + texts.push(readFileSync(isAbsolute(f) ? f : join(root, f), "utf8")); + } catch {} + } + return fingerprintText(texts.join("\n")); +} + +// --------------------------------------------------------------------------- +// Slop distance — the shipped generic-template signature set. Each entry is the +// measurable footprint of a recognizable "AI default" look; being NEAR one of these +// is the failure. Curated, versioned, extensible (spec §2). +// --------------------------------------------------------------------------- + +/** + * @typedef {{id:string, why:string, hues:number[], spacingBase:number, + * spacingCount:number, fontCount:number, radii:number[], radiusLevels:number, + * shadowLevels:number}} GenericSignature + */ + +/** @type {GenericSignature[]} */ +export const GENERIC_SIGNATURES = [ + { + id: "tailwind-default", + why: "the untouched Tailwind starter: blue-500/indigo-500 accents, flat 8px-everything spacing (the p-2/p-4/p-8 trio), one sans stack, rounded-xl on every card, one soft shadow", + hues: [217, 239], // blue-500 #3b82f6 ≈ h217, indigo-500 #6366f1 ≈ h239 + spacingBase: 8, + spacingCount: 3, // 8/16/32 — a flat scale with no rhythm + fontCount: 1, + radii: [12], // rounded-xl + radiusLevels: 1, + shadowLevels: 1, + }, + { + id: "bootstrap-default", + why: "stock Bootstrap: the #0d6efd primary (h≈216), 1rem spacers halving to 8px steps, one system stack, uniform 6px --bs-border-radius, one shadow", + hues: [216], + spacingBase: 8, + spacingCount: 4, // the $spacer/2 ladder: 8/16/24/48 + fontCount: 1, + radii: [6], + radiusLevels: 1, + shadowLevels: 1, + }, + { + id: "ai-landing-gradient", + why: "the canonical AI landing page: violet→purple gradient hero (h 258–271), airy uniform 8px spacing, rounded-2xl cards, two layered soft shadows", + hues: [258, 271], // violet-500 #8b5cf6, purple-500 #a855f7 + spacingBase: 8, + spacingCount: 3, + fontCount: 1, + radii: [16], // rounded-2xl + radiusLevels: 1, + shadowLevels: 2, + }, +]; + +const round3 = (x) => Math.round(x * 1000) / 1000; +const hueDist = (a, b) => { + const d = Math.abs(a - b) % 360; + return Math.min(d, 360 - d); +}; +const chromaticHues = (fp) => fp.palette.filter((c) => c.s >= NEUTRAL_S).map((c) => c.h); + +/** Tolerate a raw ledger body (or partial vector) anywhere a fingerprint is read. */ +const asFp = (fp) => ({ + palette: fp?.palette ?? [], + paletteSize: fp?.paletteSize ?? 0, + hueBuckets: fp?.hueBuckets ?? new Array(12).fill(0), + spacing: fp?.spacing ?? [], + spacingBase: fp?.spacingBase ?? null, + spacingOnScale: fp?.spacingOnScale ?? 1, + fontFamilies: fp?.fontFamilies ?? [], + radii: fp?.radii ?? [], + radiusLevels: fp?.radiusLevels ?? 0, + shadowLevels: fp?.shadowLevels ?? 0, +}); + +// Per-feature distances (each in [0,1]) from a fingerprint to one generic signature. +// Only features the input actually exhibits are compared — an all-CSS-variables file +// with no radii must not be judged on radii it doesn't have. +function sigFeatures(fp, sig) { + const out = []; + if (fp.paletteSize > 0) { + const hues = chromaticHues(fp); + // A neutral-only palette is maximally far from a blue-band template (grayscale is + // a deliberate stance, e.g. brutalist), hence d=1 rather than "incomparable". + const d = hues.length + ? hues.reduce((s, h) => s + Math.min(...sig.hues.map((z) => hueDist(h, z))), 0) / + hues.length / + 180 + : 1; + out.push({ feature: "palette", d: round3(d) }); + } + if (fp.spacing.length) { + const baseD = fp.spacingBase === sig.spacingBase ? 0 : 1; + const divD = Math.min(1, Math.abs(fp.spacing.length - sig.spacingCount) / 8); + out.push({ feature: "spacing", d: round3((baseD + divD) / 2) }); + } + if (fp.fontFamilies.length) + out.push({ + feature: "type", + d: round3(Math.min(1, Math.abs(fp.fontFamilies.length - sig.fontCount) / 2)), + }); + if (fp.radii.length) { + const near = Math.min( + ...fp.radii.map((r) => Math.min(...sig.radii.map((z) => Math.min(1, Math.abs(r - z) / 16)))), + ); + const lev = Math.min(1, Math.abs(fp.radiusLevels - sig.radiusLevels) / 3); + out.push({ feature: "shape", d: round3((near + lev) / 2) }); + } + // Zero shadows on a UI that exhibits other features IS signal (flat ≠ template + // soft-shadow); but on an empty vector it would be judging nothing, so gate on + // having at least one other measurable feature. + if (out.length || fp.shadowLevels > 0) + out.push({ + feature: "elevation", + d: round3(Math.min(1, Math.abs(fp.shadowLevels - sig.shadowLevels) / 3)), + }); + return out; +} + +/** + * Nearest generic signature with its per-feature breakdown (what uiGate turns into + * actionable violations). Null when the input had nothing measurable. + * @param {Fingerprint} fingerprint + * @returns {{id:string, why:string, distance:number, features:{feature:string,d:number}[]}|null} + */ +export function nearestGeneric(fingerprint) { + const fp = asFp(fingerprint); + let best = null; + for (const sig of GENERIC_SIGNATURES) { + const features = sigFeatures(fp, sig); + if (!features.length) continue; + const distance = round3(features.reduce((s, f) => s + f.d, 0) / features.length); + if (!best || distance < best.distance) best = { id: sig.id, why: sig.why, distance, features }; + } + return best; +} + +/** Normalized distance (0..1) to the NEAREST generic-template signature. Low = slop. + * An empty/unmeasurable vector returns 1 — nothing measurable is not generic. */ +export function slopDistance(fingerprint) { + return nearestGeneric(fingerprint)?.distance ?? 1; +} + +// --------------------------------------------------------------------------- +// Conformance — distance from the project's own design system. +// --------------------------------------------------------------------------- + +const jaccardDist = (a, b) => { + const A = new Set(a); + const B = new Set(b); + const uni = new Set([...A, ...B]).size; + if (!uni) return 0; + return 1 - [...A].filter((x) => B.has(x)).length / uni; +}; + +// Per-feature distances between two fingerprints; comparable features only. +function conformFeatures(a, b) { + const out = []; + if (a.paletteSize || b.paletteSize) { + const ha = chromaticHues(a); + const hb = chromaticHues(b); + let d; + if (!ha.length && !hb.length) + d = 0; // two neutral-only systems agree + else if (!ha.length || !hb.length) + d = 1; // chromatic vs grayscale — different worlds + else { + // Symmetric mean nearest-hue distance (an averaged Hausdorff): every hue in + // each palette must have a home in the other. + const dir = (xs, ys) => + xs.reduce((s, h) => s + Math.min(...ys.map((z) => hueDist(h, z))), 0) / xs.length; + d = (dir(ha, hb) + dir(hb, ha)) / 2 / 180; + } + out.push({ feature: "palette", d: round3(d) }); + } + if (a.spacing.length && b.spacing.length) { + // Base disagreement in octaves (2 vs 8 is worse than 4 vs 8) blended with how + // much of the OUTPUT sits off the PROJECT's base — the actionable half. + const baseD = Math.min(1, Math.abs(Math.log2((a.spacingBase || 1) / (b.spacingBase || 1))) / 2); + const scaleD = 1 - onScaleFraction(a.spacing, b.spacingBase ?? undefined); + out.push({ feature: "spacing", d: round3((baseD + scaleD) / 2) }); + } + if (a.fontFamilies.length || b.fontFamilies.length) + out.push({ feature: "type", d: round3(jaccardDist(a.fontFamilies, b.fontFamilies)) }); + if (a.radii.length || b.radii.length) { + const lev = Math.min(1, Math.abs(a.radiusLevels - b.radiusLevels) / 3); + out.push({ feature: "shape", d: round3((jaccardDist(a.radii, b.radii) + lev) / 2) }); + } + out.push({ + feature: "elevation", + d: round3(Math.min(1, Math.abs(a.shadowLevels - b.shadowLevels) / 3)), + }); + return out; +} + +/** + * Normalized distance (0..1) between a fingerprint and the project's. High = the + * output ignored the system the codebase already has. + * @param {Fingerprint} fingerprint @param {Fingerprint} projectFp + */ +export function conformance(fingerprint, projectFp) { + const feats = conformFeatures(asFp(fingerprint), asFp(projectFp)); + return feats.length ? round3(feats.reduce((s, f) => s + f.d, 0) / feats.length) : 0; +} + +// --------------------------------------------------------------------------- +// The gate — actionable violations, never a bare score (spec §2: each failing +// feature maps to a concrete edit). +// --------------------------------------------------------------------------- + +/** Default thresholds; taste profiles override these in P8 once fixtures show separation. */ +export const UI_GATE_DEFAULTS = { tauSlop: 0.25, tauConform: 0.5 }; + +const SLOP_HINTS = { + palette: (_fp, near) => + `palette hues sit in the ${near.id} band — pick a brand hue (or the project's accent) outside it`, + spacing: (fp) => + `spacing is the uniform ${fp.spacingBase ?? 8}px template rhythm — use a deliberate scale (e.g. the project's 4-based scale with real jumps)`, + type: () => + "a single default font stack is the strongest template tell — use the project's faces or a deliberate pairing", + shape: () => + "one uniform large radius on everything reads as template — commit to few deliberate radii (0 counts)", + elevation: () => + "the one-soft-shadow-on-every-card look is generic — flatten, or define explicit elevation steps", +}; + +const CONFORM_HINTS = { + palette: (fp, proj) => + `output hues [${chromaticHues(proj).join(", ") || "neutral-only"}] are the project's — reuse those accents instead of [${chromaticHues(fp).join(", ") || "none"}]`, + spacing: (fp, proj) => + `the project spacing base is ${proj.spacingBase}px — put values on that scale (output inferred base ${fp.spacingBase}px)`, + type: (fp, proj) => + `output fonts [${fp.fontFamilies.join(", ")}] don't match the project's [${proj.fontFamilies.join(", ")}] — use the project stacks`, + shape: (fp, proj) => + `project radii are [${proj.radii.join(", ")}] — use those levels, not [${fp.radii.join(", ")}]`, + elevation: (fp, proj) => + `project uses ${proj.shadowLevels} shadow level(s), output uses ${fp.shadowLevels} — match the project's elevation system`, +}; + +/** + * The two-sided quality gate: PASS iff slop ≥ tauSlop AND (when a project + * fingerprint exists) conform ≤ tauConform. Violations name the driving feature and + * a concrete edit; because each per-feature distance is in [0,1], a failing mean + * always has at least one failing feature — a FAIL can never arrive hint-less. + * @param {Fingerprint} fingerprint + * @param {{projectFp?:Fingerprint|null, tauSlop?:number, tauConform?:number}} [opts] + * @returns {{pass:boolean, slop:number, conform:number|null, + * violations:{feature:string, detail:string, hint:string}[]}} + */ +export function uiGate(fingerprint, opts = {}) { + const { projectFp = null, tauSlop, tauConform } = { ...UI_GATE_DEFAULTS, ...opts }; + const fp = asFp(fingerprint); + const violations = []; + const near = nearestGeneric(fp); + const slop = near?.distance ?? 1; + if (near && slop < tauSlop) { + for (const f of near.features) + if (f.d < tauSlop) + violations.push({ + feature: f.feature, + detail: `${f.feature} is Δ${f.d} from the "${near.id}" template (need ≥ ${tauSlop} overall)`, + hint: SLOP_HINTS[f.feature](fp, near), + }); + } + let conform = null; + if (projectFp) { + const proj = asFp(projectFp); + const feats = conformFeatures(fp, proj); + conform = feats.length ? round3(feats.reduce((s, f) => s + f.d, 0) / feats.length) : 0; + if (conform > tauConform) { + for (const f of feats) + if (f.d > tauConform) + violations.push({ + feature: f.feature, + detail: `${f.feature} is Δ${f.d} from the project fingerprint (need ≤ ${tauConform} overall)`, + hint: CONFORM_HINTS[f.feature](fp, proj), + }); + } + } + return { pass: violations.length === 0, slop, conform, violations }; +} + +// --------------------------------------------------------------------------- +// Scale-conformance checks — the ASSERTABLE_CHECKS extension (spec §4). Same shape +// as uicheck's list: deterministic, per-fingerprint, pass|fail + fix hint. The ids +// mirror the entries added to uicheck.ASSERTABLE_CHECKS (drift-tested). +// --------------------------------------------------------------------------- + +/** A design system uses FEW levels, deliberately — these caps encode that. */ +export const SCALE_CHECK_DEFAULTS = { + epsilon: 0.5, // px tolerance for "on the base scale" (sub-pixel = rounding noise) + minOnScale: 0.9, // ≥90% of spacing values must sit on the base + maxRadiusLevels: 3, + maxShadowLevels: 3, + maxPalette: 8, // distinct normalized colors — beyond this it's pixel-soup, not a palette +}; + +/** + * Run the deterministic scale checks over a fingerprint. + * @param {Fingerprint} fingerprint + * @param {{base?:number|null, epsilon?:number, minOnScale?:number, maxRadiusLevels?:number, + * maxShadowLevels?:number, maxPalette?:number}} [opts] + * `base` = the DECLARED design-system base; defaults to the inferred one. + * @returns {{id:string, pass:boolean, detail:string, hint:string}[]} + */ +export function scaleChecks(fingerprint, opts = {}) { + const fp = asFp(fingerprint); + const o = { ...SCALE_CHECK_DEFAULTS, base: fp.spacingBase, ...opts }; + const off = o.base ? fp.spacing.filter((s) => offBase(s, o.base) > o.epsilon) : []; + const onFrac = fp.spacing.length ? 1 - off.length / fp.spacing.length : 1; + return [ + { + id: "spacing-scale", + pass: onFrac >= o.minOnScale, + detail: `${Math.round(onFrac * 100)}% of ${fp.spacing.length} spacing value(s) on the ${o.base ?? "(none)"}px base (ε ${o.epsilon}px)`, + hint: off.length ? `move ${off.join(", ")}px onto the ${o.base}px scale` : "", + }, + { + id: "radius-levels", + pass: fp.radiusLevels <= o.maxRadiusLevels, + detail: `${fp.radiusLevels} distinct radius level(s) (max ${o.maxRadiusLevels})`, + hint: + fp.radiusLevels > o.maxRadiusLevels + ? `collapse [${fp.radii.join(", ")}] to ≤${o.maxRadiusLevels} deliberate levels` + : "", + }, + { + id: "shadow-levels", + pass: fp.shadowLevels <= o.maxShadowLevels, + detail: `${fp.shadowLevels} distinct shadow level(s) (max ${o.maxShadowLevels})`, + hint: + fp.shadowLevels > o.maxShadowLevels + ? `define ≤${o.maxShadowLevels} elevation steps and reuse them` + : "", + }, + { + id: "palette-size", + pass: fp.paletteSize <= o.maxPalette, + detail: `${fp.paletteSize} distinct color(s) (max ${o.maxPalette})`, + hint: + fp.paletteSize > o.maxPalette + ? "consolidate to design tokens — a palette is a decision, not an accumulation" + : "", + }, + ]; +} + +// --------------------------------------------------------------------------- +// The project fingerprint claim — v_proj lives in the PCM ledger so it is shared +// with the team and updated by the same evidence rules as everything else. +// --------------------------------------------------------------------------- + +/** + * Extract the project fingerprint from `files` and store it as a `fingerprint` + * claim. Content-addressed: the same UI surface mints the same id on every machine, + * so teammates converge on one claim instead of duplicating. + * @param {string} root @param {string[]} files @param {{t?:number}} [opts] + * @returns {{ok:true, id:string, existed:boolean, fingerprint:Fingerprint}|{ok:false, reason:string}} + */ +export function mintProjectFingerprint(root, files, { t = 0 } = {}) { + const fingerprint = fingerprintFiles(root, files); + const minted = mintClaim({ + kind: "fingerprint", + body: fingerprint, + scope: { level: "repo" }, + provenance: { agent: "uicheck", author: gitAuthor() }, + t, + }); + if (!minted.ok) return { ok: false, reason: "reason" in minted ? minted.reason : "mint failed" }; + const dir = repoLedger(root); + const put = putClaim(dir, minted.claim); + if (!put.ok) return { ok: false, reason: put.reason ?? "putClaim failed" }; + reindex(dir, t); + return { ok: true, id: minted.claim.id, existed: Boolean(put.existed), fingerprint }; +} + +/** + * The stored project fingerprint (latest live `fingerprint` claim), or null on a + * greenfield repo — the gate then runs slop-only (spec §2). + * @param {string} root + * @returns {Fingerprint|null} + */ +export function loadProjectFingerprint(root) { + const live = loadClaims(repoLedger(root)).filter((c) => c.kind === "fingerprint" && !c.tombstone); + if (!live.length) return null; + live.sort((a, b) => (b.provenance?.t ?? 0) - (a.provenance?.t ?? 0) || (a.id < b.id ? -1 : 1)); + return live[0].body; +} diff --git a/src/verify.js b/src/verify.js index 90258ed..d49969a 100644 --- a/src/verify.js +++ b/src/verify.js @@ -57,6 +57,28 @@ function runTests(cwd) { } } +/** + * M6 — checkpoint cadence as an optimal-stopping threshold rule (spec §6: + * docs/plans/substrate-v2/06-faculties-and-mechanisms.md). Insert a checkpoint once + * the expected loss of continuing-while-wrong exceeds the check's price: + * pErr·tokensPerStep·costPerToken·n > checkCost, i.e. check every + * n* = ⌈checkCost / (pErr · tokensPerStep · costPerToken)⌉ meaningful steps. No + * magic constants: pErr is measured per tier from ledger outcome history, the costs + * are priced — riskier/cheaper tiers get smaller n* automatically. Clamped to + * [1, 50]: even a near-free check shouldn't fire more than every step, and even a + * near-riskless run must still checkpoint eventually. Pure. + * @param {{pErr: number, tokensPerStep: number, costPerToken?: number, checkCost: number}} f + * pErr = per-step error hazard; tokensPerStep = tokens put at risk per step; + * checkCost priced in the same token-cost unit. + * @returns {number} integer steps between checkpoints, in [1, 50] + */ +export function checkpointCadence({ pErr, tokensPerStep, costPerToken = 1, checkCost }) { + const n = Math.ceil(checkCost / (pErr * tokensPerStep * costPerToken)); + // Degenerate inputs (NaN from bad measurements) fail SAFE: check every step. + if (Number.isNaN(n)) return 1; + return Math.min(50, Math.max(1, n)); // zero risk → Infinity → the 50-step ceiling +} + export function verify({ targetRoot = process.cwd(), base = "HEAD" } = {}) { const diff = git(["diff", "--unified=0", base], targetRoot) || diff --git a/test/anchor.test.js b/test/anchor.test.js index a83b5c6..be50eea 100644 --- a/test/anchor.test.js +++ b/test/anchor.test.js @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { goalDrift, renderAnchor } from "../src/anchor.js"; +import { cusum, goalDrift, renderAnchor } from "../src/anchor.js"; // changed[] is injected so these are pure (no git needed). test("goalDrift flags a changed file unrelated to the goal", () => { @@ -76,3 +76,43 @@ test("goalDrift (llm on): a throwing runner falls back to the deterministic spli assert.ok(r.offGoal.includes("src/reports.js")); assert.equal(r.drift, true); }); + +// --------------------------------------------------------------------------- +// M4 — one-sided CUSUM drift control (pure). +// --------------------------------------------------------------------------- + +test("cusum alarms on sustained small drift (the decaying-anchor failure)", () => { + // Each checkpoint drifts 0.6, only 0.25 over the allowance — no single step is + // alarming, but the excess accumulates: C = 0.25, 0.5, 0.75, 1.0, 1.25 → alarm. + const r = cusum([0.6, 0.6, 0.6, 0.6, 0.6, 0.6]); + assert.equal(r.alarm, true); + assert.equal(r.firstAlarm, 4, "alarms at the first checkpoint where C exceeds h"); + assert.equal(r.C.length, 6); + assert.ok(Math.abs(r.C[0] - 0.25) < 1e-9); +}); + +test("cusum does not alarm on a single spike within tolerance", () => { + const r = cusum([0.1, 1.2, 0.1, 0.1, 0.1, 0.1]); + assert.equal(r.alarm, false); + assert.equal(r.firstAlarm, -1); + assert.equal(r.C.at(-1), 0, "the statistic drains back to zero after the spike"); +}); + +test("cusum is quiet on on-goal signals and on an empty series", () => { + assert.equal(cusum([0.1, 0.2, 0.3, 0.1]).alarm, false); + assert.deepEqual(cusum([]), { alarm: false, C: [], firstAlarm: -1 }); +}); + +test("cusum honors custom k and h", () => { + // With a zero allowance every step accumulates fully; h=0.5 trips on step 2. + const r = cusum([0.3, 0.3, 0.3], { k: 0, h: 0.5 }); + assert.equal(r.firstAlarm, 1); + // A generous h absorbs the same series entirely. + assert.equal(cusum([0.3, 0.3, 0.3], { k: 0, h: 2 }).alarm, false); +}); + +test("cusum treats non-numeric signals as zero drift (never NaN-poisons the chart)", () => { + const r = cusum([0.6, Number.NaN, 0.6]); + assert.ok(r.C.every((c) => Number.isFinite(c))); + assert.equal(r.alarm, false); +}); diff --git a/test/cost_report.test.js b/test/cost_report.test.js new file mode 100644 index 0000000..3c77f4e --- /dev/null +++ b/test/cost_report.test.js @@ -0,0 +1,223 @@ +import assert from "node:assert/strict"; +import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { + composedReduction, + recordGate, + recordRoute, + renderCostReport, + report, + stageFactors, +} from "../src/cost_report.js"; +import { read as readMetrics } from "../src/metrics.js"; +import { substrateCheck } from "../src/substrate.js"; + +const tmp = () => mkdtempSync(join(tmpdir(), "forge-cost-")); + +/** Write a fixture .forge/metrics.jsonl from entry objects. */ +function seed(root, entries) { + mkdirSync(join(root, ".forge"), { recursive: true }); + writeFileSync( + join(root, ".forge", "metrics.jsonl"), + `${entries.map((e) => JSON.stringify({ t: 1, ...e })).join("\n")}\n`, + ); +} + +// --- stageFactors: the arithmetic is exact over fixture lines ------------------------- + +test("stageFactors: gate factor is the exact halt fraction", () => { + const root = tmp(); + seed(root, [ + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + ]); + const f = stageFactors(root); + assert.deepEqual(f.gate, { measured: true, value: 2 / 5, events: 5 }); +}); + +test("stageFactors: cache factor is the tier-weighted hit rate (exact 1.0 / near 0.85 / adapt 0.5)", () => { + const root = tmp(); + seed(root, [ + { stage: "cache", outcome: "hit_exact" }, + { stage: "cache", outcome: "hit_near" }, + { stage: "cache", outcome: "hit_adapt" }, + { stage: "cache", outcome: "miss" }, + ]); + const f = stageFactors(root); + assert.equal(f.cache.measured, true); + assert.equal(f.cache.events, 4); + assert.ok(Math.abs(f.cache.value - (1.0 + 0.85 + 0.5) / 4) < 1e-12); +}); + +test("stageFactors: route factor prices tokens vs the always-premium baseline; unpriceable events excluded", () => { + const root = tmp(); + seed(root, [ + // haiku (in 1, out 5) vs opus baseline (in 5, out 25): 1000·1+1000·5=6000 vs 30000. + { stage: "route", tier: "haiku", tokensIn: 1000, tokensOut: 1000 }, + // Tier names resolve too ("simple" is haiku's tier) — but no tokens ⇒ excluded, not estimated. + { stage: "route", tier: "simple" }, + { stage: "route", tier: "nonsense", tokensIn: 50, tokensOut: 50 }, + ]); + const f = stageFactors(root); + assert.deepEqual(f.route, { measured: true, value: 1 - 6000 / 30000, events: 1 }); +}); + +test("stageFactors: context factor is saved / (saved + actual) input tokens", () => { + const root = tmp(); + seed(root, [ + { stage: "context", savedEstimate: 300, tokensIn: 700 }, + { stage: "context", savedEstimate: 0, tokensIn: 1000 }, + ]); + const f = stageFactors(root); + assert.equal(f.context.measured, true); + assert.equal(f.context.events, 2); + assert.ok(Math.abs(f.context.value - 300 / 2000) < 1e-12); +}); + +test("stageFactors: an empty store never invents a number — every stage is measured:false / null", () => { + const root = tmp(); + const f = stageFactors(root); + for (const name of ["gate", "cache", "route", "context"]) { + assert.deepEqual(f[name], { measured: false, value: null, events: 0 }, name); + } +}); + +// --- composedReduction: multiplicative over MEASURED stages only ---------------------- + +test("composedReduction: C = Π(1 − f) over measured factors only; missing stages named", () => { + const c = composedReduction({ + gate: { measured: true, value: 0.1, events: 10 }, + cache: { measured: true, value: 0.5, events: 4 }, + route: { measured: false, value: null, events: 0 }, + context: { measured: false, value: null, events: 0 }, + }); + assert.ok(Math.abs(c.measuredReduction - (1 - 0.9 * 0.5)) < 1e-12); + assert.deepEqual(c.stagesIncluded, ["gate", "cache"]); + assert.deepEqual(c.stagesMissing, ["route", "context"]); +}); + +test("composedReduction: nothing measured composes to exactly 0 — never a target restated", () => { + const c = composedReduction(stageFactors(tmp())); + assert.equal(c.measuredReduction, 0); + assert.deepEqual(c.stagesIncluded, []); + assert.deepEqual(c.stagesMissing, ["gate", "cache", "route", "context"]); +}); + +// --- report: totals + a caveat for every unmeasured stage ------------------------------ + +test("report: empty store carries a caveat naming EVERY unmeasured stage plus workload dependence", () => { + const r = report(tmp()); + assert.equal(r.totals.events, 0); + assert.equal(r.totals.savedEstimateTokens, 0); + for (const s of ["gate", "cache", "route", "context"]) { + assert.ok( + r.caveats.some((c) => c.includes(`"${s}"`)), + `caveat names unmeasured stage ${s}`, + ); + } + assert.ok(r.caveats.some((c) => c.includes("workload-dependent"))); +}); + +test("report: totals sum every event and savedEstimate across stages", () => { + const root = tmp(); + seed(root, [ + { stage: "cache", outcome: "hit_exact", savedEstimate: 120 }, + { stage: "cache", outcome: "miss", savedEstimate: 0 }, + { stage: "gate", outcome: "halt" }, + ]); + const r = report(root); + assert.equal(r.totals.events, 3); + assert.equal(r.totals.savedEstimateTokens, 120); + assert.equal(r.composed.stagesIncluded.length, 2); +}); + +// --- renderCostReport: the honesty register ------------------------------------------- + +const assert90OnlyAsTarget = (text) => { + for (let i = text.indexOf("90"); i !== -1; i = text.indexOf("90", i + 1)) { + assert.ok( + /target/i.test(text.slice(Math.max(0, i - 60), i)), + `"90" at index ${i} not preceded by "target": …${text.slice(Math.max(0, i - 60), i + 2)}`, + ); + } +}; + +test('renderCostReport: "90" appears ONLY behind the word "target" — never as achieved', () => { + // Empty store, a mixed store, and a pathological store whose factors would round to 90 %. + assert90OnlyAsTarget(renderCostReport(report(tmp()))); + const mixed = tmp(); + seed(mixed, [ + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "pass" }, + { stage: "cache", outcome: "hit_near", savedEstimate: 50 }, + ]); + assert90OnlyAsTarget(renderCostReport(report(mixed))); +}); + +test("renderCostReport: unmeasured stages print as no-data; the 62% figure is labeled as paper context", () => { + const out = renderCostReport(report(tmp())); + assert.ok(out.includes("gate")); + assert.ok((out.match(/no data/g) || []).length === 4, "all four stages show no data"); + assert.ok(/context \(not a local measurement\).*62%.*paper §9/.test(out)); + assert.ok(out.includes("caveats:")); +}); + +test("renderCostReport: measured factors print as percentages with event counts", () => { + const root = tmp(); + seed(root, [ + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + ]); + const out = renderCostReport(report(root)); + assert.ok(out.includes("25.0%"), "gate halt fraction rendered"); + assert.ok(out.includes("composed measured reduction: 25.0%")); +}); + +// --- emit-side helpers + substrate wiring ---------------------------------------------- + +test("recordGate / recordRoute: thin wrappers land stage-tagged lines in metrics.jsonl", () => { + const root = tmp(); + recordGate(root, { halted: true, ref: "task-1" }); + recordGate(root, { halted: false }); + recordRoute(root, { tier: "haiku", tokensIn: 10, tokensOut: 20, ref: "task-2" }); + const gate = readMetrics(root, { stage: "gate" }); + assert.deepEqual( + gate.map((e) => e.outcome), + ["halt", "pass"], + ); + assert.equal(gate[0].ref, "task-1"); + const route = readMetrics(root, { stage: "route" }); + assert.equal(route.length, 1); + assert.equal(route[0].tokensOut, 20); +}); + +test("substrateCheck meters the gate on the explicit path only — ambient hooks stay write-free", () => { + const specified = + "Add a computeVat(rate) function to math.js next to computeTax; must return x*rate; add a unit test"; + const explicit = tmp(); + writeFileSync(join(explicit, "math.js"), "export function computeTax(x){ return x * 0.2 }\n"); + substrateCheck(explicit, specified, { allowBuild: true }); + const gate = readMetrics(explicit, { stage: "gate" }); + assert.equal(gate.length, 1); + assert.equal(gate[0].outcome, "pass"); + + const ambient = tmp(); + writeFileSync(join(ambient, "math.js"), "export function computeTax(x){ return x * 0.2 }\n"); + substrateCheck(ambient, specified, { allowBuild: false }); + assert.ok(!existsSync(join(ambient, ".forge", "metrics.jsonl")), "ambient path never writes"); +}); + +test("substrateCheck meters a halt when the gate asks first", () => { + const root = tmp(); + substrateCheck(root, "Fix it.", { allowBuild: true }); + const gate = readMetrics(root, { stage: "gate" }); + assert.equal(gate.length, 1); + assert.equal(gate[0].outcome, "halt"); +}); diff --git a/test/dash.test.js b/test/dash.test.js new file mode 100644 index 0000000..8b4e8b8 --- /dev/null +++ b/test/dash.test.js @@ -0,0 +1,166 @@ +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { dashData, serve } from "../src/dash.js"; +import { mintClaim, outcomeRecord } from "../src/ledger.js"; +import { appendEvidence, putClaim, repoLedger, tombstone } from "../src/ledger_store.js"; +import { record } from "../src/metrics.js"; + +const tmp = () => mkdtempSync(join(tmpdir(), "forge-dash-")); +const NOW = 100; + +const mint = (dir, name, text, author = "alice") => { + const c = mintClaim({ kind: "fact", body: { name, text }, provenance: { author }, t: NOW }).claim; + putClaim(dir, c); + return c; +}; +const ev = (dir, id, result, ref) => + appendEvidence(dir, id, outcomeRecord({ oracle: "test.run", result, ref, t: NOW }).outcome); + +/** A fixture repo: 3 claims (one contested, one tombstoned) + 2 metrics lines. */ +function fixture() { + const root = tmp(); + const dir = repoLedger(root); + const trusted = mint(dir, "style", "tabs are actually spaces here"); + ev(dir, trusted.id, "confirm", "run-1"); + const contested = mint(dir, "port", "the dev server listens on 4242"); + ev(dir, contested.id, "confirm", "run-2"); // 1 confirm + 1 contradict → val = 0.5 + ev(dir, contested.id, "contradict", "run-3"); + const retracted = mint(dir, "old", "we deploy from the ops box"); + tombstone(dir, retracted.id, { author: "alice", reason: "obsolete", t: NOW }); + record(root, { stage: "cache", outcome: "exact", savedEstimate: 1200 }); + record(root, { stage: "gate", outcome: "pass" }); + return { root, trusted, contested, retracted }; +} + +test("dashData: one payload with ledger, metrics, and atlas sections in shape", () => { + const { root, contested, retracted } = fixture(); + const d = dashData(root, { nowDay: NOW }); + + assert.equal(d.repo, root.split("/").pop()); + assert.equal(d.nowDay, NOW); + assert.equal(d.ledger.stats.total, 3); + assert.equal(d.ledger.stats.tombstoned, 1); + assert.equal(d.ledger.stats.byKind.fact, 3); + + assert.equal(d.ledger.claims.length, 3); + for (const c of d.ledger.claims) { + assert.equal(c.id8.length, 8, "id8 is the blame handle"); + assert.equal(c.kind, "fact"); + assert.equal(typeof c.val, "number"); + assert.equal(typeof c.evidenceCount, "number"); + assert.equal(c.author, "alice"); + assert.equal(typeof c.tombstoned, "boolean"); + assert.equal(typeof c.text, "string"); + assert.ok(c.text.length > 0, "text comes from claimText, never empty for a fact"); + } + const tomb = d.ledger.claims.find((c) => c.id8 === retracted.id.slice(0, 8)); + assert.equal(tomb.tombstoned, true); + + // Contested = val ∈ [0.4, 0.6] AND ≥1 contradiction — exactly the mixed-evidence claim. + assert.deepEqual( + d.ledger.contested.map((c) => c.id8), + [contested.id.slice(0, 8)], + ); + assert.equal(d.ledger.contested[0].val, 0.5); + + assert.equal(typeof d.ledger.trust.alice, "number", "authorTrust map keyed by author"); + + assert.equal(d.metrics.stages.cache.savedEstimate, 1200); + assert.equal(d.metrics.stages.gate.byOutcome.pass, 1); + assert.equal(d.metrics.recent.length, 2); + assert.equal(d.metrics.recent[0].stage, "cache"); + + assert.deepEqual(d.atlas, { built: false, symbols: 0, files: 0 }); +}); + +test("dashData: atlas section reports a built atlas", () => { + const { root } = fixture(); + mkdirSync(join(root, ".forge"), { recursive: true }); + writeFileSync( + join(root, ".forge", "atlas.json"), + JSON.stringify({ version: 2, files: 3, symbols: [{ name: "a" }, { name: "b" }] }), + ); + assert.deepEqual(dashData(root, { nowDay: NOW }).atlas, { built: true, symbols: 2, files: 3 }); +}); + +test("dashData: corrupt/missing stores degrade to empty sections, never throw", () => { + const empty = dashData(tmp(), { nowDay: NOW }); + assert.equal(empty.ledger.stats.total, 0); + assert.deepEqual(empty.ledger.claims, []); + assert.deepEqual(empty.metrics, { stages: {}, recent: [] }); + + const root = tmp(); + mkdirSync(join(root, ".forge", "ledger"), { recursive: true }); + writeFileSync(join(root, ".forge", "ledger", "claims"), "not a directory"); // readdir throws + writeFileSync(join(root, ".forge", "metrics.jsonl"), "{nope\n{{also nope\n"); + writeFileSync(join(root, ".forge", "atlas.json"), "{corrupt"); + const d = dashData(root, { nowDay: NOW }); + assert.deepEqual(d.ledger.claims, [], "broken ledger → empty section"); + assert.deepEqual(d.metrics.recent, [], "corrupt metrics lines are skipped"); + assert.deepEqual(d.atlas, { built: false, symbols: 0, files: 0 }); +}); + +test("dashData: claims list is capped at 200 while stats keep the true total", () => { + const root = tmp(); + const dir = repoLedger(root); + for (let i = 0; i < 205; i++) mint(dir, `n${i}`, `fact number ${i}`); + const d = dashData(root, { nowDay: NOW }); + assert.equal(d.ledger.claims.length, 200); + assert.equal(d.ledger.stats.total, 205); +}); + +test("serve: / is the page, /api/data is the payload, unknown routes 404, GET-only", async () => { + const { root } = fixture(); + const server = serve(root, { port: 0 }); // ephemeral port, localhost-only default + await new Promise((resolve) => server.on("listening", resolve)); + const addr = /** @type {import("node:net").AddressInfo} */ (server.address()); + const base = `http://127.0.0.1:${addr.port}`; + try { + const page = await fetch(`${base}/`); + assert.equal(page.status, 200); + assert.match(page.headers.get("content-type"), /text\/html/); + const html = await page.text(); + assert.match(html, /forge/); + assert.doesNotMatch(html, /https?:\/\/(?!localhost)/, "self-contained — no CDN, no remote"); + + const data = await fetch(`${base}/api/data`); + assert.equal(data.status, 200); + const d = await data.json(); + assert.equal(d.ledger.stats.total, 3); + + assert.equal((await fetch(`${base}/api/impact`)).status, 400, "missing target"); + assert.equal((await fetch(`${base}/api/impact?target=x`)).status, 404, "no atlas yet"); + assert.equal((await fetch(`${base}/nope`)).status, 404); + assert.equal( + (await fetch(`${base}/api/data`, { method: "POST" })).status, + 404, + "no writes in this phase", + ); + } finally { + server.close(); + } +}); + +test("serve: /api/impact traces blast radius once an atlas exists", async () => { + const root = tmp(); + mkdirSync(join(root, "src"), { recursive: true }); + writeFileSync(join(root, "src", "a.js"), "export function core() { return 1; }\n"); + writeFileSync(join(root, "src", "b.js"), 'import { core } from "./a.js";\ncore();\n'); + const { build } = await import("../src/atlas.js"); + build({ root }); + const server = serve(root, { port: 0 }); + await new Promise((resolve) => server.on("listening", resolve)); + const addr = /** @type {import("node:net").AddressInfo} */ (server.address()); + try { + const res = await fetch(`http://127.0.0.1:${addr.port}/api/impact?target=core`); + assert.equal(res.status, 200); + const r = await res.json(); + assert.equal(r.found, true); + assert.ok(r.impactedFiles.includes("src/b.js"), "dependent file is in the radius"); + } finally { + server.close(); + } +}); diff --git a/test/diagnose.test.js b/test/diagnose.test.js new file mode 100644 index 0000000..f673c4e --- /dev/null +++ b/test/diagnose.test.js @@ -0,0 +1,186 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { appendFileSync, mkdirSync, mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { + diagnose, + failureSignature, + failuresPath, + normalizeError, + RING_SIZE, + readFailures, + recordFailure, + THRASH_K, +} from "../src/diagnose.js"; +import { loadClaims, repoLedger } from "../src/ledger_store.js"; + +const fixture = () => mkdtempSync(join(tmpdir(), "forge-diagnose-")); + +// --------------------------------------------------------------------------- +// failureSignature — pure, and stable under everything that varies between two +// runs of the SAME broken code (the whole point: recurrences must accumulate). +// --------------------------------------------------------------------------- + +test("failureSignature is stable under line/col numbers, hex addresses, timestamps, absolute paths", () => { + const where = { file: "src/auth.js", symbol: "verifyToken" }; + const a = failureSignature( + "TypeError: x is undefined\n at verifyToken (/home/alice/repo/src/auth.js:12:5) 0xdeadbeef 2026-07-07T10:00:00Z", + where, + ); + const b = failureSignature( + "TypeError: x is undefined\n at verifyToken (/Users/bob/work/src/auth.js:99:21) 0x1234abcd 2026-07-08T23:59:59Z", + where, + ); + assert.equal(a, b, "volatile parts must not change the signature"); +}); + +test("failureSignature distinguishes error class, file, and symbol", () => { + const base = failureSignature("TypeError: x is undefined", { file: "a.js", symbol: "f" }); + assert.notEqual(base, failureSignature("RangeError: y overflow", { file: "a.js", symbol: "f" })); + assert.notEqual( + base, + failureSignature("TypeError: x is undefined", { file: "b.js", symbol: "f" }), + ); + assert.notEqual( + base, + failureSignature("TypeError: x is undefined", { file: "a.js", symbol: "g" }), + ); +}); + +test("normalizeError keeps the basename (signal) while dropping the machine prefix", () => { + const n = normalizeError("Error at /home/ci/build/src/auth.js:3:1 after 120ms"); + assert.match(n, /auth\.js/); + assert.doesNotMatch(n, /home|ci|build|120|:3/); +}); + +// --------------------------------------------------------------------------- +// recordFailure — append-only trace with a corrupt-tolerant reader. +// --------------------------------------------------------------------------- + +test("recordFailure counts recurrences of the same signature", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", t: 1 }; + assert.equal(recordFailure(root, f).count, 1); + assert.equal(recordFailure(root, f).count, 2); + const other = recordFailure(root, { ...f, errorText: "RangeError: other" }); + assert.equal(other.count, 1, "a different signature counts separately"); + assert.equal(recordFailure(root, f).count, 3); +}); + +test("recordFailure tolerates corrupt trace lines (killed-process append)", () => { + const root = fixture(); + recordFailure(root, { errorText: "boom", t: 1 }); + appendFileSync(failuresPath(root), '{"truncated: \n'); + const r = recordFailure(root, { errorText: "boom", t: 2 }); + assert.equal(r.count, 2, "the corrupt line is skipped, not fatal"); + assert.equal(readFailures(root).length, 2); +}); + +test("recordFailure only counts within the last RING_SIZE entries", () => { + const root = fixture(); + const f = { errorText: "old failure", file: "src/a.js" }; + recordFailure(root, { ...f, t: 1 }); + recordFailure(root, { ...f, t: 2 }); + for (let i = 0; i < RING_SIZE; i++) recordFailure(root, { errorText: `noise ${i}`, t: 3 + i }); + const r = recordFailure(root, { ...f, t: 999 }); + assert.equal(r.count, 1, "hits pushed out of the ring no longer count toward thrash"); +}); + +test("recordFailure never persists a secret-shaped error head", () => { + const root = fixture(); + recordFailure(root, { errorText: `auth failed: api_key="hunter2-super-secret"`, t: 1 }); + const [e] = readFailures(root); + assert.match(e.head, /redacted/); + assert.doesNotMatch(JSON.stringify(e), /hunter2/); +}); + +// --------------------------------------------------------------------------- +// diagnose — thrash at k=3 mints ONE content-addressed diagnosis claim. +// --------------------------------------------------------------------------- + +test("diagnose stays quiet below the thrash threshold", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", nowDay: 10 }; + for (let i = 1; i < THRASH_K; i++) { + const r = diagnose(root, { ...f, t: i }); + assert.equal(r.thrash, false); + assert.equal(r.count, i); + assert.equal(r.escalate, undefined); + } + assert.equal(loadClaims(repoLedger(root)).length, 0, "no claim minted before thrash"); +}); + +test("diagnose at the 3rd recurrence mints a diagnosis claim and says STOP + escalate one tier", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", nowDay: 10 }; + let r; + for (let i = 1; i <= THRASH_K; i++) r = diagnose(root, { ...f, t: i }); + assert.equal(r.thrash, true); + assert.equal(r.count, THRASH_K); + assert.match(r.escalate, /STOP retrying/i); + assert.match(r.escalate, /ONE model tier/i); + assert.match(r.escalate, /diagnosis as the head/i); + const claims = loadClaims(repoLedger(root)); + assert.equal(claims.length, 1); + assert.equal(claims[0].kind, "diagnosis"); + assert.equal(claims[0].id, r.claimId); + assert.equal(claims[0].body.signature, r.signature); + assert.deepEqual(claims[0].body.triedFixes, []); + assert.equal(claims[0].provenance.agent, "doomloop"); +}); + +test("diagnose is idempotent — further recurrences resolve to the SAME claim", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", nowDay: 10 }; + let third; + for (let i = 1; i <= THRASH_K; i++) third = diagnose(root, { ...f, t: i }); + const fourth = diagnose(root, { ...f, t: THRASH_K + 1 }); + assert.equal(fourth.thrash, true); + assert.equal(fourth.claimId, third.claimId, "content addressing dedupes the mint"); + assert.equal(loadClaims(repoLedger(root)).length, 1, "no duplicate diagnosis claims"); +}); + +test("diagnose prefers the caller's root-cause note over the error head", () => { + const root = fixture(); + const f = { + errorText: "boom", + note: "circular import between auth and session", + nowDay: 1, + }; + let r; + for (let i = 1; i <= THRASH_K; i++) r = diagnose(root, { ...f, t: i }); + const [claim] = loadClaims(repoLedger(root)); + assert.equal(claim.body.note, f.note); + assert.equal(claim.id, r.claimId); +}); + +// --------------------------------------------------------------------------- +// CLI — forge diagnose "" [--file f] [--symbol s] [--json] +// --------------------------------------------------------------------------- + +const CLI = fileURLToPath(new URL("../src/cli.js", import.meta.url)); +const runCli = (args, cwd) => spawnSync("node", [CLI, ...args], { cwd, encoding: "utf8" }); + +test("forge diagnose: records, then escalates on the 3rd identical failure", () => { + const cwd = fixture(); + mkdirSync(join(cwd, ".forge"), { recursive: true }); + const args = ["diagnose", "TypeError: boom", "--file", "src/a.js", "--symbol", "f"]; + let out; + for (let i = 0; i < THRASH_K; i++) out = runCli(args, cwd); + assert.equal(out.status, 0, "advisory — never fails the process"); + assert.match(out.stdout, /doom-loop/); + assert.match(out.stdout, /STOP retrying/i); + const j = JSON.parse(runCli([...args, "--json"], cwd).stdout); + assert.equal(j.thrash, true); + assert.ok(j.claimId); +}); + +test("forge diagnose: no error text prints usage and exits 1", () => { + const cwd = fixture(); + const r = runCli(["diagnose"], cwd); + assert.equal(r.status, 1); + assert.match(r.stderr, /usage: forge diagnose/); +}); diff --git a/test/imagine.test.js b/test/imagine.test.js new file mode 100644 index 0000000..b091061 --- /dev/null +++ b/test/imagine.test.js @@ -0,0 +1,127 @@ +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { test } from "node:test"; +import { build } from "../src/atlas.js"; +import { imagineTask, renderImagine, selectTests, selectTestsReport } from "../src/imagine.js"; + +const fixture = () => mkdtempSync(join(tmpdir(), "forge-imagine-")); + +const put = (root, rel, text) => { + mkdirSync(dirname(join(root, rel)), { recursive: true }); + writeFileSync(join(root, rel), text); +}; + +// --------------------------------------------------------------------------- +// selectTests — weighted greedy set cover (weight = file size). +// --------------------------------------------------------------------------- + +test("selectTests covers every impacted source via its cheapest covering test", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/b.js", "export function b() {}\n"); + // src/a.js has TWO covering candidates; the smaller one must win the cover. + put(root, "src/a.test.js", "// tiny\n"); + put(root, "test/a.js", `// huge duration proxy\n${"x".repeat(4000)}\n`); + put(root, "src/b.test.js", "// b test\n"); + const tests = selectTests(root, ["src/a.js", "src/b.js"]); + assert.ok(tests.includes("src/a.test.js"), "cheapest cover for src/a.js chosen"); + assert.ok(!tests.includes("test/a.js"), "redundant heavier test dropped"); + assert.ok(tests.includes("src/b.test.js")); + assert.equal(tests.length, 2, "minimal: one test per otherwise-uncovered source"); +}); + +test("selectTests orders the suite best-value-first (greedy pick order)", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/b.js", "export function b() {}\n"); + put(root, "src/a.test.js", "// tiny a test\n"); + put(root, "src/b.test.js", `// expensive b test\n${"y".repeat(4000)}\n`); + const tests = selectTests(root, ["src/a.js", "src/b.js"]); + assert.deepEqual(tests, ["src/a.test.js", "src/b.test.js"], "cheapest per covered file first"); +}); + +test("selectTests includes an impacted test file (it covers itself)", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/a.test.js", "// a test\n"); + put(root, "test/util.js", "// an impacted test file\n"); + const tests = selectTests(root, ["src/a.js", "test/util.js"]); + assert.ok(tests.includes("test/util.js"), "predicted-to-break test is part of the suite"); + assert.ok(tests.includes("src/a.test.js")); +}); + +test("selectTestsReport names the sources no known test covers, without blocking", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/a.test.js", "// a test\n"); + put(root, "src/orphan.js", "export function orphan() {}\n"); + const { tests, uncovered } = selectTestsReport(root, ["src/a.js", "src/orphan.js"]); + assert.deepEqual(tests, ["src/a.test.js"]); + assert.deepEqual(uncovered, ["src/orphan.js"], "the gap is surfaced, not silently dropped"); +}); + +test("selectTests on nothing impacted is an empty suite", () => { + assert.deepEqual(selectTests(fixture(), []), []); +}); + +// --------------------------------------------------------------------------- +// imagineTask — entities → impact → predicted breaks + dry-run suite + risk. +// --------------------------------------------------------------------------- + +function appFixture() { + const root = fixture(); + put(root, "src/util.js", "export function helper(x) {\n return x * 2;\n}\n"); + put( + root, + "src/app.js", + 'import { helper } from "./util.js";\nexport function runApp() {\n return helper(1);\n}\n', + ); + put(root, "src/app.test.js", 'import { runApp } from "./app.js";\nrunApp();\n'); + return { root, atlas: build({ root }) }; +} + +test("imagineTask predicts breaks with confidence and selects the minimal dry-run suite", () => { + const { root, atlas } = appFixture(); + const r = imagineTask(root, "update `helper` in src/util.js", { atlas }); + assert.equal(r.found, true); + assert.ok(r.targets.includes("helper")); + const files = r.predictedBreaks.map((b) => b.file); + assert.ok(files.includes("src/app.js"), "the caller of helper is a predicted break"); + for (const b of r.predictedBreaks) { + assert.ok(b.confidence > 0 && b.confidence <= 1, `confidence in (0,1]: ${b.confidence}`); + } + assert.ok(r.tests.includes("src/app.test.js"), "the impacted caller's test is in the suite"); + const sum = r.predictedBreaks.reduce((s, b) => s + b.confidence, 0); + assert.ok(Math.abs(r.riskScore - sum) < 1e-9, "riskScore = Σ confidence"); + assert.ok(r.riskScore > 0); +}); + +test("imagineTask predictedBreaks are sorted by confidence, descending", () => { + const { root, atlas } = appFixture(); + const r = imagineTask(root, "update `helper` in src/util.js", { atlas }); + for (let i = 1; i < r.predictedBreaks.length; i++) { + assert.ok(r.predictedBreaks[i - 1].confidence >= r.predictedBreaks[i].confidence); + } +}); + +test("imagineTask on a task naming nothing in the graph predicts nothing", () => { + const { root, atlas } = appFixture(); + const r = imagineTask(root, "polish `totallyUnknownThing` please", { atlas }); + assert.equal(r.found, false); + assert.deepEqual(r.predictedBreaks, []); + assert.deepEqual(r.tests, []); + assert.equal(r.riskScore, 0); + assert.match(renderImagine(r), /nothing in the code graph/); +}); + +test("renderImagine prints breaks, the suite, and the sandbox follow-up note", () => { + const { root, atlas } = appFixture(); + const out = renderImagine(imagineTask(root, "update `helper` in src/util.js", { atlas })); + assert.match(out, /predicted breaks/); + assert.match(out, /minimal dry-run suite/); + assert.match(out, /src\/app\.test\.js/); + assert.match(out, /risk score/); + assert.match(out, /P5 follow-up/); +}); diff --git a/test/uifingerprint.test.js b/test/uifingerprint.test.js new file mode 100644 index 0000000..6a36abc --- /dev/null +++ b/test/uifingerprint.test.js @@ -0,0 +1,245 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { loadClaims, repoLedger } from "../src/ledger_store.js"; +import { ASSERTABLE_CHECKS } from "../src/uicheck.js"; +import { + conformance, + fingerprintFiles, + fingerprintText, + GENERIC_SIGNATURES, + inferSpacingBase, + loadProjectFingerprint, + mintProjectFingerprint, + nearestGeneric, + onScaleFraction, + scaleChecks, + slopDistance, + UI_GATE_DEFAULTS, + uiGate, +} from "../src/uifingerprint.js"; + +const CLI = fileURLToPath(new URL("../src/cli.js", import.meta.url)); +const runCli = (args, cwd) => spawnSync("node", [CLI, ...args], { cwd, encoding: "utf8" }); +const tmp = () => mkdtempSync(join(tmpdir(), "forge-uifp-")); + +// The generic-template look: default-Tailwind blue/indigo, flat 8px spacing, one +// font, uniform rounded-xl, one soft shadow — must FAIL the gate. +const GENERIC_CSS = ` +.card { background: #ffffff; color: #3b82f6; border: 1px solid #6366f1; + padding: 16px; margin: 8px; gap: 32px; + font-family: Inter, sans-serif; + border-radius: 12px; + box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1); } +.hero { background: #6366f1; padding: 32px 16px; border-radius: 12px; } +`; + +// A distinctive custom system: warm ink/paper neutrals + a red accent, 4-based +// spacing with real jumps, two deliberate faces, near-square corners, no shadows — +// must PASS. +const CUSTOM_CSS = ` +:root { --ink: #1c1b18; --paper: #f5f1e8; --accent: #e63946; } +h1 { font-family: "Fraunces", serif; margin: 4px 0 20px; } +body { font-family: "Atkinson Hyperlegible", sans-serif; color: #1c1b18; + background: #f5f1e8; padding: 12px 28px; } +.note { border-radius: 2px; gap: 44px; border: 2px solid #e63946; } +`; + +const TW_JSX = `export const Card = () => ( +
    + +
    );`; + +test("fingerprintText: CSS fixture — palette normalized to HSL, spacing/type/shape extracted", () => { + const fp = fingerprintText(GENERIC_CSS); + // #fff, #3b82f6, #6366f1 (deduped), rgba(0,0,0) from the shadow. + assert.equal(fp.paletteSize, 4); + assert.ok( + fp.palette.some((c) => c.h === 217 && c.s === 91), + "blue-500 hex lands on h217", + ); + assert.ok( + fp.palette.some((c) => c.h === 239), + "indigo-500 hex lands on h239", + ); + assert.equal(fp.hueBuckets[7], 2, "both chromatic hues fall in the 210–239 bin"); + assert.deepEqual(fp.spacing, [8, 16, 32]); + assert.equal(fp.spacingBase, 8); + assert.equal(fp.spacingOnScale, 1); + assert.deepEqual(fp.fontFamilies, ["inter"]); + assert.deepEqual(fp.radii, [12]); + assert.equal(fp.radiusLevels, 1); + assert.equal(fp.shadowLevels, 1); +}); + +test("fingerprintText: Tailwind-class JSX — classes map to px / hues without any CSS", () => { + const fp = fingerprintText(TW_JSX); + assert.deepEqual(fp.spacing, [8, 16, 24, 32], "p-4/m-2/gap-6/px-8/py-2 → n×4 px"); + assert.equal(fp.spacingBase, 8); + assert.ok( + fp.palette.some((c) => c.h === 217), + "bg-blue-500 → h217", + ); + assert.ok( + fp.palette.some((c) => c.s === 0 && c.l === 100), + "text-white → neutral white", + ); + assert.deepEqual(fp.fontFamilies, ["sans-serif"]); + assert.deepEqual(fp.radii, [12], "rounded-xl → 12px"); + assert.equal(fp.shadowLevels, 1, "shadow-lg twice is ONE level"); +}); + +test("fingerprintText is deterministic (same text → deep-equal vector)", () => { + assert.deepEqual(fingerprintText(CUSTOM_CSS), fingerprintText(CUSTOM_CSS)); +}); + +test("inferSpacingBase: largest base that fits wins; off-scale falls back to argmin", () => { + assert.equal(inferSpacingBase([8, 16, 32]), 8); + assert.equal(inferSpacingBase([4, 12, 20]), 4, "multiples of 4 but not 8 → 4, not 2"); + assert.equal(inferSpacingBase([5, 13, 21]), 2, "nothing fits cleanly → smallest residual"); + assert.equal(inferSpacingBase([]), null); + assert.equal(onScaleFraction([8, 16, 31], 8), 2 / 3); +}); + +test("slopDistance: the generic fixture sits ON a signature; the custom one is far", () => { + const generic = fingerprintText(GENERIC_CSS); + const custom = fingerprintText(CUSTOM_CSS); + assert.ok(slopDistance(generic) < UI_GATE_DEFAULTS.tauSlop, "generic is inside the slop radius"); + assert.equal(nearestGeneric(generic)?.id, "tailwind-default"); + assert.ok(slopDistance(custom) >= UI_GATE_DEFAULTS.tauSlop, "custom clears the slop radius"); + assert.equal(slopDistance({}), 1, "nothing measurable is not generic"); +}); + +test("uiGate: known-generic FAILS with named, actionable violations", () => { + const gate = uiGate(fingerprintText(GENERIC_CSS)); + assert.equal(gate.pass, false); + const byFeature = Object.fromEntries(gate.violations.map((v) => [v.feature, v])); + assert.ok(byFeature.palette, "the default-Tailwind palette is named"); + assert.match(byFeature.palette.hint, /brand hue/); + assert.ok(byFeature.spacing); + assert.match(byFeature.spacing.hint, /8px/); + for (const v of gate.violations) assert.ok(v.hint.length > 0, "every violation carries a fix"); +}); + +test("uiGate: distinctive custom fixture PASSES (slop-only and vs its own system)", () => { + const custom = fingerprintText(CUSTOM_CSS); + assert.equal(uiGate(custom).pass, true); + const gate = uiGate(custom, { projectFp: custom }); + assert.equal(gate.pass, true); + assert.equal(gate.conform, 0, "a fingerprint conforms perfectly to itself"); +}); + +test("conformance: flags divergence from a provided project fingerprint", () => { + const generic = fingerprintText(GENERIC_CSS); + const project = fingerprintText(CUSTOM_CSS); + assert.equal(conformance(project, project), 0); + assert.ok(conformance(generic, project) > UI_GATE_DEFAULTS.tauConform); + const gate = uiGate(generic, { projectFp: project }); + assert.equal(gate.pass, false); + const type = gate.violations.find((v) => v.feature === "type" && /project/.test(v.detail)); + assert.ok(type, "font divergence from the project is a named violation"); + assert.match(type.hint, /fraunces/); +}); + +test("scaleChecks: off-scale spacing, level sprawl, and palette bloat all fail with hints", () => { + const messy = { + palette: [], + paletteSize: 12, + spacing: [8, 16, 31], + spacingBase: 8, + fontFamilies: [], + radii: [1, 2, 3, 4, 6], + radiusLevels: 5, + shadowLevels: 5, + }; + const byId = Object.fromEntries(scaleChecks(messy).map((c) => [c.id, c])); + assert.equal(byId["spacing-scale"].pass, false); + assert.match(byId["spacing-scale"].hint, /31/); + assert.equal(byId["radius-levels"].pass, false); + assert.equal(byId["shadow-levels"].pass, false); + assert.equal(byId["palette-size"].pass, false); + assert.ok(scaleChecks(fingerprintText(CUSTOM_CSS)).every((c) => c.pass)); +}); + +test("scaleChecks ids are wired into uicheck's ASSERTABLE_CHECKS (no drift)", () => { + const assertable = new Set(ASSERTABLE_CHECKS.map((c) => c.id)); + for (const c of scaleChecks(fingerprintText(GENERIC_CSS))) + assert.ok(assertable.has(c.id), `${c.id} missing from ASSERTABLE_CHECKS`); + assert.ok(assertable.has("contrast"), "the existing WCAG surface is untouched"); +}); + +test("generic signatures document their rationale", () => { + for (const sig of GENERIC_SIGNATURES) { + assert.ok(sig.why.length > 20, `${sig.id} needs a why`); + assert.ok(sig.hues.length >= 1); + } +}); + +test("mintProjectFingerprint: claim lands in the ledger; id stable for the same inputs", () => { + const root = tmp(); + writeFileSync(join(root, "app.css"), CUSTOM_CSS); + const a = mintProjectFingerprint(root, ["app.css"], { t: 1 }); + assert.equal(a.ok, true); + if (!a.ok) return; + assert.equal(a.existed, false); + const b = mintProjectFingerprint(root, ["app.css"], { t: 2 }); + if (!b.ok) assert.fail(b.reason); + else { + assert.equal(b.id, a.id, "content-addressed: same UI surface → same claim id"); + assert.equal(b.existed, true); + } + const claims = loadClaims(repoLedger(root)).filter((c) => c.kind === "fingerprint"); + assert.equal(claims.length, 1); + assert.equal(claims[0].provenance.agent, "uicheck"); + assert.deepEqual(loadProjectFingerprint(root), a.fingerprint); +}); + +test("fingerprintFiles: argument order never changes the vector; missing files are skipped", () => { + const root = tmp(); + writeFileSync(join(root, "a.css"), GENERIC_CSS); + writeFileSync(join(root, "b.jsx"), TW_JSX); + assert.deepEqual( + fingerprintFiles(root, ["a.css", "b.jsx", "ghost.css"]), + fingerprintFiles(root, ["b.jsx", "a.css"]), + ); +}); + +test("cli: legacy `uicheck ` and new `uicheck contrast ` both work", () => { + const cwd = tmp(); + const legacy = runCli(["uicheck", "#000000", "#ffffff"], cwd); + assert.equal(legacy.status, 0); + assert.match(legacy.stdout, /21:1/); + const named = runCli(["uicheck", "contrast", "#000000", "#ffffff"], cwd); + assert.equal(named.status, 0); + assert.match(named.stdout, /21:1/); +}); + +test("cli: `uicheck design` exits 1 on the generic fixture, names the fixes", () => { + const cwd = tmp(); + writeFileSync(join(cwd, "generic.css"), GENERIC_CSS); + const r = runCli(["uicheck", "design", "generic.css"], cwd); + assert.equal(r.status, 1); + assert.match(r.stdout, /slop distance/); + assert.match(r.stdout, /fix:/); + assert.match(r.stdout, /FAIL/); +}); + +test("cli: fingerprint --mint stores the project claim; design then gates against it", () => { + const cwd = tmp(); + writeFileSync(join(cwd, "app.css"), CUSTOM_CSS); + const mint = runCli(["uicheck", "fingerprint", "app.css", "--mint", "--json"], cwd); + assert.equal(mint.status, 0); + const out = JSON.parse(mint.stdout); + assert.equal(out.minted.ok, true); + assert.ok(out.fingerprint.paletteSize > 0); + const design = runCli(["uicheck", "design", "app.css", "--json"], cwd); + assert.equal(design.status, 0, design.stdout); + const gate = JSON.parse(design.stdout); + assert.equal(gate.pass, true); + assert.equal(gate.hasProjectFingerprint, true); + assert.equal(gate.conform, 0); +}); diff --git a/test/verify.test.js b/test/verify.test.js index d12e32f..38a2e2b 100644 --- a/test/verify.test.js +++ b/test/verify.test.js @@ -37,3 +37,42 @@ test("shared extractor: atlas and verify use the same call-site extraction (no d assert.ok(!syms.includes("JSON"), "builtins ignored"); assert.ok(CALL_IGNORE.has("console")); }); + +// --------------------------------------------------------------------------- +// M6 — checkpoint cadence (optimal-stopping threshold rule, pure). +// --------------------------------------------------------------------------- + +test("checkpointCadence computes n* = ceil(checkCost / (pErr·tokensPerStep·costPerToken))", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + // risk per step = 0.05 · 200 · 1 = 10 → n* = 100/10 = 10 + assert.equal(checkpointCadence({ pErr: 0.05, tokensPerStep: 200, checkCost: 100 }), 10); + // non-integer ratio rounds UP — checking a step late is worse than a step early + assert.equal(checkpointCadence({ pErr: 0.05, tokensPerStep: 200, checkCost: 105 }), 11); + // costPerToken scales the at-risk side + assert.equal( + checkpointCadence({ pErr: 0.05, tokensPerStep: 200, costPerToken: 2, checkCost: 100 }), + 5, + ); +}); + +test("checkpointCadence: riskier (cheaper) tiers checkpoint more often", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + const haiku = checkpointCadence({ pErr: 0.2, tokensPerStep: 500, checkCost: 400 }); + const opus = checkpointCadence({ pErr: 0.01, tokensPerStep: 500, checkCost: 400 }); + assert.ok(haiku < opus, `higher hazard → smaller n* (${haiku} < ${opus})`); +}); + +test("checkpointCadence clamps to [1, 50]", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + // near-free check → never below every-step + assert.equal(checkpointCadence({ pErr: 0.5, tokensPerStep: 1000, checkCost: 0 }), 1); + // near-riskless run (or pErr measured at 0) → still checkpoints by the ceiling + assert.equal(checkpointCadence({ pErr: 0, tokensPerStep: 1000, checkCost: 100 }), 50); + assert.equal(checkpointCadence({ pErr: 1e-9, tokensPerStep: 1, checkCost: 100 }), 50); +}); + +test("checkpointCadence fails safe on degenerate inputs (check every step)", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + assert.equal(checkpointCadence({ pErr: Number.NaN, tokensPerStep: 100, checkCost: 100 }), 1); + assert.equal(checkpointCadence({ pErr: 0, tokensPerStep: 100, checkCost: 0 }), 1); +});