From 5f29ff5933b3a6518d98ab2fd4549f0dd540675d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 17:13:08 +0000 Subject: [PATCH 1/5] =?UTF-8?q?feat:=20measured=20cost=20report=20(P8)=20?= =?UTF-8?q?=E2=80=94=20forge=20cost=20--stages=20over=20stage-tagged=20met?= =?UTF-8?q?rics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-stage cost factors computed as pure arithmetic over .forge/metrics.jsonl (src/cost_report.js): gate halt rate, tier-weighted cache hit rate, route saving priced against the always-premium baseline, and context assembly. Composition C = C0 * prod(1 - f_i) runs over ONLY the measured stages — an eventless stage reports measured:false / value:null and lands in the caveats, never a default. The render cites the paper's 62% routing figure as context (not a local measurement) and prints ~90% only as a labeled target. substrateCheck now meters the assumption gate on the explicit path (halt/pass, try/catch best-effort; ambient hooks stay write-free, same contract as reuseQuery vs reusePeek). recordGate/recordRoute give future stage wiring one obvious call each. reports/cost-eval.md scaffolds the paired-run harness report with a truthful no-data-yet state and the commands that populate it. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Fc2MvWJbQ1cyNQ76ymv4hs --- CHANGELOG.md | 13 +++ reports/cost-eval.md | 66 ++++++++++++ src/cli.js | 10 +- src/cost_report.js | 207 ++++++++++++++++++++++++++++++++++++ src/substrate.js | 9 ++ test/cost_report.test.js | 223 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 527 insertions(+), 1 deletion(-) create mode 100644 reports/cost-eval.md create mode 100644 src/cost_report.js create mode 100644 test/cost_report.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 917c5f7..ee935c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,19 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **Measured cost report (P8 of the substrate-v2 plan).** `forge cost --stages [--json]` + computes per-stage cost factors as pure arithmetic over `.forge/metrics.jsonl` + (`src/cost_report.js`): gate halt rate, tier-weighted cache hit rate (exact 1.0 / near + 0.85 / adapt 0.5), route saving priced against the always-premium baseline, and context + assembly — then composes `C = C₀ · Π(1 − fᵢ)` over ONLY the measured stages. A stage with + no events reports "no data", never a default; the composed figure is a lower bound whose + caveats name every unmeasured stage; the paper's 62 % routing figure is cited as context, + and ~90 % appears only as a labeled target. `substrateCheck` now meters the assumption + gate on the explicit path (one `gate` halt/pass line per decision; ambient hooks stay + write-free), `recordGate`/`recordRoute` give future stage wiring one obvious call each, + and `reports/cost-eval.md` scaffolds the paired-run harness report with a truthful + empty state. + - **Proof-carrying reuse cache (P3 of the substrate-v2 plan).** `forge reuse` turns "reuse already-generated code" from prose into a deterministic system: verified code becomes an `artifact` claim keyed by a normalized task fingerprint (volatile literals → diff --git a/reports/cost-eval.md b/reports/cost-eval.md new file mode 100644 index 0000000..9bfe203 --- /dev/null +++ b/reports/cost-eval.md @@ -0,0 +1,66 @@ +# Cost evaluation — measured stage factors (P8) + +> Status: **no data yet.** This document is the artifact the P8 harness +> ([docs/plans/substrate-v2/05-cost-model.md](../docs/plans/substrate-v2/05-cost-model.md) §3) +> fills with measurements. Until a cell below holds a measured number, the only claimable +> figures are the paper's: 62 % routing saving on live tokens (paper §9). The plan's ~90 % +> composed figure is a **target**, not a result, and does not appear in this table. + +## Methodology + +The cost model is multiplicative — `C = C₀ · Π(1 − fᵢ)` over independent stages — so each +stage factor is measured separately and composed arithmetically, never asserted: + +1. **Instrumentation.** Every substrate stage appends one line to `.forge/metrics.jsonl` + (`{t, stage, outcome, tokensIn, tokensOut, tier, savedEstimate, ref}` — `src/metrics.js`). + `forge cost --stages` computes the per-stage factors from those lines (`src/cost_report.js`); + a stage with no events reports **no data**, never a default. +2. **Paired runs.** Baseline (always-premium, read-everything, no cache) vs. substrate over + the same replay corpus (N ≥ 100 real tasks, stratified repeat-heavy / mixed / cold), the + paper §9 methodology: identical tokens repriced, so every saving is arithmetic on measured + tokens. +3. **Correctness guard (spec §3).** A saving counts only if the external verifier passes the + output. A routed-down answer that fails is not a saving; a cache hit that gets reverted is + recorded as a *negative* entry. + +## Measured factors + +| stage | factor | events | measured? | +|---|---|---|---| +| gate (M2 halt rate) | — | 0 | no data yet — run with metrics enabled | +| cache (reuse, tier-weighted) | — | 0 | no data yet — run with metrics enabled | +| route (vs always-premium) | — | 0 | no data yet — run with metrics enabled | +| context (assembly ρ) | — | 0 | no data yet — run with metrics enabled | +| **composed (measured stages only)** | — | 0 | nothing to compose yet | + +Secondary counters (doom-loop halts avoided, M5 lean, avoided rework) are reported alongside +when populated — they are deliberately excluded from the multiplication (spec §1). + +## How to populate this table + +Metrics accrue as the substrate is actually used — each command below appends stage-tagged +lines to `.forge/metrics.jsonl`: + +```sh +# gate + cache: every explicit pre-action check meters both stages +forge substrate "" + +# cache: explicit reuse queries and mints +forge reuse query "" + +# then read the measured factors (and paste them here): +forge cost --stages # human table +forge cost --stages --json # machine-readable, for this report +``` + +Route and context events are emitted via `recordRoute` / a future context-assembly hook +(`src/cost_report.js`) as those stages gain live wiring. + +## Caveats that ship with any number placed here + +- Stage rates are **workload-dependent**: factors describe the recorded traffic of one repo, + not a general claim (spec §2 — repeat-heavy warm-ledger workloads differ from cold starts). +- The composed reduction is a **lower bound from measured stages only**; unmeasured stages + contribute nothing rather than a target. +- Until the paired-run harness with the correctness guard has run, per-stage factors from + live metrics are unguarded observational numbers, not eval results. diff --git a/src/cli.js b/src/cli.js index 64fc182..4ab5da7 100755 --- a/src/cli.js +++ b/src/cli.js @@ -16,7 +16,7 @@ const COMMANDS = { harden: "wire security controls — gitleaks pre-commit + sandbox settings", remember: "add a durable fact to this repo's portable memory (forge brain)", brain: "show / rebuild the portable project memory index", - cost: "real per-day spend via ccusage + the cost ceiling", + cost: "real per-day spend via ccusage + measured stage factors (--stages)", spec: "spec-as-contract — init (OpenSpec) / lock / check drift", cortex: "self-correcting project memory — status / why ", ledger: "proof-carrying memory — stats / verify / show / blame / query / merge / import", @@ -556,6 +556,14 @@ async function run(argv) { return; } if (cmd === "cost") { + // `--stages` is the P8 measured report (per-stage factors from .forge/metrics.jsonl); + // the default path stays the ccusage per-day spend view, untouched. + if (argv.includes("--stages")) { + const { renderCostReport, report } = await import("./cost_report.js"); + const r = report(process.cwd()); + console.log(argv.includes("--json") ? JSON.stringify(r, null, 2) : renderCostReport(r)); + return; + } const { execFileSync } = await import("node:child_process"); const run = (bin, args) => execFileSync(bin, args, { encoding: "utf8", stdio: "pipe" }); console.log(`${BRAND.brand} cost — real per-day spend (ccusage)\n`); diff --git a/src/cost_report.js b/src/cost_report.js new file mode 100644 index 0000000..8bf021d --- /dev/null +++ b/src/cost_report.js @@ -0,0 +1,207 @@ +// forge cost report — the P8 measured-stage report (docs/plans/substrate-v2/05-cost-model.md). +// The cost model is multiplicative: C = C₀ · Π(1 − fᵢ) over independent stages. The discipline +// this module enforces is the paper's (§4, C6): a number is an assumption until measured. Every +// factor here is ARITHMETIC over .forge/metrics.jsonl lines that stages actually emitted; a +// stage with no events reports measured:false and value:null — it is never guessed, defaulted, +// or backfilled from a target. The ~90 % figure in the plan stays a TARGET everywhere in this +// module's output; the only measured external figure (62 % routing, paper §9) is cited as +// context, clearly labeled as not-local. +import { read, record } from "./metrics.js"; +import { MODELS } from "./model_tiers.js"; + +/** Saving weight per cache-hit tier — must stay consistent with reuse.js savedEstimate + * (exact = full regeneration avoided; near/adapt still spend adaptation tokens). */ +export const CACHE_TIER_SAVINGS = { hit_exact: 1.0, hit_near: 0.85, hit_adapt: 0.5 }; + +// The route factor's baseline is "always-premium": the tier an unrouted agent defaults to. +// That is the complex tier (opus), NOT the extreme tier — pricing the baseline at the +// rarely-justified top model would flatter the savings, and honesty is the point. +const ROUTE_BASELINE_KEY = "opus"; + +/** Resolve a metrics `tier` field to a pricing row — accepts the model key ("haiku") + * or the tier name ("simple"), since both appear in route results. */ +const modelForTier = (tier) => + MODELS[tier] ?? Object.values(MODELS).find((m) => m.tier === tier) ?? null; + +const tokenCost = (m, tokensIn, tokensOut) => tokensIn * m.inCost + tokensOut * m.outCost; + +const unmeasured = () => ({ measured: false, value: null, events: 0 }); + +/** + * Per-stage measured factors from .forge/metrics.jsonl. Each factor is + * {measured:boolean, value:number|null, events:number} — `events` counts only the lines + * usable for that stage's arithmetic, and a stage with none reports measured:false / + * value:null. NEVER invents a number. + * + * - gate: fraction of "gate" events with outcome "halt" — requests where spend was avoided + * entirely. (This measures h_gate; the plan's g ≈ per-halt spend share is taken as 1 — + * i.e. a halted task is assumed to have cost an average task, the simplest honest reading + * until the paired harness prices halts individually.) + * - cache: hit rate weighted by tier savings (exact 1.0 / near 0.85 / adapt 0.5); a miss + * contributes 0 to the numerator but counts in the denominator. + * - route: 1 − (actual token cost / always-premium cost) over "route" events that carry a + * resolvable tier AND real token counts — events without tokens can't be priced and are + * excluded rather than estimated. + * - context: fraction of would-have-been input tokens avoided, Σsaved / (Σsaved + Σactual), + * over "context" events carrying both savedEstimate and tokensIn. + * @param {string} root + */ +export function stageFactors(root) { + const gateEvents = read(root, { stage: "gate" }); + const gate = gateEvents.length + ? { + measured: true, + value: gateEvents.filter((e) => e.outcome === "halt").length / gateEvents.length, + events: gateEvents.length, + } + : unmeasured(); + + const cacheEvents = read(root, { stage: "cache" }); + const cache = cacheEvents.length + ? { + measured: true, + value: + cacheEvents.reduce((s, e) => s + (CACHE_TIER_SAVINGS[e.outcome] ?? 0), 0) / + cacheEvents.length, + events: cacheEvents.length, + } + : unmeasured(); + + const baseline = MODELS[ROUTE_BASELINE_KEY]; + const routeEvents = read(root, { stage: "route" }).filter( + (e) => + modelForTier(e.tier) && + Number.isFinite(e.tokensIn) && + Number.isFinite(e.tokensOut) && + e.tokensIn + e.tokensOut > 0, + ); + const route = routeEvents.length + ? { + measured: true, + value: + 1 - + routeEvents.reduce( + (s, e) => s + tokenCost(modelForTier(e.tier), e.tokensIn, e.tokensOut), + 0, + ) / + routeEvents.reduce((s, e) => s + tokenCost(baseline, e.tokensIn, e.tokensOut), 0), + events: routeEvents.length, + } + : unmeasured(); + + const ctxEvents = read(root, { stage: "context" }).filter( + (e) => Number.isFinite(e.savedEstimate) && Number.isFinite(e.tokensIn), + ); + const ctxSaved = ctxEvents.reduce((s, e) => s + e.savedEstimate, 0); + const ctxActual = ctxEvents.reduce((s, e) => s + e.tokensIn, 0); + const context = + ctxEvents.length && ctxSaved + ctxActual > 0 + ? { measured: true, value: ctxSaved / (ctxSaved + ctxActual), events: ctxEvents.length } + : unmeasured(); + + return { gate, cache, route, context }; +} + +/** + * The multiplicative composition C = C₀ · Π(1 − fᵢ) over ONLY the measured factors. + * Honest framing: because unmeasured stages contribute exactly nothing (factor 0, not a + * target), the result is a LOWER BOUND built from measured stages only — it can only grow + * as more stages start emitting metrics, and it is never the plan's ~90 % target restated. + * @param {ReturnType} factors + * @returns {{measuredReduction:number, stagesIncluded:string[], stagesMissing:string[]}} + */ +export function composedReduction(factors) { + const stagesIncluded = []; + const stagesMissing = []; + let remaining = 1; + for (const [name, f] of Object.entries(factors)) { + if (f.measured && typeof f.value === "number") { + stagesIncluded.push(name); + remaining *= 1 - f.value; + } else stagesMissing.push(name); + } + return { measuredReduction: 1 - remaining, stagesIncluded, stagesMissing }; +} + +/** + * Assemble the full report: factors, composition, raw totals, and a caveat per unmeasured + * stage plus the workload-dependence caveat — the caveats ship WITH the numbers so no + * consumer can quote the reduction without its conditions. + * @param {string} root + */ +export function report(root) { + const factors = stageFactors(root); + const composed = composedReduction(factors); + const all = read(root); + const totals = { + events: all.length, + savedEstimateTokens: all.reduce( + (s, e) => s + (Number.isFinite(e.savedEstimate) ? e.savedEstimate : 0), + 0, + ), + }; + const caveats = composed.stagesMissing.map( + (s) => + `stage "${s}" has no recorded events — unmeasured, contributes nothing to the composition`, + ); + caveats.push( + "stage rates are workload-dependent: these factors describe this repo's recorded traffic, not a general claim (05-cost-model.md §2)", + "the composed figure is a lower bound from measured stages only; savings are not correctness-guarded until the P8 paired harness runs (05-cost-model.md §3)", + ); + return { factors, composed, totals, caveats }; +} + +const pct = (v) => `${(v * 100).toFixed(1)}%`; + +/** + * Human rendering. Register matters as much as arithmetic: measured numbers print as + * measurements, the paper's 62 % routing figure prints as CONTEXT (a citation, not a local + * result), and the ~90 % figure appears only with the word "target" in front of it — this + * report never claims it as achieved. + * @param {ReturnType} r + */ +export function renderCostReport(r) { + const lines = ["Forge cost — measured stage factors (.forge/metrics.jsonl)", ""]; + lines.push(` ${"stage".padEnd(9)} ${"factor".padEnd(10)} events`); + for (const [name, f] of Object.entries(r.factors)) { + const shown = f.measured && typeof f.value === "number" ? pct(f.value) : "no data"; + lines.push(` ${name.padEnd(9)} ${shown.padEnd(10)} ${f.events}`); + } + lines.push(""); + lines.push( + r.composed.stagesIncluded.length + ? ` composed measured reduction: ${pct(r.composed.measuredReduction)} (from: ${r.composed.stagesIncluded.join(", ")}) — lower bound, measured stages only` + : " composed measured reduction: 0.0% — no stage has recorded events yet", + ); + lines.push( + ` totals: ${r.totals.events} metric event(s) · ~${r.totals.savedEstimateTokens} tokens saved (stage self-estimates)`, + ); + lines.push( + "", + " context (not a local measurement): the paper measured a 62% routing saving on live tokens (paper §9)", + " target (unmet until measured): the plan's composed target is ~90% (docs/plans/substrate-v2/05-cost-model.md)", + ); + lines.push("", " caveats:"); + for (const c of r.caveats) lines.push(` - ${c}`); + return lines.join("\n"); +} + +// --------------------------------------------------------------------------- +// Emit-side helpers — one obvious call per stage, so future wiring (context assembly, +// real route execution) adds a single line instead of re-deriving the schema. Thin +// wrappers over metrics.record, which is already best-effort (never throws). +// --------------------------------------------------------------------------- + +/** Record one assumption-gate decision: halted = spend avoided. + * @param {string} root + * @param {{halted?: boolean, ref?: string}} [opts] */ +export function recordGate(root, { halted, ref } = {}) { + return record(root, { stage: "gate", outcome: halted ? "halt" : "pass", ref }); +} + +/** Record one routed generation with its tier and real token counts. + * @param {string} root + * @param {{tier?: string, tokensIn?: number, tokensOut?: number, ref?: string}} [opts] */ +export function recordRoute(root, { tier, tokensIn, tokensOut, ref } = {}) { + return record(root, { stage: "route", tier, tokensIn, tokensOut, ref }); +} diff --git a/src/substrate.js b/src/substrate.js index 00814ca..8259d5a 100644 --- a/src/substrate.js +++ b/src/substrate.js @@ -9,6 +9,7 @@ import { buildRunner, llmEnabled } from "./adjudicate.js"; import { goalDrift } from "./anchor.js"; import { build as buildAtlas, impact as impactGraph, load as loadAtlas } from "./atlas.js"; import { matchingLessons } from "./cortex.js"; +import { recordGate } from "./cost_report.js"; import { leanRepo } from "./lean.js"; import { load as loadLessons } from "./lessons_store.js"; import { clarifyBlock, preflightRepo, referencedEntities } from "./preflight.js"; @@ -192,6 +193,14 @@ export function substrateCheck( }; const entities = referencedEntities(text); const preflight = preflightRepo(root, text, { askThreshold, allowBuild, ...llmOpts }); + // P8 gate metering: one metrics line per explicit gate decision (halt = spend avoided). + // Same write contract as reuseQuery vs reusePeek below — the ambient hook path + // (allowBuild:false) never appends. Best-effort: measurement must never block the gate. + if (allowBuild) { + try { + recordGate(root, { halted: preflight.assumption.shouldAsk }); + } catch {} + } // Reuse the gap preflight already computed — routeTask would otherwise recompute it (and, with // FORGE_LLM on, fire a second, redundant assumption model call whose result it discards). const route = routeTask(root, text, { ...llmOpts, ambiguity: preflight.gap }); diff --git a/test/cost_report.test.js b/test/cost_report.test.js new file mode 100644 index 0000000..3c77f4e --- /dev/null +++ b/test/cost_report.test.js @@ -0,0 +1,223 @@ +import assert from "node:assert/strict"; +import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { + composedReduction, + recordGate, + recordRoute, + renderCostReport, + report, + stageFactors, +} from "../src/cost_report.js"; +import { read as readMetrics } from "../src/metrics.js"; +import { substrateCheck } from "../src/substrate.js"; + +const tmp = () => mkdtempSync(join(tmpdir(), "forge-cost-")); + +/** Write a fixture .forge/metrics.jsonl from entry objects. */ +function seed(root, entries) { + mkdirSync(join(root, ".forge"), { recursive: true }); + writeFileSync( + join(root, ".forge", "metrics.jsonl"), + `${entries.map((e) => JSON.stringify({ t: 1, ...e })).join("\n")}\n`, + ); +} + +// --- stageFactors: the arithmetic is exact over fixture lines ------------------------- + +test("stageFactors: gate factor is the exact halt fraction", () => { + const root = tmp(); + seed(root, [ + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + ]); + const f = stageFactors(root); + assert.deepEqual(f.gate, { measured: true, value: 2 / 5, events: 5 }); +}); + +test("stageFactors: cache factor is the tier-weighted hit rate (exact 1.0 / near 0.85 / adapt 0.5)", () => { + const root = tmp(); + seed(root, [ + { stage: "cache", outcome: "hit_exact" }, + { stage: "cache", outcome: "hit_near" }, + { stage: "cache", outcome: "hit_adapt" }, + { stage: "cache", outcome: "miss" }, + ]); + const f = stageFactors(root); + assert.equal(f.cache.measured, true); + assert.equal(f.cache.events, 4); + assert.ok(Math.abs(f.cache.value - (1.0 + 0.85 + 0.5) / 4) < 1e-12); +}); + +test("stageFactors: route factor prices tokens vs the always-premium baseline; unpriceable events excluded", () => { + const root = tmp(); + seed(root, [ + // haiku (in 1, out 5) vs opus baseline (in 5, out 25): 1000·1+1000·5=6000 vs 30000. + { stage: "route", tier: "haiku", tokensIn: 1000, tokensOut: 1000 }, + // Tier names resolve too ("simple" is haiku's tier) — but no tokens ⇒ excluded, not estimated. + { stage: "route", tier: "simple" }, + { stage: "route", tier: "nonsense", tokensIn: 50, tokensOut: 50 }, + ]); + const f = stageFactors(root); + assert.deepEqual(f.route, { measured: true, value: 1 - 6000 / 30000, events: 1 }); +}); + +test("stageFactors: context factor is saved / (saved + actual) input tokens", () => { + const root = tmp(); + seed(root, [ + { stage: "context", savedEstimate: 300, tokensIn: 700 }, + { stage: "context", savedEstimate: 0, tokensIn: 1000 }, + ]); + const f = stageFactors(root); + assert.equal(f.context.measured, true); + assert.equal(f.context.events, 2); + assert.ok(Math.abs(f.context.value - 300 / 2000) < 1e-12); +}); + +test("stageFactors: an empty store never invents a number — every stage is measured:false / null", () => { + const root = tmp(); + const f = stageFactors(root); + for (const name of ["gate", "cache", "route", "context"]) { + assert.deepEqual(f[name], { measured: false, value: null, events: 0 }, name); + } +}); + +// --- composedReduction: multiplicative over MEASURED stages only ---------------------- + +test("composedReduction: C = Π(1 − f) over measured factors only; missing stages named", () => { + const c = composedReduction({ + gate: { measured: true, value: 0.1, events: 10 }, + cache: { measured: true, value: 0.5, events: 4 }, + route: { measured: false, value: null, events: 0 }, + context: { measured: false, value: null, events: 0 }, + }); + assert.ok(Math.abs(c.measuredReduction - (1 - 0.9 * 0.5)) < 1e-12); + assert.deepEqual(c.stagesIncluded, ["gate", "cache"]); + assert.deepEqual(c.stagesMissing, ["route", "context"]); +}); + +test("composedReduction: nothing measured composes to exactly 0 — never a target restated", () => { + const c = composedReduction(stageFactors(tmp())); + assert.equal(c.measuredReduction, 0); + assert.deepEqual(c.stagesIncluded, []); + assert.deepEqual(c.stagesMissing, ["gate", "cache", "route", "context"]); +}); + +// --- report: totals + a caveat for every unmeasured stage ------------------------------ + +test("report: empty store carries a caveat naming EVERY unmeasured stage plus workload dependence", () => { + const r = report(tmp()); + assert.equal(r.totals.events, 0); + assert.equal(r.totals.savedEstimateTokens, 0); + for (const s of ["gate", "cache", "route", "context"]) { + assert.ok( + r.caveats.some((c) => c.includes(`"${s}"`)), + `caveat names unmeasured stage ${s}`, + ); + } + assert.ok(r.caveats.some((c) => c.includes("workload-dependent"))); +}); + +test("report: totals sum every event and savedEstimate across stages", () => { + const root = tmp(); + seed(root, [ + { stage: "cache", outcome: "hit_exact", savedEstimate: 120 }, + { stage: "cache", outcome: "miss", savedEstimate: 0 }, + { stage: "gate", outcome: "halt" }, + ]); + const r = report(root); + assert.equal(r.totals.events, 3); + assert.equal(r.totals.savedEstimateTokens, 120); + assert.equal(r.composed.stagesIncluded.length, 2); +}); + +// --- renderCostReport: the honesty register ------------------------------------------- + +const assert90OnlyAsTarget = (text) => { + for (let i = text.indexOf("90"); i !== -1; i = text.indexOf("90", i + 1)) { + assert.ok( + /target/i.test(text.slice(Math.max(0, i - 60), i)), + `"90" at index ${i} not preceded by "target": …${text.slice(Math.max(0, i - 60), i + 2)}`, + ); + } +}; + +test('renderCostReport: "90" appears ONLY behind the word "target" — never as achieved', () => { + // Empty store, a mixed store, and a pathological store whose factors would round to 90 %. + assert90OnlyAsTarget(renderCostReport(report(tmp()))); + const mixed = tmp(); + seed(mixed, [ + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "pass" }, + { stage: "cache", outcome: "hit_near", savedEstimate: 50 }, + ]); + assert90OnlyAsTarget(renderCostReport(report(mixed))); +}); + +test("renderCostReport: unmeasured stages print as no-data; the 62% figure is labeled as paper context", () => { + const out = renderCostReport(report(tmp())); + assert.ok(out.includes("gate")); + assert.ok((out.match(/no data/g) || []).length === 4, "all four stages show no data"); + assert.ok(/context \(not a local measurement\).*62%.*paper §9/.test(out)); + assert.ok(out.includes("caveats:")); +}); + +test("renderCostReport: measured factors print as percentages with event counts", () => { + const root = tmp(); + seed(root, [ + { stage: "gate", outcome: "halt" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + { stage: "gate", outcome: "pass" }, + ]); + const out = renderCostReport(report(root)); + assert.ok(out.includes("25.0%"), "gate halt fraction rendered"); + assert.ok(out.includes("composed measured reduction: 25.0%")); +}); + +// --- emit-side helpers + substrate wiring ---------------------------------------------- + +test("recordGate / recordRoute: thin wrappers land stage-tagged lines in metrics.jsonl", () => { + const root = tmp(); + recordGate(root, { halted: true, ref: "task-1" }); + recordGate(root, { halted: false }); + recordRoute(root, { tier: "haiku", tokensIn: 10, tokensOut: 20, ref: "task-2" }); + const gate = readMetrics(root, { stage: "gate" }); + assert.deepEqual( + gate.map((e) => e.outcome), + ["halt", "pass"], + ); + assert.equal(gate[0].ref, "task-1"); + const route = readMetrics(root, { stage: "route" }); + assert.equal(route.length, 1); + assert.equal(route[0].tokensOut, 20); +}); + +test("substrateCheck meters the gate on the explicit path only — ambient hooks stay write-free", () => { + const specified = + "Add a computeVat(rate) function to math.js next to computeTax; must return x*rate; add a unit test"; + const explicit = tmp(); + writeFileSync(join(explicit, "math.js"), "export function computeTax(x){ return x * 0.2 }\n"); + substrateCheck(explicit, specified, { allowBuild: true }); + const gate = readMetrics(explicit, { stage: "gate" }); + assert.equal(gate.length, 1); + assert.equal(gate[0].outcome, "pass"); + + const ambient = tmp(); + writeFileSync(join(ambient, "math.js"), "export function computeTax(x){ return x * 0.2 }\n"); + substrateCheck(ambient, specified, { allowBuild: false }); + assert.ok(!existsSync(join(ambient, ".forge", "metrics.jsonl")), "ambient path never writes"); +}); + +test("substrateCheck meters a halt when the gate asks first", () => { + const root = tmp(); + substrateCheck(root, "Fix it.", { allowBuild: true }); + const gate = readMetrics(root, { stage: "gate" }); + assert.equal(gate.length, 1); + assert.equal(gate[0].outcome, "halt"); +}); From 076838ec4baad0c1d26573c83dc7f1207343587c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 17:13:27 +0000 Subject: [PATCH 2/5] =?UTF-8?q?feat:=20forge=20dash=20=E2=80=94=20local=20?= =?UTF-8?q?dashboard=20over=20the=20ledger,=20metrics,=20and=20blast=20rad?= =?UTF-8?q?ius=20(P7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One self-contained page (src/dash.html: inline CSS/JS, no CDN, no framework, no build step) served by a node:http stdlib server, localhost-only by default, zero runtime deps. src/dash.js separates DATA from SERVING: dashData(root) builds the whole /api/data payload (ledger stats + claims capped at 200 with val/evidence/author, contested claims — val in [0.4,0.6] with >=1 contradiction — author trust, metrics summary + recent-events tail, atlas presence) and degrades corrupt/missing stores to empty sections instead of throwing; serve(root) wires GET /, /api/data, and /api/impact?target=X (404 otherwise, GET-only — the ratify/retract POSTs are a later phase). Panels per docs/plans/substrate-v2/08-dashboard-ux.md: Ledger (val bars, kind filter, contested section, per-author trust), Cost/Cache (stage counters and measured saved-token estimates from .forge/metrics.jsonl), Impact (atlas blast-radius explorer). Every claim row carries its provenance affordance — click-to-copy `forge ledger blame ` — so no number in the UI is unexplained. Taste discipline: one accent (the brand ember, not framework blue), a 4px spacing scale, two radius levels, system sans + one mono. CLI: `forge dash [--port N]` prints the URL and stays alive serving. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Fc2MvWJbQ1cyNQ76ymv4hs --- CHANGELOG.md | 13 ++ src/cli.js | 22 +++ src/dash.html | 339 ++++++++++++++++++++++++++++++++++++++++++++++ src/dash.js | 146 ++++++++++++++++++++ test/dash.test.js | 166 +++++++++++++++++++++++ 5 files changed, 686 insertions(+) create mode 100644 src/dash.html create mode 100644 src/dash.js create mode 100644 test/dash.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 917c5f7..ed580c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,19 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **Local dashboard (P7 of the substrate-v2 plan).** `forge dash [--port N]` serves a + read-only lens on the substrate's state: a `node:http` stdlib server (localhost-only, + zero runtime deps) with ONE self-contained HTML page — inline CSS/JS, no CDN, no + framework, no build step. Panels: Ledger (claims with val bars, kind filter, contested + claims — val ∈ [0.4, 0.6] with ≥1 contradiction — and per-author trust), Cost/Cache + (stage counters + measured saved-token estimates from `.forge/metrics.jsonl`), and + Impact (atlas blast-radius explorer via `/api/impact?target=X`). Every claim row shows + its `forge ledger blame ` command — no unexplained scores anywhere in the UI. Data + is separated from serving (`dashData()` vs `serve()` in `src/dash.js`) so the payload + is tested without sockets, and corrupt/missing stores degrade to empty sections instead + of taking down the lens. The ratify/retract POSTs are a follow-up; this phase never + writes. + - **Proof-carrying reuse cache (P3 of the substrate-v2 plan).** `forge reuse` turns "reuse already-generated code" from prose into a deterministic system: verified code becomes an `artifact` claim keyed by a normalized task fingerprint (volatile literals → diff --git a/src/cli.js b/src/cli.js index 64fc182..28805ba 100755 --- a/src/cli.js +++ b/src/cli.js @@ -29,6 +29,7 @@ const COMMANDS = { anchor: "goal-drift check — are your actual (git) changes still on the stated goal?", lean: "scope-minimality (M5) — measure the diff's footprint vs what the task asked for", uicheck: "deterministic UI check — WCAG contrast (assertable, no guessing)", + dash: "local dashboard over the ledger, metrics, and blast radius", brand: "print the active brand token map", }; @@ -851,6 +852,27 @@ async function run(argv) { console.log(` ADVISE (subjective, human-only): ${ADVISORY_ONLY.slice(0, 4).join(", ")} …`); return; } + if (cmd === "dash") { + const { serve } = await import("./dash.js"); + const i = argv.indexOf("--port"); + const port = i >= 0 ? Number(argv[i + 1]) : 4242; + if (!Number.isInteger(port) || port < 0 || port > 65535) { + console.error("usage: forge dash [--port N]"); + process.exitCode = 1; + return; + } + const server = serve(process.cwd(), { port }); + server.on("listening", () => { + const addr = /** @type {import("node:net").AddressInfo} */ (server.address()); + console.log(`${BRAND.brand} dash — read-only lens on .forge/\n`); + console.log(` http://127.0.0.1:${addr.port} (localhost-only · Ctrl-C to stop)`); + }); + server.on("error", (err) => { + console.error(` ${err.message}`); + process.exitCode = 1; + }); + return; // the process stays alive serving — that's the command + } if (!(cmd in COMMANDS)) { console.error(`Unknown command: ${cmd}\nRun \`${BRAND.cli} --help\` to see commands.`); process.exitCode = 1; diff --git a/src/dash.html b/src/dash.html new file mode 100644 index 0000000..2f187c7 --- /dev/null +++ b/src/dash.html @@ -0,0 +1,339 @@ + + + + + +forge dash + + + +
+ + forge dash + + +
+ +
+
+

Ledger — proof-carrying memory .forge/ledger/

+
+
+ + + + +
+
+

Contested — val ∈ [0.40, 0.60] with ≥1 contradiction

+
+

Author trust — earned from oracle outcomes, floored at 0.50

+
+
+ +
+

Cost / Cache — measured, not asserted .forge/metrics.jsonl

+
+
+

Recent events

+
    +
    + +
    +

    Impact — blast radius .forge/atlas.json

    +
    + + +
    +
    pick a symbol to see what a change would touch
    +
    +
    + +
    read-only lens on .forge/ — every number traces to forge ledger blame <id>. ratify/retract land in a later phase.
    + + + + diff --git a/src/dash.js b/src/dash.js new file mode 100644 index 0000000..c69b399 --- /dev/null +++ b/src/dash.js @@ -0,0 +1,146 @@ +// forge dash — the local dashboard (docs/plans/substrate-v2/08-dashboard-ux.md, P7): +// a read-only lens on the .forge/ stores. DATA (dashData → one JSON payload) is +// separated from SERVING (serve → a node:http stdlib server) so the payload is +// testable without sockets. One self-contained HTML page (src/dash.html — inline +// CSS/JS, no CDN, no framework, no build step), localhost-only by default, zero +// runtime deps. Read-mostly by design: the two writes the spec names (ratify / +// retract POSTs) are a follow-up — this phase never writes anything. +import { readFileSync } from "node:fs"; +import { createServer } from "node:http"; +import { basename, dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { impact, load as loadAtlas } from "./atlas.js"; +import { authorTrust, claimText, val, validOutcome } from "./ledger.js"; +import { loadClaims, repoLedger, stats } from "./ledger_store.js"; +import { read as readMetrics, summarize } from "./metrics.js"; +import { epochDay } from "./util.js"; + +/** A claim is contested when its val sits in this band AND it carries ≥1 oracle + * contradiction — genuinely disputed, not merely fresh (a fresh claim is 0.5 with + * no evidence at all). Spec: 08-dashboard-ux.md §1, Ledger panel. */ +export const CONTESTED_BAND = [0.4, 0.6]; + +/** Payload caps — the dashboard is a lens, not an export format. */ +const CLAIM_CAP = 200; +const RECENT_CAP = 20; + +const emptyLedger = () => ({ + stats: { total: 0, tombstoned: 0, byKind: {}, val: { dormant: 0, uncertain: 0, trusted: 0 } }, + claims: [], + contested: [], + trust: {}, +}); + +/** The row shape every ledger table in the UI renders — id8 is the handle the + * provenance affordance (`forge ledger blame `) is built from. */ +const claimRow = (c, nowDay) => ({ + id8: c.id.slice(0, 8), + kind: c.kind, + val: Number(val(c, nowDay).toFixed(3)), + evidenceCount: (c.evidence ?? []).filter(validOutcome).length, + author: c.provenance?.author ?? "", + tombstoned: Boolean(c.tombstone), + text: claimText(c).slice(0, 140), +}); + +function ledgerSection(root, nowDay) { + const dir = repoLedger(root); + const all = loadClaims(dir); + const claims = all + .map((c) => claimRow(c, nowDay)) + .sort((a, b) => b.val - a.val || (a.id8 < b.id8 ? -1 : 1)) + .slice(0, CLAIM_CAP); + const contested = all + .filter((c) => { + if (c.tombstone) return false; + const v = val(c, nowDay); + if (v < CONTESTED_BAND[0] || v > CONTESTED_BAND[1]) return false; + return (c.evidence ?? []).some((e) => validOutcome(e) && e.result === "contradict"); + }) + .map((c) => claimRow(c, nowDay)); + return { stats: stats(dir, nowDay), claims, contested, trust: authorTrust(all) }; +} + +function metricsSection(root) { + const recent = readMetrics(root) + .slice(-RECENT_CAP) + .map((e) => ({ + t: e.t ?? 0, + stage: e.stage ?? "", + outcome: e.outcome ?? "", + savedEstimate: Number.isFinite(e.savedEstimate) ? e.savedEstimate : 0, + })); + return { stages: summarize(root), recent }; +} + +/** + * One JSON payload for the whole page — everything /api/data serves. Corrupt or + * missing stores degrade to empty sections; this NEVER throws (a broken .forge/ + * must not take down the lens that would let you see it's broken). + * @param {string} root + * @param {{nowDay?: number}} [opts] + */ +export function dashData(root, { nowDay = epochDay() } = {}) { + let ledger = emptyLedger(); + try { + ledger = ledgerSection(root, nowDay); + } catch {} + let metrics = { stages: {}, recent: [] }; + try { + metrics = metricsSection(root); + } catch {} + let atlas = { built: false, symbols: 0, files: 0 }; + try { + const a = loadAtlas(root); + if (a) atlas = { built: true, symbols: a.symbols?.length ?? 0, files: a.files ?? 0 }; + } catch {} + return { repo: basename(root), nowDay, ledger, metrics, atlas }; +} + +const HTML_PATH = join(dirname(fileURLToPath(import.meta.url)), "dash.html"); + +const sendJson = (res, code, body) => { + res.writeHead(code, { + "content-type": "application/json; charset=utf-8", + "cache-control": "no-store", + }); + res.end(JSON.stringify(body)); +}; + +/** + * The dashboard server: GET / → the page, GET /api/data → dashData, GET + * /api/impact?target=X → blast radius (when an atlas exists). Everything else 404. + * Localhost-only by default — pass a host explicitly to expose it, on your own head. + * @param {string} root + * @param {{port?: number, host?: string}} [opts] + * @returns {import("node:http").Server} + */ +export function serve(root, { port = 4242, host = "127.0.0.1" } = {}) { + const html = readFileSync(HTML_PATH, "utf8"); // read once at startup, self-contained + const server = createServer((req, res) => { + const url = new URL(req.url ?? "/", "http://localhost"); + if (req.method !== "GET") + return sendJson(res, 404, { error: "read-only in this phase — GET only" }); + if (url.pathname === "/") { + res.writeHead(200, { + "content-type": "text/html; charset=utf-8", + "cache-control": "no-store", + }); + return res.end(html); + } + if (url.pathname === "/api/data") return sendJson(res, 200, dashData(root)); + if (url.pathname === "/api/impact") { + const target = url.searchParams.get("target"); + if (!target) return sendJson(res, 400, { error: "usage: /api/impact?target=" }); + let atlas = null; + try { + atlas = loadAtlas(root); + } catch {} + if (!atlas) return sendJson(res, 404, { error: "no atlas — run `forge atlas build` first" }); + return sendJson(res, 200, impact(atlas, target)); + } + return sendJson(res, 404, { error: "not found" }); + }); + server.listen(port, host); + return server; +} diff --git a/test/dash.test.js b/test/dash.test.js new file mode 100644 index 0000000..8b4e8b8 --- /dev/null +++ b/test/dash.test.js @@ -0,0 +1,166 @@ +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { dashData, serve } from "../src/dash.js"; +import { mintClaim, outcomeRecord } from "../src/ledger.js"; +import { appendEvidence, putClaim, repoLedger, tombstone } from "../src/ledger_store.js"; +import { record } from "../src/metrics.js"; + +const tmp = () => mkdtempSync(join(tmpdir(), "forge-dash-")); +const NOW = 100; + +const mint = (dir, name, text, author = "alice") => { + const c = mintClaim({ kind: "fact", body: { name, text }, provenance: { author }, t: NOW }).claim; + putClaim(dir, c); + return c; +}; +const ev = (dir, id, result, ref) => + appendEvidence(dir, id, outcomeRecord({ oracle: "test.run", result, ref, t: NOW }).outcome); + +/** A fixture repo: 3 claims (one contested, one tombstoned) + 2 metrics lines. */ +function fixture() { + const root = tmp(); + const dir = repoLedger(root); + const trusted = mint(dir, "style", "tabs are actually spaces here"); + ev(dir, trusted.id, "confirm", "run-1"); + const contested = mint(dir, "port", "the dev server listens on 4242"); + ev(dir, contested.id, "confirm", "run-2"); // 1 confirm + 1 contradict → val = 0.5 + ev(dir, contested.id, "contradict", "run-3"); + const retracted = mint(dir, "old", "we deploy from the ops box"); + tombstone(dir, retracted.id, { author: "alice", reason: "obsolete", t: NOW }); + record(root, { stage: "cache", outcome: "exact", savedEstimate: 1200 }); + record(root, { stage: "gate", outcome: "pass" }); + return { root, trusted, contested, retracted }; +} + +test("dashData: one payload with ledger, metrics, and atlas sections in shape", () => { + const { root, contested, retracted } = fixture(); + const d = dashData(root, { nowDay: NOW }); + + assert.equal(d.repo, root.split("/").pop()); + assert.equal(d.nowDay, NOW); + assert.equal(d.ledger.stats.total, 3); + assert.equal(d.ledger.stats.tombstoned, 1); + assert.equal(d.ledger.stats.byKind.fact, 3); + + assert.equal(d.ledger.claims.length, 3); + for (const c of d.ledger.claims) { + assert.equal(c.id8.length, 8, "id8 is the blame handle"); + assert.equal(c.kind, "fact"); + assert.equal(typeof c.val, "number"); + assert.equal(typeof c.evidenceCount, "number"); + assert.equal(c.author, "alice"); + assert.equal(typeof c.tombstoned, "boolean"); + assert.equal(typeof c.text, "string"); + assert.ok(c.text.length > 0, "text comes from claimText, never empty for a fact"); + } + const tomb = d.ledger.claims.find((c) => c.id8 === retracted.id.slice(0, 8)); + assert.equal(tomb.tombstoned, true); + + // Contested = val ∈ [0.4, 0.6] AND ≥1 contradiction — exactly the mixed-evidence claim. + assert.deepEqual( + d.ledger.contested.map((c) => c.id8), + [contested.id.slice(0, 8)], + ); + assert.equal(d.ledger.contested[0].val, 0.5); + + assert.equal(typeof d.ledger.trust.alice, "number", "authorTrust map keyed by author"); + + assert.equal(d.metrics.stages.cache.savedEstimate, 1200); + assert.equal(d.metrics.stages.gate.byOutcome.pass, 1); + assert.equal(d.metrics.recent.length, 2); + assert.equal(d.metrics.recent[0].stage, "cache"); + + assert.deepEqual(d.atlas, { built: false, symbols: 0, files: 0 }); +}); + +test("dashData: atlas section reports a built atlas", () => { + const { root } = fixture(); + mkdirSync(join(root, ".forge"), { recursive: true }); + writeFileSync( + join(root, ".forge", "atlas.json"), + JSON.stringify({ version: 2, files: 3, symbols: [{ name: "a" }, { name: "b" }] }), + ); + assert.deepEqual(dashData(root, { nowDay: NOW }).atlas, { built: true, symbols: 2, files: 3 }); +}); + +test("dashData: corrupt/missing stores degrade to empty sections, never throw", () => { + const empty = dashData(tmp(), { nowDay: NOW }); + assert.equal(empty.ledger.stats.total, 0); + assert.deepEqual(empty.ledger.claims, []); + assert.deepEqual(empty.metrics, { stages: {}, recent: [] }); + + const root = tmp(); + mkdirSync(join(root, ".forge", "ledger"), { recursive: true }); + writeFileSync(join(root, ".forge", "ledger", "claims"), "not a directory"); // readdir throws + writeFileSync(join(root, ".forge", "metrics.jsonl"), "{nope\n{{also nope\n"); + writeFileSync(join(root, ".forge", "atlas.json"), "{corrupt"); + const d = dashData(root, { nowDay: NOW }); + assert.deepEqual(d.ledger.claims, [], "broken ledger → empty section"); + assert.deepEqual(d.metrics.recent, [], "corrupt metrics lines are skipped"); + assert.deepEqual(d.atlas, { built: false, symbols: 0, files: 0 }); +}); + +test("dashData: claims list is capped at 200 while stats keep the true total", () => { + const root = tmp(); + const dir = repoLedger(root); + for (let i = 0; i < 205; i++) mint(dir, `n${i}`, `fact number ${i}`); + const d = dashData(root, { nowDay: NOW }); + assert.equal(d.ledger.claims.length, 200); + assert.equal(d.ledger.stats.total, 205); +}); + +test("serve: / is the page, /api/data is the payload, unknown routes 404, GET-only", async () => { + const { root } = fixture(); + const server = serve(root, { port: 0 }); // ephemeral port, localhost-only default + await new Promise((resolve) => server.on("listening", resolve)); + const addr = /** @type {import("node:net").AddressInfo} */ (server.address()); + const base = `http://127.0.0.1:${addr.port}`; + try { + const page = await fetch(`${base}/`); + assert.equal(page.status, 200); + assert.match(page.headers.get("content-type"), /text\/html/); + const html = await page.text(); + assert.match(html, /forge/); + assert.doesNotMatch(html, /https?:\/\/(?!localhost)/, "self-contained — no CDN, no remote"); + + const data = await fetch(`${base}/api/data`); + assert.equal(data.status, 200); + const d = await data.json(); + assert.equal(d.ledger.stats.total, 3); + + assert.equal((await fetch(`${base}/api/impact`)).status, 400, "missing target"); + assert.equal((await fetch(`${base}/api/impact?target=x`)).status, 404, "no atlas yet"); + assert.equal((await fetch(`${base}/nope`)).status, 404); + assert.equal( + (await fetch(`${base}/api/data`, { method: "POST" })).status, + 404, + "no writes in this phase", + ); + } finally { + server.close(); + } +}); + +test("serve: /api/impact traces blast radius once an atlas exists", async () => { + const root = tmp(); + mkdirSync(join(root, "src"), { recursive: true }); + writeFileSync(join(root, "src", "a.js"), "export function core() { return 1; }\n"); + writeFileSync(join(root, "src", "b.js"), 'import { core } from "./a.js";\ncore();\n'); + const { build } = await import("../src/atlas.js"); + build({ root }); + const server = serve(root, { port: 0 }); + await new Promise((resolve) => server.on("listening", resolve)); + const addr = /** @type {import("node:net").AddressInfo} */ (server.address()); + try { + const res = await fetch(`http://127.0.0.1:${addr.port}/api/impact?target=core`); + assert.equal(res.status, 200); + const r = await res.json(); + assert.equal(r.found, true); + assert.ok(r.impactedFiles.includes("src/b.js"), "dependent file is in the radius"); + } finally { + server.close(); + } +}); From 5c6f69ac6d1566786f43bb30ec63058cad0d4c9b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 17:16:11 +0000 Subject: [PATCH 3/5] =?UTF-8?q?feat:=20close=20the=20loop=20(P5)=20?= =?UTF-8?q?=E2=80=94=20doom-loop=20diagnosis,=20imagination,=20CUSUM=20dri?= =?UTF-8?q?ft,=20checkpoint=20cadence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four loop-closure faculties from docs/plans/substrate-v2/06-faculties-and-mechanisms.md: - forge diagnose (src/diagnose.js): failure signatures (sha256 of the normalized error + file + symbol, volatile parts stripped) counted in a 50-entry ring (.forge/trace/failures.jsonl, corrupt-tolerant); the 3rd identical hit mints a content-addressed `diagnosis` claim into the team ledger and directs the agent to stop retrying and escalate one model tier with the diagnosis as the prompt's head (spec §5). - forge imagine (src/imagine.js): the static half of the consequence simulator (Eq. 4) — entities → blast radius → predicted breaks with confidence, plus the minimal dry-run suite via weighted greedy set cover (weight = file size; ln-n approximation) and riskScore = Σ confidence. The sandboxed worktree runner is the documented P5 follow-up (spec §2). - anchor.cusum(): one-sided CUSUM control chart (k=0.35, h=1.0) — sustained small drift alarms, a single within-tolerance spike drains back to zero (spec §5, M4). - verify.checkpointCadence(): the optimal-stopping threshold rule n* = ⌈checkCost / (pErr·tokensPerStep·costPerToken)⌉ clamped to [1, 50] — every input measured or priced (spec §6, M6). Tests: 319 (31 new), typecheck clean, biome clean on touched files. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Fc2MvWJbQ1cyNQ76ymv4hs --- CHANGELOG.md | 17 ++++ src/anchor.js | 27 ++++++ src/cli.js | 56 +++++++++++++ src/diagnose.js | Bin 0 -> 7135 bytes src/imagine.js | 145 ++++++++++++++++++++++++++++++++ src/verify.js | 22 +++++ test/anchor.test.js | 42 +++++++++- test/diagnose.test.js | 186 ++++++++++++++++++++++++++++++++++++++++++ test/imagine.test.js | 127 ++++++++++++++++++++++++++++ test/verify.test.js | 39 +++++++++ 10 files changed, 660 insertions(+), 1 deletion(-) create mode 100644 src/diagnose.js create mode 100644 src/imagine.js create mode 100644 test/diagnose.test.js create mode 100644 test/imagine.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 917c5f7..9b19d34 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,23 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **Loop closure (P5 of the substrate-v2 plan): doom-loop diagnosis, imagination, CUSUM + drift, checkpoint cadence.** `forge diagnose ""` hashes each failure into a + signature (line numbers, addresses, timestamps, and absolute paths normalized out) and + counts recurrences in a 50-entry ring; the 3rd identical hit is thrash — it mints a + content-addressed `diagnosis` claim into the team ledger and tells the agent to STOP + retrying and escalate ONE model tier with the diagnosis as the prompt's head (the same + loop becomes a one-per-team event, not one-per-session). `forge imagine ""` is the + static half of the consequence simulator (paper Eq. 4): entities → blast radius → + predicted breaks with confidence, plus the minimal dry-run test suite via weighted greedy + set cover (weight = file size as a duration proxy; classic ln-n approximation) and + `riskScore = Σ confidence` — the sandboxed worktree runner that executes the suite is the + P5 follow-up. `anchor.cusum()` adds the M4 one-sided CUSUM control chart (k = 0.35, + h = 1.0): sustained small drift alarms, a single exploratory spike drains back to zero. + `verify.checkpointCadence()` prices M6's "when to check?" as the optimal-stopping + threshold rule `n* = ⌈checkCost / (pErr·tokensPerStep·costPerToken)⌉`, clamped to + [1, 50] — every input measured or priced, no magic constants. + - **Proof-carrying reuse cache (P3 of the substrate-v2 plan).** `forge reuse` turns "reuse already-generated code" from prose into a deterministic system: verified code becomes an `artifact` claim keyed by a normalized task fingerprint (volatile literals → diff --git a/src/anchor.js b/src/anchor.js index 94e784b..10bd8c6 100644 --- a/src/anchor.js +++ b/src/anchor.js @@ -165,6 +165,33 @@ export function goalDrift(root, goal, opts = {}) { }; } +/** + * M4 — one-sided CUSUM control chart over a drift-signal series (spec §5: + * docs/plans/substrate-v2/06-faculties-and-mechanisms.md). A raw threshold on a + * single checkpoint's drift Dₜ is noisy — one exploratory step legitimately wanders. + * CUSUM accumulates only the excess over the allowance k (Cₜ = max(0, Cₜ₋₁ + Dₜ − k)) + * and alarms at Cₜ > h, which detects SUSTAINED small drift with provably minimal + * detection delay for a given false-alarm rate (classical sequential analysis), + * while a single within-tolerance spike drains back to zero instead of alarming. + * Defaults k = 0.35, h = 1.0 per the spec (calibration lands in P8). Pure. + * @param {number[]} signals drift per checkpoint, Dₜ ∈ [0, 1] (non-numeric → 0) + * @param {{k?: number, h?: number}} [opts] + * @returns {{alarm: boolean, C: number[], firstAlarm: number}} firstAlarm = index of + * the first checkpoint whose statistic crossed h, or -1 if none did. + */ +export function cusum(signals, { k = 0.35, h = 1.0 } = {}) { + const C = []; + let c = 0; + let firstAlarm = -1; + for (let i = 0; i < signals.length; i++) { + const d = Number(signals[i]); + c = Math.max(0, c + (Number.isFinite(d) ? d : 0) - k); + C.push(c); + if (firstAlarm < 0 && c > h) firstAlarm = i; + } + return { alarm: firstAlarm >= 0, C, firstAlarm }; +} + export function renderAnchor(r) { const lines = ["Forge anchor — goal-drift check", ""]; if (!r.changed.length) diff --git a/src/cli.js b/src/cli.js index 64fc182..1275aad 100755 --- a/src/cli.js +++ b/src/cli.js @@ -27,6 +27,9 @@ const COMMANDS = { substrate: "one pre-action gate: assumptions, route, impact, scope, memory, verify", scope: "decompose files into independent clusters (+ coupled files you didn't name)", anchor: "goal-drift check — are your actual (git) changes still on the stated goal?", + diagnose: + "doom-loop check — record a failure; 3× the same signature mints a diagnosis + escalation", + imagine: "consequence simulation — predicted breaks + the minimal dry-run test suite for a task", lean: "scope-minimality (M5) — measure the diff's footprint vs what the task asked for", uicheck: "deterministic UI check — WCAG contrast (assertable, no guessing)", brand: "print the active brand token map", @@ -779,6 +782,59 @@ async function run(argv) { console.log(json ? JSON.stringify(r, null, 2) : renderAnchor(r)); return; // advisory — never fails the process } + if (cmd === "diagnose") { + const { diagnose, THRASH_K } = await import("./diagnose.js"); + const json = argv.includes("--json"); + const flagVal = (name) => { + const i = argv.indexOf(name); + return i >= 0 ? argv[i + 1] : undefined; + }; + const args = argv.filter( + (a, i) => !a.startsWith("--") && argv[i - 1] !== "--file" && argv[i - 1] !== "--symbol", + ); + const errorText = args.slice(1).join(" "); + if (!errorText) { + console.error('usage: forge diagnose "" [--file f] [--symbol s] [--json]'); + process.exitCode = 1; + return; + } + const r = diagnose(process.cwd(), { + errorText, + file: flagVal("--file"), + symbol: flagVal("--symbol"), + }); + if (json) return console.log(JSON.stringify(r, null, 2)); + console.log(`${BRAND.brand} diagnose — doom-loop check\n`); + console.log( + ` signature: ${r.signature.slice(0, 12)} · seen ${r.count}× in the recent failure window`, + ); + if (r.thrash) { + if (r.claimId) + console.log( + ` diagnosis claim: ${r.claimId.slice(0, 12)} (\`forge ledger show ${r.claimId.slice(0, 8)}\`)`, + ); + console.log(`\n ${r.escalate ?? r.reason}`); + } else { + console.log(` below the thrash threshold (${THRASH_K}) — recorded; keep going.`); + } + return; // advisory — halting the retry loop is the AGENT's move, not an exit code + } + if (cmd === "imagine") { + const { imagineTask, renderImagine } = await import("./imagine.js"); + const json = argv.includes("--json"); + const task = argv + .slice(1) + .filter((a) => a !== "--json") + .join(" "); + if (!task) { + console.error('usage: forge imagine "" [--json]'); + process.exitCode = 1; + return; + } + const r = imagineTask(process.cwd(), task); + console.log(json ? JSON.stringify(r, null, 2) : renderImagine(r)); + return; + } if (cmd === "lean") { const { leanRepo, renderLean } = await import("./lean.js"); const json = argv.includes("--json"); diff --git a/src/diagnose.js b/src/diagnose.js new file mode 100644 index 0000000000000000000000000000000000000000..b774a45abbd8ef702886dedab070275963124a75 GIT binary patch literal 7135 zcmb_h>uwvz74C06#R*{~ON+adoXAaRE2`iqv3jusOGSZ*f;rqBl56gT*qIf@Ff973 z4^R~CBlJDmH_4OqJ7@MninfCSHDGC}ojG&nT)y+2!+xJ8c`?-##cG=6M$^xK`8!2< zp883iFQ~|K>xZhui)uw|$cysO^cRWBOy87aV+&=qfA^$+(C^dHQ-7kuGO@8XzRDs$ z)!|HKu}MvkM)dE0opb{_r$R3@c11K%aZ(nVt}efOb$bD)V#s@p;H$s*;%2?tVdaHX;I|ULK`!{A+}iYrg2$hSd`B-e&rb{;@h!U7Pn9f zUEtDTnd;0MQijT`&|!Y33pGwOEn_>&OG}wvQk=nHsj_jN1$2$wVV>C{PiUrbi`cNx z>dK)?2G9trllVt1N6B@_yUP!=l37ZZgqDk6>+5R?eCGcE%>76wVRqf`6`YvXEprhWL&mn}FDw}>-? zPZ8yl=B+_Mu*(kObdeX9)}$6tA^H|xxLRdl4=TmR8uy&eqquO71q9WLXhV|%hj+3( z(u0ZVJZr_h%W=P*_X}lb+bh@=FZiB4y1IDr=HmJ-g1z1m^xbWPcRNxSWxWLnlmA`} zvbX0gTvq4@J2@<2iX*Z zzl&sn2RI{5_Cv(O=@d5`9FN}nU!ANLR7Z?-y=GM27izB#{o`Sy4G`q_b; z`KAeMBwH5c%Kti{o2LjVL`@sYOReXIIM9oIRo`5``tI%3wOSzDJ!vAh&}jAZMugp_T+U;NG3CGYenj3OX;u4i#2R_{t7P z>W21efQf6ahaBT_0Ny`$GIMe+Qh?SM;SQiG!qwh&Y5rd}2 zM;oWwSd}jDti~o!N`L@IGoU|J;SBqI_HsOlLl(FwG~308*jmH$@XTS5RcPPBknA*6G)3W4P% z`_u$1?R62s>p*Qd8Fit(APm<;GyqT+aY&=V^C(81n6F2;wK25J%LH)fpfweNzkxG= zP;1maAn?Zd;~I+Lv?kgoWfpRsXzy{s)yxy;@!~G65gMpcC{N<(Y`XOluZRNVJr_E! zKP-k>e~MTcW}WWCchUNbjX#XmPpY3c*K{i%2U|bAv%xSJMu*)qJpJ&~&7ohdIg>@* zGZFtB7vDc%j3vjtC!6=n8;(D=_2~ZQ$p1?DlXL&u+rxS-2i*QatYiOXxD5X6-yU`k zhkcJf@Uxuo=(y7*)Pw#kPj&ykZ;G(5kTmr3U;jq@@X#>hn{)pM<^Om)7!Dy@Boo99 zhuz2T7yV)|ydQo{CQNV$QT(fWR^4XsJi}dBGu}Bkelj>|RN{AgYH<=0B=n~-Tq>VL z#NI-8Smn2&d7Pe^+JaHZ2INDZo_uw3931~THPoli38-9xlOgmfdmu_v51_IUwB?Lo z=2BCy7VDY(nVEWW^3-c`50TGN3-tI0O_;OrgIT5HoOAhm5ENI!`5jAU>fl}z>Vhzw zI?){_*%ec;MOiGsH<66cL=fP5wa}6vOl>SMB8r*U0LEBsolGD;pa|P<8E?VTQUDFp zKk$JH7ML61(=R40)!Dq}JDd#wTb$LXDRRr^51S3$EScurzBkJ&b6(ZFtxe?~=!Tf? zlsZ8B$~yc3Rrr+Fn;jt7s@&e_(YkTOL-5%A>X9hSzn#dTZRrH@CdTiD43dO71Vfee&F&)n`Lfdu%p!@cC)n%ss*dHY zflIG0AA%mG)O;YpR4c)Ee#XJUqBJw_w^y%U1t=ayd%U#^hw=Lj!KmR3p@5J4zfvCi zwzKwJIIDzsoxEV|l0lB&;!y62I0s$FfHKy05}z5!-R>Ybv3Pf)5DhKYsack1XGk_q zD3Gh9L?pp=BRc}=n<%t++)~j}C5ay|y7R>BE>%{U}# zY|e}Xd;wI$6F4YJV2uKRfeciE_|7nUV7y?$8-cfgi{J?~GEg~LIV#d( z8!LjXbG>>mATW31vl=XDs&tD7ABP7JCqRWGRe*|#NOOKPv|cxwwDQD7^%kaKDym;@ z5CrdG5-y{wG^zOHx@8}|55NpmL)?D@;|5v2^tx0bjtl=rvo^y%0%8q*Z6tORZJi%Z zKTLDBv*+gMHZVyHMsw7oRKSg=mY}?#H|itJ|6{X0cT6wO5;2N|9=KI zYR8IQl!b|A@8@V=(GnnkAd%@GNFO)A<1$VFdfdB7NK4?foxzwZ2A>&`hfrGPNY5U$ zQS&ZhMewER*aC1$>#+bpjK~3wdx%CAVw?f@Ii&?at4Qyn3#<_ceI=~?v2H*N(lnqR zHgNfnyyb=&1+yqM9LO?vdsuHD9(#leBADZ9T0B{KYqIcRU8P*e)J}Cl2J#o z8^aSKjm+_>gTCeyz6;s>0pL3Xn|jkLPLbkVprOMs#|5+HUIxiO!}LvmG(^ zW*ZLQNPR;$SA!vN4=sxufz(|lPeR0cGB@eDfM6uppNSy-Jp|Zl0Ak~Ciw;4;v`!hZo>MjA5+hCmvxTwy9oX%rk8a)A!G z+5^fS3#~>WU)Wjyq#E*|vqkTgCwDwOarNNp{0|pZ%4|(TI2o7~MJ37^mvPq~)tFaz zGye}PWUDpINfhTgC;wWQMb z1pcHj_-dl3hVz~)$1T>2$jj&h7NRpn5#hhFj1$wJC%WmS)SuG z&zlKSL-HU~CcpzgLBinOc%ZU80XS+fV>d#TQ}O2up2&G?yd;p=3<%z7WLKeN9;t|&=lG6ZQaG^z^5V!vHlK|(AMOrKyfDF^dUDJg50=w40vpbE+ zb%((X9g2I4CJVDwg&`X+k!dlT3bR<&3X!8^tT<>Vj7&4!Udy_0p3~VG)w4*i%P(CN z2!nTS3G%s{ZZs97_AJ-Eooh%m^s2lNeK|oo+&)$=KC2$Mg~2PrB3!7A1dLY^2KA5? zRlJ_Dy5_YElv7zr=*If zcZ?!`X=2aW+!;sSR9G{3cz|12kNF3(7O2bf-=?BAQVC6FT@zc)j-PZl@;$?&wH&zl z&p)fs2XqR0?td=>u%*N1u{G(yfWOAIg#<4%CXyvasxqA>R_LDJ+#tPxvo zBPR|=GF7dMG2Sej1;Y_|PXR_o3S5NMk+62M(GEf2o*ZRwl*RL+{ z6-J~JOc7CR_^Y7@YNGsX9mZD>=28ti`8y=!Qo69NmD%D5GV_%0QXWX;KHUvA-|<+q R{X(Vp-fvV!l2}ZX{tHLkVKD#z literal 0 HcmV?d00001 diff --git a/src/imagine.js b/src/imagine.js new file mode 100644 index 0000000..ac38b0d --- /dev/null +++ b/src/imagine.js @@ -0,0 +1,145 @@ +// forge imagine — the consequence simulator ĉ = g(a, C) (paper Eq. 4; +// docs/plans/substrate-v2/06-faculties-and-mechanisms.md §2). The atlas gives the +// exact-structure half: entities → blast radius → predicted breaks with confidence. +// This module adds impacted-test SELECTION — the minimal dry-run suite that makes +// pre-action simulation affordable at all (minutes → seconds vs "run everything"). +// The sandboxed worktree runner that EXECUTES the suite against a proposed diff is +// the P5 follow-up (spec §2.2) — selection had to exist first, and it is useful on +// its own as "run these, in this order, before you touch anything". +import { statSync } from "node:fs"; +import { join } from "node:path"; +import { build as buildAtlas, impact, load as loadAtlas } from "./atlas.js"; +import { referencedEntities } from "./preflight.js"; +import { isTestFile, predictFailingTests } from "./substrate.js"; + +/** + * Weighted greedy set cover over the covers(test, source) relation: candidates are + * substrate's predicted-failing tests plus any test files among the impacted set + * (those cover themselves — they are predicted to break, so the suite must include + * them). Weight = file size, a duration proxy until measured runtimes exist (P8). + * Each round picks the test minimizing weight / newly-covered — the classical + * greedy H(n) ≈ ln n approximation to weighted set cover (Chvátal 1979), and exact + * on the tiny instances a single edit produces. Impacted sources with no covering + * test simply can't constrain the suite; the gaps surface via selectTestsReport. + * @param {string} root + * @param {string[]} impactedFiles + * @returns {string[]} minimal ordered dry-run suite (best value first) + */ +export function selectTests(root, impactedFiles) { + return selectTestsReport(root, impactedFiles).tests; +} + +/** selectTests plus the sources no known test covers (the honest gap in the cover). */ +export function selectTestsReport(root, impactedFiles) { + const files = [...new Set(impactedFiles.map(String))]; + const covers = new Map(); // test → Set of covered elements (sources + itself) + const addCover = (test, el) => { + if (!covers.has(test)) covers.set(test, new Set()); + covers.get(test).add(el); + }; + for (const f of files) { + if (isTestFile(f)) addCover(f, f); + else for (const t of predictFailingTests(root, [f])) addCover(t, f); + } + const universe = new Set(); + for (const els of covers.values()) for (const el of els) universe.add(el); + const weights = new Map( + [...covers.keys()].map((t) => { + // Unreadable candidate → weight 1, never a throw: selection must not die on a + // just-deleted test file; running it later fails honestly instead. + let w = 1; + try { + w = Math.max(1, statSync(join(root, t)).size); + } catch {} + return [t, w]; + }), + ); + const tests = []; + const covered = new Set(); + while (covered.size < universe.size) { + let best = null; + let bestRatio = Infinity; + for (const [t, els] of covers) { + let gain = 0; + for (const el of els) if (!covered.has(el)) gain++; + if (!gain) continue; // also skips already-chosen tests (their gain is 0) + const ratio = weights.get(t) / gain; + // Deterministic tie-break by path so the suite order is stable across runs. + if (ratio < bestRatio || (ratio === bestRatio && best !== null && t < best)) { + best = t; + bestRatio = ratio; + } + } + if (best === null) break; + tests.push(best); + for (const el of covers.get(best)) covered.add(el); + } + const uncovered = files.filter((f) => !isTestFile(f) && !covered.has(f)).sort(); + return { tests, uncovered }; +} + +/** + * Imagine the consequences of a task before acting: entities → impact() blast + * radius → predicted breaks (per-file max confidence across targets), the minimal + * dry-run suite, and riskScore = Σ confidence — the number spec §2.3 thresholds to + * decide whether the (follow-up) sandboxed dry-run is worth paying for. + * @param {string} root + * @param {string} task + * @param {{atlas?: object, threshold?: number}} [opts] inject `atlas` to skip the build. + */ +export function imagineTask(root, task, { atlas, threshold = 0.1 } = {}) { + const graph = atlas || loadAtlas(root) || buildAtlas({ root }); + const entities = referencedEntities(task); + const targets = [...new Set([...entities.symbols, ...entities.files])].slice(0, 8); + const reports = targets.map((t) => impact(graph, t, { threshold })); + const byFile = new Map(); + for (const r of reports) { + for (const x of r.impacted) { + const f = x.node?.file; + if (f) byFile.set(f, Math.max(byFile.get(f) ?? 0, x.confidence)); + } + } + const predictedBreaks = [...byFile] + .map(([file, confidence]) => ({ file, confidence })) + .sort((a, b) => b.confidence - a.confidence || (a.file < b.file ? -1 : 1)); + const { tests, uncovered } = selectTestsReport( + root, + predictedBreaks.map((b) => b.file), + ); + return { + task: String(task), + targets, + found: reports.some((r) => r.found), + predictedBreaks, + tests, + uncovered, + riskScore: Number(predictedBreaks.reduce((s, b) => s + b.confidence, 0).toFixed(4)), + }; +} + +export function renderImagine(r) { + const lines = ["Forge imagine — consequence simulation (pre-action)", ""]; + lines.push(` targets: ${r.targets.length ? r.targets.join(", ") : "(none named)"}`); + if (!r.found) { + lines.push("", " nothing in the code graph matches this task — no consequences to predict."); + return lines.join("\n"); + } + lines.push(` risk score: ${r.riskScore} (Σ confidence over predicted breaks)`); + lines.push("", ` predicted breaks (${r.predictedBreaks.length}):`); + for (const b of r.predictedBreaks.slice(0, 12)) + lines.push(` ${b.confidence.toFixed(2)} ${b.file}`); + if (r.predictedBreaks.length > 12) lines.push(` … ${r.predictedBreaks.length - 12} more`); + if (r.tests.length) { + lines.push("", ` minimal dry-run suite (${r.tests.length}) — run these, in this order:`); + for (const t of r.tests) lines.push(` - ${t}`); + } else { + lines.push("", " no covering tests found for the predicted breaks."); + } + if (r.uncovered.length) + lines.push( + "", + ` ! no test covers: ${r.uncovered.slice(0, 6).join(", ")}${r.uncovered.length > 6 ? " …" : ""}`, + ); + lines.push("", " (sandboxed worktree dry-run of this suite lands as the P5 follow-up)"); + return lines.join("\n"); +} diff --git a/src/verify.js b/src/verify.js index 90258ed..d49969a 100644 --- a/src/verify.js +++ b/src/verify.js @@ -57,6 +57,28 @@ function runTests(cwd) { } } +/** + * M6 — checkpoint cadence as an optimal-stopping threshold rule (spec §6: + * docs/plans/substrate-v2/06-faculties-and-mechanisms.md). Insert a checkpoint once + * the expected loss of continuing-while-wrong exceeds the check's price: + * pErr·tokensPerStep·costPerToken·n > checkCost, i.e. check every + * n* = ⌈checkCost / (pErr · tokensPerStep · costPerToken)⌉ meaningful steps. No + * magic constants: pErr is measured per tier from ledger outcome history, the costs + * are priced — riskier/cheaper tiers get smaller n* automatically. Clamped to + * [1, 50]: even a near-free check shouldn't fire more than every step, and even a + * near-riskless run must still checkpoint eventually. Pure. + * @param {{pErr: number, tokensPerStep: number, costPerToken?: number, checkCost: number}} f + * pErr = per-step error hazard; tokensPerStep = tokens put at risk per step; + * checkCost priced in the same token-cost unit. + * @returns {number} integer steps between checkpoints, in [1, 50] + */ +export function checkpointCadence({ pErr, tokensPerStep, costPerToken = 1, checkCost }) { + const n = Math.ceil(checkCost / (pErr * tokensPerStep * costPerToken)); + // Degenerate inputs (NaN from bad measurements) fail SAFE: check every step. + if (Number.isNaN(n)) return 1; + return Math.min(50, Math.max(1, n)); // zero risk → Infinity → the 50-step ceiling +} + export function verify({ targetRoot = process.cwd(), base = "HEAD" } = {}) { const diff = git(["diff", "--unified=0", base], targetRoot) || diff --git a/test/anchor.test.js b/test/anchor.test.js index a83b5c6..be50eea 100644 --- a/test/anchor.test.js +++ b/test/anchor.test.js @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import test from "node:test"; -import { goalDrift, renderAnchor } from "../src/anchor.js"; +import { cusum, goalDrift, renderAnchor } from "../src/anchor.js"; // changed[] is injected so these are pure (no git needed). test("goalDrift flags a changed file unrelated to the goal", () => { @@ -76,3 +76,43 @@ test("goalDrift (llm on): a throwing runner falls back to the deterministic spli assert.ok(r.offGoal.includes("src/reports.js")); assert.equal(r.drift, true); }); + +// --------------------------------------------------------------------------- +// M4 — one-sided CUSUM drift control (pure). +// --------------------------------------------------------------------------- + +test("cusum alarms on sustained small drift (the decaying-anchor failure)", () => { + // Each checkpoint drifts 0.6, only 0.25 over the allowance — no single step is + // alarming, but the excess accumulates: C = 0.25, 0.5, 0.75, 1.0, 1.25 → alarm. + const r = cusum([0.6, 0.6, 0.6, 0.6, 0.6, 0.6]); + assert.equal(r.alarm, true); + assert.equal(r.firstAlarm, 4, "alarms at the first checkpoint where C exceeds h"); + assert.equal(r.C.length, 6); + assert.ok(Math.abs(r.C[0] - 0.25) < 1e-9); +}); + +test("cusum does not alarm on a single spike within tolerance", () => { + const r = cusum([0.1, 1.2, 0.1, 0.1, 0.1, 0.1]); + assert.equal(r.alarm, false); + assert.equal(r.firstAlarm, -1); + assert.equal(r.C.at(-1), 0, "the statistic drains back to zero after the spike"); +}); + +test("cusum is quiet on on-goal signals and on an empty series", () => { + assert.equal(cusum([0.1, 0.2, 0.3, 0.1]).alarm, false); + assert.deepEqual(cusum([]), { alarm: false, C: [], firstAlarm: -1 }); +}); + +test("cusum honors custom k and h", () => { + // With a zero allowance every step accumulates fully; h=0.5 trips on step 2. + const r = cusum([0.3, 0.3, 0.3], { k: 0, h: 0.5 }); + assert.equal(r.firstAlarm, 1); + // A generous h absorbs the same series entirely. + assert.equal(cusum([0.3, 0.3, 0.3], { k: 0, h: 2 }).alarm, false); +}); + +test("cusum treats non-numeric signals as zero drift (never NaN-poisons the chart)", () => { + const r = cusum([0.6, Number.NaN, 0.6]); + assert.ok(r.C.every((c) => Number.isFinite(c))); + assert.equal(r.alarm, false); +}); diff --git a/test/diagnose.test.js b/test/diagnose.test.js new file mode 100644 index 0000000..f673c4e --- /dev/null +++ b/test/diagnose.test.js @@ -0,0 +1,186 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { appendFileSync, mkdirSync, mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { + diagnose, + failureSignature, + failuresPath, + normalizeError, + RING_SIZE, + readFailures, + recordFailure, + THRASH_K, +} from "../src/diagnose.js"; +import { loadClaims, repoLedger } from "../src/ledger_store.js"; + +const fixture = () => mkdtempSync(join(tmpdir(), "forge-diagnose-")); + +// --------------------------------------------------------------------------- +// failureSignature — pure, and stable under everything that varies between two +// runs of the SAME broken code (the whole point: recurrences must accumulate). +// --------------------------------------------------------------------------- + +test("failureSignature is stable under line/col numbers, hex addresses, timestamps, absolute paths", () => { + const where = { file: "src/auth.js", symbol: "verifyToken" }; + const a = failureSignature( + "TypeError: x is undefined\n at verifyToken (/home/alice/repo/src/auth.js:12:5) 0xdeadbeef 2026-07-07T10:00:00Z", + where, + ); + const b = failureSignature( + "TypeError: x is undefined\n at verifyToken (/Users/bob/work/src/auth.js:99:21) 0x1234abcd 2026-07-08T23:59:59Z", + where, + ); + assert.equal(a, b, "volatile parts must not change the signature"); +}); + +test("failureSignature distinguishes error class, file, and symbol", () => { + const base = failureSignature("TypeError: x is undefined", { file: "a.js", symbol: "f" }); + assert.notEqual(base, failureSignature("RangeError: y overflow", { file: "a.js", symbol: "f" })); + assert.notEqual( + base, + failureSignature("TypeError: x is undefined", { file: "b.js", symbol: "f" }), + ); + assert.notEqual( + base, + failureSignature("TypeError: x is undefined", { file: "a.js", symbol: "g" }), + ); +}); + +test("normalizeError keeps the basename (signal) while dropping the machine prefix", () => { + const n = normalizeError("Error at /home/ci/build/src/auth.js:3:1 after 120ms"); + assert.match(n, /auth\.js/); + assert.doesNotMatch(n, /home|ci|build|120|:3/); +}); + +// --------------------------------------------------------------------------- +// recordFailure — append-only trace with a corrupt-tolerant reader. +// --------------------------------------------------------------------------- + +test("recordFailure counts recurrences of the same signature", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", t: 1 }; + assert.equal(recordFailure(root, f).count, 1); + assert.equal(recordFailure(root, f).count, 2); + const other = recordFailure(root, { ...f, errorText: "RangeError: other" }); + assert.equal(other.count, 1, "a different signature counts separately"); + assert.equal(recordFailure(root, f).count, 3); +}); + +test("recordFailure tolerates corrupt trace lines (killed-process append)", () => { + const root = fixture(); + recordFailure(root, { errorText: "boom", t: 1 }); + appendFileSync(failuresPath(root), '{"truncated: \n'); + const r = recordFailure(root, { errorText: "boom", t: 2 }); + assert.equal(r.count, 2, "the corrupt line is skipped, not fatal"); + assert.equal(readFailures(root).length, 2); +}); + +test("recordFailure only counts within the last RING_SIZE entries", () => { + const root = fixture(); + const f = { errorText: "old failure", file: "src/a.js" }; + recordFailure(root, { ...f, t: 1 }); + recordFailure(root, { ...f, t: 2 }); + for (let i = 0; i < RING_SIZE; i++) recordFailure(root, { errorText: `noise ${i}`, t: 3 + i }); + const r = recordFailure(root, { ...f, t: 999 }); + assert.equal(r.count, 1, "hits pushed out of the ring no longer count toward thrash"); +}); + +test("recordFailure never persists a secret-shaped error head", () => { + const root = fixture(); + recordFailure(root, { errorText: `auth failed: api_key="hunter2-super-secret"`, t: 1 }); + const [e] = readFailures(root); + assert.match(e.head, /redacted/); + assert.doesNotMatch(JSON.stringify(e), /hunter2/); +}); + +// --------------------------------------------------------------------------- +// diagnose — thrash at k=3 mints ONE content-addressed diagnosis claim. +// --------------------------------------------------------------------------- + +test("diagnose stays quiet below the thrash threshold", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", nowDay: 10 }; + for (let i = 1; i < THRASH_K; i++) { + const r = diagnose(root, { ...f, t: i }); + assert.equal(r.thrash, false); + assert.equal(r.count, i); + assert.equal(r.escalate, undefined); + } + assert.equal(loadClaims(repoLedger(root)).length, 0, "no claim minted before thrash"); +}); + +test("diagnose at the 3rd recurrence mints a diagnosis claim and says STOP + escalate one tier", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", nowDay: 10 }; + let r; + for (let i = 1; i <= THRASH_K; i++) r = diagnose(root, { ...f, t: i }); + assert.equal(r.thrash, true); + assert.equal(r.count, THRASH_K); + assert.match(r.escalate, /STOP retrying/i); + assert.match(r.escalate, /ONE model tier/i); + assert.match(r.escalate, /diagnosis as the head/i); + const claims = loadClaims(repoLedger(root)); + assert.equal(claims.length, 1); + assert.equal(claims[0].kind, "diagnosis"); + assert.equal(claims[0].id, r.claimId); + assert.equal(claims[0].body.signature, r.signature); + assert.deepEqual(claims[0].body.triedFixes, []); + assert.equal(claims[0].provenance.agent, "doomloop"); +}); + +test("diagnose is idempotent — further recurrences resolve to the SAME claim", () => { + const root = fixture(); + const f = { errorText: "TypeError: boom", file: "src/a.js", symbol: "f", nowDay: 10 }; + let third; + for (let i = 1; i <= THRASH_K; i++) third = diagnose(root, { ...f, t: i }); + const fourth = diagnose(root, { ...f, t: THRASH_K + 1 }); + assert.equal(fourth.thrash, true); + assert.equal(fourth.claimId, third.claimId, "content addressing dedupes the mint"); + assert.equal(loadClaims(repoLedger(root)).length, 1, "no duplicate diagnosis claims"); +}); + +test("diagnose prefers the caller's root-cause note over the error head", () => { + const root = fixture(); + const f = { + errorText: "boom", + note: "circular import between auth and session", + nowDay: 1, + }; + let r; + for (let i = 1; i <= THRASH_K; i++) r = diagnose(root, { ...f, t: i }); + const [claim] = loadClaims(repoLedger(root)); + assert.equal(claim.body.note, f.note); + assert.equal(claim.id, r.claimId); +}); + +// --------------------------------------------------------------------------- +// CLI — forge diagnose "" [--file f] [--symbol s] [--json] +// --------------------------------------------------------------------------- + +const CLI = fileURLToPath(new URL("../src/cli.js", import.meta.url)); +const runCli = (args, cwd) => spawnSync("node", [CLI, ...args], { cwd, encoding: "utf8" }); + +test("forge diagnose: records, then escalates on the 3rd identical failure", () => { + const cwd = fixture(); + mkdirSync(join(cwd, ".forge"), { recursive: true }); + const args = ["diagnose", "TypeError: boom", "--file", "src/a.js", "--symbol", "f"]; + let out; + for (let i = 0; i < THRASH_K; i++) out = runCli(args, cwd); + assert.equal(out.status, 0, "advisory — never fails the process"); + assert.match(out.stdout, /doom-loop/); + assert.match(out.stdout, /STOP retrying/i); + const j = JSON.parse(runCli([...args, "--json"], cwd).stdout); + assert.equal(j.thrash, true); + assert.ok(j.claimId); +}); + +test("forge diagnose: no error text prints usage and exits 1", () => { + const cwd = fixture(); + const r = runCli(["diagnose"], cwd); + assert.equal(r.status, 1); + assert.match(r.stderr, /usage: forge diagnose/); +}); diff --git a/test/imagine.test.js b/test/imagine.test.js new file mode 100644 index 0000000..b091061 --- /dev/null +++ b/test/imagine.test.js @@ -0,0 +1,127 @@ +import assert from "node:assert/strict"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { test } from "node:test"; +import { build } from "../src/atlas.js"; +import { imagineTask, renderImagine, selectTests, selectTestsReport } from "../src/imagine.js"; + +const fixture = () => mkdtempSync(join(tmpdir(), "forge-imagine-")); + +const put = (root, rel, text) => { + mkdirSync(dirname(join(root, rel)), { recursive: true }); + writeFileSync(join(root, rel), text); +}; + +// --------------------------------------------------------------------------- +// selectTests — weighted greedy set cover (weight = file size). +// --------------------------------------------------------------------------- + +test("selectTests covers every impacted source via its cheapest covering test", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/b.js", "export function b() {}\n"); + // src/a.js has TWO covering candidates; the smaller one must win the cover. + put(root, "src/a.test.js", "// tiny\n"); + put(root, "test/a.js", `// huge duration proxy\n${"x".repeat(4000)}\n`); + put(root, "src/b.test.js", "// b test\n"); + const tests = selectTests(root, ["src/a.js", "src/b.js"]); + assert.ok(tests.includes("src/a.test.js"), "cheapest cover for src/a.js chosen"); + assert.ok(!tests.includes("test/a.js"), "redundant heavier test dropped"); + assert.ok(tests.includes("src/b.test.js")); + assert.equal(tests.length, 2, "minimal: one test per otherwise-uncovered source"); +}); + +test("selectTests orders the suite best-value-first (greedy pick order)", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/b.js", "export function b() {}\n"); + put(root, "src/a.test.js", "// tiny a test\n"); + put(root, "src/b.test.js", `// expensive b test\n${"y".repeat(4000)}\n`); + const tests = selectTests(root, ["src/a.js", "src/b.js"]); + assert.deepEqual(tests, ["src/a.test.js", "src/b.test.js"], "cheapest per covered file first"); +}); + +test("selectTests includes an impacted test file (it covers itself)", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/a.test.js", "// a test\n"); + put(root, "test/util.js", "// an impacted test file\n"); + const tests = selectTests(root, ["src/a.js", "test/util.js"]); + assert.ok(tests.includes("test/util.js"), "predicted-to-break test is part of the suite"); + assert.ok(tests.includes("src/a.test.js")); +}); + +test("selectTestsReport names the sources no known test covers, without blocking", () => { + const root = fixture(); + put(root, "src/a.js", "export function a() {}\n"); + put(root, "src/a.test.js", "// a test\n"); + put(root, "src/orphan.js", "export function orphan() {}\n"); + const { tests, uncovered } = selectTestsReport(root, ["src/a.js", "src/orphan.js"]); + assert.deepEqual(tests, ["src/a.test.js"]); + assert.deepEqual(uncovered, ["src/orphan.js"], "the gap is surfaced, not silently dropped"); +}); + +test("selectTests on nothing impacted is an empty suite", () => { + assert.deepEqual(selectTests(fixture(), []), []); +}); + +// --------------------------------------------------------------------------- +// imagineTask — entities → impact → predicted breaks + dry-run suite + risk. +// --------------------------------------------------------------------------- + +function appFixture() { + const root = fixture(); + put(root, "src/util.js", "export function helper(x) {\n return x * 2;\n}\n"); + put( + root, + "src/app.js", + 'import { helper } from "./util.js";\nexport function runApp() {\n return helper(1);\n}\n', + ); + put(root, "src/app.test.js", 'import { runApp } from "./app.js";\nrunApp();\n'); + return { root, atlas: build({ root }) }; +} + +test("imagineTask predicts breaks with confidence and selects the minimal dry-run suite", () => { + const { root, atlas } = appFixture(); + const r = imagineTask(root, "update `helper` in src/util.js", { atlas }); + assert.equal(r.found, true); + assert.ok(r.targets.includes("helper")); + const files = r.predictedBreaks.map((b) => b.file); + assert.ok(files.includes("src/app.js"), "the caller of helper is a predicted break"); + for (const b of r.predictedBreaks) { + assert.ok(b.confidence > 0 && b.confidence <= 1, `confidence in (0,1]: ${b.confidence}`); + } + assert.ok(r.tests.includes("src/app.test.js"), "the impacted caller's test is in the suite"); + const sum = r.predictedBreaks.reduce((s, b) => s + b.confidence, 0); + assert.ok(Math.abs(r.riskScore - sum) < 1e-9, "riskScore = Σ confidence"); + assert.ok(r.riskScore > 0); +}); + +test("imagineTask predictedBreaks are sorted by confidence, descending", () => { + const { root, atlas } = appFixture(); + const r = imagineTask(root, "update `helper` in src/util.js", { atlas }); + for (let i = 1; i < r.predictedBreaks.length; i++) { + assert.ok(r.predictedBreaks[i - 1].confidence >= r.predictedBreaks[i].confidence); + } +}); + +test("imagineTask on a task naming nothing in the graph predicts nothing", () => { + const { root, atlas } = appFixture(); + const r = imagineTask(root, "polish `totallyUnknownThing` please", { atlas }); + assert.equal(r.found, false); + assert.deepEqual(r.predictedBreaks, []); + assert.deepEqual(r.tests, []); + assert.equal(r.riskScore, 0); + assert.match(renderImagine(r), /nothing in the code graph/); +}); + +test("renderImagine prints breaks, the suite, and the sandbox follow-up note", () => { + const { root, atlas } = appFixture(); + const out = renderImagine(imagineTask(root, "update `helper` in src/util.js", { atlas })); + assert.match(out, /predicted breaks/); + assert.match(out, /minimal dry-run suite/); + assert.match(out, /src\/app\.test\.js/); + assert.match(out, /risk score/); + assert.match(out, /P5 follow-up/); +}); diff --git a/test/verify.test.js b/test/verify.test.js index d12e32f..38a2e2b 100644 --- a/test/verify.test.js +++ b/test/verify.test.js @@ -37,3 +37,42 @@ test("shared extractor: atlas and verify use the same call-site extraction (no d assert.ok(!syms.includes("JSON"), "builtins ignored"); assert.ok(CALL_IGNORE.has("console")); }); + +// --------------------------------------------------------------------------- +// M6 — checkpoint cadence (optimal-stopping threshold rule, pure). +// --------------------------------------------------------------------------- + +test("checkpointCadence computes n* = ceil(checkCost / (pErr·tokensPerStep·costPerToken))", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + // risk per step = 0.05 · 200 · 1 = 10 → n* = 100/10 = 10 + assert.equal(checkpointCadence({ pErr: 0.05, tokensPerStep: 200, checkCost: 100 }), 10); + // non-integer ratio rounds UP — checking a step late is worse than a step early + assert.equal(checkpointCadence({ pErr: 0.05, tokensPerStep: 200, checkCost: 105 }), 11); + // costPerToken scales the at-risk side + assert.equal( + checkpointCadence({ pErr: 0.05, tokensPerStep: 200, costPerToken: 2, checkCost: 100 }), + 5, + ); +}); + +test("checkpointCadence: riskier (cheaper) tiers checkpoint more often", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + const haiku = checkpointCadence({ pErr: 0.2, tokensPerStep: 500, checkCost: 400 }); + const opus = checkpointCadence({ pErr: 0.01, tokensPerStep: 500, checkCost: 400 }); + assert.ok(haiku < opus, `higher hazard → smaller n* (${haiku} < ${opus})`); +}); + +test("checkpointCadence clamps to [1, 50]", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + // near-free check → never below every-step + assert.equal(checkpointCadence({ pErr: 0.5, tokensPerStep: 1000, checkCost: 0 }), 1); + // near-riskless run (or pErr measured at 0) → still checkpoints by the ceiling + assert.equal(checkpointCadence({ pErr: 0, tokensPerStep: 1000, checkCost: 100 }), 50); + assert.equal(checkpointCadence({ pErr: 1e-9, tokensPerStep: 1, checkCost: 100 }), 50); +}); + +test("checkpointCadence fails safe on degenerate inputs (check every step)", async () => { + const { checkpointCadence } = await import("../src/verify.js"); + assert.equal(checkpointCadence({ pErr: Number.NaN, tokensPerStep: 100, checkCost: 100 }), 1); + assert.equal(checkpointCadence({ pErr: 0, tokensPerStep: 100, checkCost: 0 }), 1); +}); From 659f40d6cf7b2c619401c481438398eec8a67916 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 17:19:55 +0000 Subject: [PATCH 4/5] =?UTF-8?q?feat:=20generated-UI=20quality=20gate=20?= =?UTF-8?q?=E2=80=94=20design=20fingerprints,=20slop=20distance,=20project?= =?UTF-8?q?=20conformance=20(P6)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Taste becomes measurable (guard-over-prose applied to design). New src/uifingerprint.js extracts a deterministic design fingerprint from CSS/JSX/Tailwind classes — pure static parsing, zero LLM calls, zero screenshots: HSL palette + 12-bin hue histogram, spacing with an approximate-GCD base unit (residual minimization over 2/4/8) and on-scale fraction, font families, radius and shadow levels. Two distances gate generated UI: slopDistance to a shipped, rationale-documented generic-template signature set (default-Tailwind blue/indigo, stock Bootstrap, the AI-landing gradient) must stay high; conformance to the project's own fingerprint — stored as a shared "fingerprint" ledger claim via mintProjectFingerprint — must stay low. uiGate emits actionable per-feature violations, never a bare score. Scale-conformance checks (spacing-on-base within ε, radius/shadow level caps, palette bound) join uicheck's ASSERTABLE_CHECKS; the executable versions live in uifingerprint.scaleChecks under the same ids (drift-tested). forge uicheck gains `fingerprint [--mint] [--json]` and `design [--json]` (exit 1 on fail); the existing contrast surface is unchanged and also reachable as `uicheck contrast `. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Fc2MvWJbQ1cyNQ76ymv4hs --- CHANGELOG.md | 14 + src/cli.js | 83 ++++- src/uicheck.js | 12 + src/uifingerprint.js | 682 +++++++++++++++++++++++++++++++++++++ test/uifingerprint.test.js | 245 +++++++++++++ 5 files changed, 1034 insertions(+), 2 deletions(-) create mode 100644 src/uifingerprint.js create mode 100644 test/uifingerprint.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 917c5f7..d5b943c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,20 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **Generated-UI quality gate (P6 of the substrate-v2 plan).** Taste becomes measurable: + `src/uifingerprint.js` extracts a deterministic design fingerprint from CSS/JSX/Tailwind + classes — pure static parsing, no LLM, no screenshots — covering palette (HSL + 12-bin hue + histogram), spacing (base unit by residual-minimization approximate GCD, on-scale + fraction), font families, radius and shadow levels. Two distances gate generated UI: + `slopDistance` to a shipped, rationale-documented generic-template signature set + (default-Tailwind blue/indigo, stock Bootstrap, the AI-landing gradient) must stay HIGH, + and `conformance` to the project's own fingerprint — stored as a shared `fingerprint` + ledger claim via `mintProjectFingerprint` — must stay LOW; `uiGate` failures are + actionable per-feature edits, never a bare score. Scale-conformance checks + (spacing-on-base, radius/shadow level caps, palette bound) join `ASSERTABLE_CHECKS`. + `forge uicheck` gains `fingerprint [--mint]` and `design ` (exit 1 on + fail) alongside the unchanged contrast math. + - **Proof-carrying reuse cache (P3 of the substrate-v2 plan).** `forge reuse` turns "reuse already-generated code" from prose into a deterministic system: verified code becomes an `artifact` claim keyed by a normalized task fingerprint (volatile literals → diff --git a/src/cli.js b/src/cli.js index 64fc182..1d6bcc6 100755 --- a/src/cli.js +++ b/src/cli.js @@ -28,7 +28,8 @@ const COMMANDS = { scope: "decompose files into independent clusters (+ coupled files you didn't name)", anchor: "goal-drift check — are your actual (git) changes still on the stated goal?", lean: "scope-minimality (M5) — measure the diff's footprint vs what the task asked for", - uicheck: "deterministic UI check — WCAG contrast (assertable, no guessing)", + uicheck: + "deterministic UI checks — contrast · fingerprint · design ", brand: "print the active brand token map", }; @@ -830,10 +831,88 @@ async function run(argv) { return; } if (cmd === "uicheck") { + const sub = argv[1]; + if (sub === "fingerprint" || sub === "design") { + const ui = await import("./uifingerprint.js"); + const json = argv.includes("--json"); + const files = argv.slice(2).filter((a) => !a.startsWith("--")); + if (!files.length) { + console.error( + `usage: ${BRAND.cli} uicheck ${sub} [--json]${sub === "fingerprint" ? " [--mint]" : ""}`, + ); + process.exitCode = 1; + return; + } + const fp = ui.fingerprintFiles(process.cwd(), files); + if (sub === "fingerprint") { + let minted = null; + if (argv.includes("--mint")) { + const { epochDay } = await import("./util.js"); + minted = ui.mintProjectFingerprint(process.cwd(), files, { t: epochDay() }); + } + if (json) { + console.log(JSON.stringify(minted ? { fingerprint: fp, minted } : fp, null, 2)); + } else { + console.log(`${BRAND.brand} uicheck fingerprint — the design feature vector\n`); + console.log( + ` palette: ${fp.paletteSize} color(s), hue bins [${fp.hueBuckets.join(" ")}]`, + ); + console.log( + ` spacing: ${fp.spacing.join(", ") || "(none)"} px — base ${fp.spacingBase ?? "(none)"}, ${Math.round(fp.spacingOnScale * 100)}% on-scale`, + ); + console.log(` type: ${fp.fontFamilies.join(", ") || "(none)"}`); + console.log( + ` shape: radii ${fp.radii.join(", ") || "(none)"} (${fp.radiusLevels} level(s)) · ${fp.shadowLevels} shadow level(s)`, + ); + if (minted) { + if (minted.ok) + console.log( + `\n minted fingerprint claim ${minted.id.slice(0, 12)}${minted.existed ? " (already in ledger)" : ""} — the gate's "home"`, + ); + else console.error(`\n mint failed: ${"reason" in minted ? minted.reason : ""}`); + } + } + if (minted && !minted.ok) process.exitCode = 1; + return; + } + // design — the two-sided gate: fail when too close to generic OR (when the + // project has minted its fingerprint) too far from the project's own system. + const projectFp = ui.loadProjectFingerprint(process.cwd()); + const gate = ui.uiGate(fp, { projectFp }); + const checks = ui.scaleChecks(fp); + const fail = !gate.pass || checks.some((c) => !c.pass); + if (json) { + console.log( + JSON.stringify({ ...gate, checks, hasProjectFingerprint: !!projectFp }, null, 2), + ); + } else { + const { tauSlop, tauConform } = ui.UI_GATE_DEFAULTS; + console.log(`${BRAND.brand} uicheck design — slop distance + project conformance\n`); + console.log( + ` slop distance: ${gate.slop} (need ≥ ${tauSlop} — farther from generic is better)`, + ); + console.log( + projectFp + ? ` conformance: ${gate.conform} (need ≤ ${tauConform} — closer to the project system is better)` + : ` conformance: (no project fingerprint claim — slop-only; mint one: \`${BRAND.cli} uicheck fingerprint --mint\`)`, + ); + for (const v of gate.violations) console.log(`\n ✗ ${v.detail}\n fix: ${v.hint}`); + console.log(""); + for (const c of checks) + console.log( + ` ${c.pass ? "✓" : "✗"} ${c.id}: ${c.detail}${c.pass || !c.hint ? "" : `\n fix: ${c.hint}`}`, + ); + console.log(`\n ${fail ? "✗ FAIL" : "✓ PASS"}`); + } + if (fail) process.exitCode = 1; + return; + } const { contrastRatio, wcagLevel, ASSERTABLE_CHECKS, ADVISORY_ONLY } = await import( "./uicheck.js" ); - const [fg, bg] = [argv[1], argv[2]]; + // `uicheck contrast ` is the named form; bare `uicheck ` stays + // supported (it predates the subcommands and hooks already call it). + const [fg, bg] = sub === "contrast" ? [argv[2], argv[3]] : [argv[1], argv[2]]; console.log(`${BRAND.brand} uicheck — deterministic UI review\n`); if (fg && bg) { try { diff --git a/src/uicheck.js b/src/uicheck.js index 246712b..e73de25 100644 --- a/src/uicheck.js +++ b/src/uicheck.js @@ -72,6 +72,18 @@ export const ASSERTABLE_CHECKS = [ id: "reduced-motion", how: "animations ≥200ms are wrapped in @media (prefers-reduced-motion)", }, + // Scale-conformance checks (P6): executable versions live in uifingerprint.js + // scaleChecks() under the SAME ids — a test pins the two lists together. + { + id: "spacing-scale", + how: "≥90% of spacing values are multiples of the declared base within ε=0.5px", + }, + { + id: "radius-levels", + how: "≤3 distinct border-radius levels (a design system uses few, deliberately)", + }, + { id: "shadow-levels", how: "≤3 distinct box-shadow levels (deliberate elevation steps)" }, + { id: "palette-size", how: "≤8 distinct colors after HSL normalization" }, ]; export const ADVISORY_ONLY = [ diff --git a/src/uifingerprint.js b/src/uifingerprint.js new file mode 100644 index 0000000..e14068c --- /dev/null +++ b/src/uifingerprint.js @@ -0,0 +1,682 @@ +// forge uifingerprint — the generated-UI quality gate (P6, +// docs/plans/substrate-v2/07-ui-quality-gate.md). AI-generated UI converges on the +// max-likelihood template — the same statistical failure as M5 over-engineering, here +// favoring the *median* design. The taste layer (global/taste/*.md) is prose, and +// prose loses to gradients; this module makes taste MEASURABLE: a deterministic +// design fingerprint (pure static CSS/Tailwind-class parsing — no LLM, no +// screenshots, same discipline as uicheck's WCAG math) and two distances over it: +// slop(v) — too CLOSE to a shipped generic-template signature → fail +// conform(v) — too FAR from the project's own stored design system → fail +// Good output is far from generic and close to home; both are geometry once UI is a +// feature vector. The subjective residue (beauty) stays with the human reviewer — +// the gate's job is to stop the template from ever reaching them. +import { readFileSync } from "node:fs"; +import { isAbsolute, join } from "node:path"; +import { mintClaim } from "./ledger.js"; +import { loadClaims, putClaim, reindex, repoLedger } from "./ledger_store.js"; +import { gitAuthor } from "./util.js"; + +// --------------------------------------------------------------------------- +// Color parsing — everything normalizes to integer {h,s,l} so hue geometry (the +// "looks like the default framework palette" signal) is comparable across syntaxes. +// --------------------------------------------------------------------------- + +/** sRGB (0..255) → {h,s,l} with h in 0..359 degrees, s/l in 0..100 percent. */ +export function rgbToHsl(r, g, b) { + const rn = r / 255; + const gn = g / 255; + const bn = b / 255; + const max = Math.max(rn, gn, bn); + const min = Math.min(rn, gn, bn); + const d = max - min; + const l = (max + min) / 2; + let h = 0; + let s = 0; + if (d > 0) { + s = d / (1 - Math.abs(2 * l - 1)); + if (max === rn) h = ((gn - bn) / d + 6) % 6; + else if (max === gn) h = (bn - rn) / d + 2; + else h = (rn - gn) / d + 4; + h *= 60; + } + return { h: Math.round(h) % 360, s: Math.round(s * 100), l: Math.round(l * 100) }; +} + +// Representative [hue, saturation] per Tailwind color family (the 500 shade — the one +// every starter reaches for). Lightness comes from the shade number. Neutral families +// get s≤5 by fiat so an all-slate UI doesn't register as "chromatic blue": their tint +// is a background choice, not an accent. +const TW_FAMILY_HS = { + red: [0, 84], + orange: [25, 95], + amber: [38, 92], + yellow: [45, 93], + lime: [84, 81], + green: [142, 71], + emerald: [160, 84], + teal: [173, 80], + cyan: [189, 94], + sky: [199, 89], + blue: [217, 91], + indigo: [239, 84], + violet: [258, 90], + purple: [271, 91], + fuchsia: [292, 84], + pink: [330, 81], + rose: [350, 89], + slate: [215, 5], + gray: [220, 5], + zinc: [240, 4], + neutral: [0, 0], + stone: [25, 5], +}; + +const HEX_RE = /#([0-9a-f]{8}|[0-9a-f]{6}|[0-9a-f]{4}|[0-9a-f]{3})\b/gi; +const RGB_RE = /rgba?\(\s*(\d{1,3})[,\s]+(\d{1,3})[,\s]+(\d{1,3})/gi; +const HSL_RE = /hsla?\(\s*([\d.]+)(?:deg)?[,\s]+([\d.]+)%[,\s]+([\d.]+)%/gi; +const TW_COLOR_RE = new RegExp( + `\\b(?:bg|text|border|from|via|to|ring|outline|fill|stroke|accent|caret|decoration|divide|shadow)-(${Object.keys(TW_FAMILY_HS).join("|")})-(50|100|200|300|400|500|600|700|800|900|950)\\b`, + "g", +); +const TW_BW_RE = /\b(?:bg|text|border|from|via|to|ring|fill|stroke)-(white|black)\b/g; + +function parseColors(text) { + /** @type {{h:number,s:number,l:number}[]} */ + const out = []; + for (const [, hex] of text.matchAll(HEX_RE)) { + // 3/4-digit shorthand expands per CSS; a trailing alpha channel is ignored — the + // fingerprint cares about hue identity, not opacity. + const full = + hex.length <= 4 ? [...hex.slice(0, 3)].map((c) => c + c).join("") : hex.slice(0, 6); + out.push( + rgbToHsl( + parseInt(full.slice(0, 2), 16), + parseInt(full.slice(2, 4), 16), + parseInt(full.slice(4, 6), 16), + ), + ); + } + for (const [, r, g, b] of text.matchAll(RGB_RE)) out.push(rgbToHsl(+r, +g, +b)); + for (const [, h, s, l] of text.matchAll(HSL_RE)) + out.push({ h: Math.round(+h) % 360, s: Math.round(+s), l: Math.round(+l) }); + for (const [, family, shade] of text.matchAll(TW_COLOR_RE)) { + const [h, s] = TW_FAMILY_HS[family]; + // Lightness from the shade number: 50→95, 500→50, 950→5 — coarse but monotone, + // and hue (what the slop signatures key on) is exact. + out.push({ h, s, l: Math.min(96, Math.max(4, Math.round(100 - +shade / 10))) }); + } + for (const [, bw] of text.matchAll(TW_BW_RE)) + out.push({ h: 0, s: 0, l: bw === "white" ? 100 : 0 }); + return out; +} + +// --------------------------------------------------------------------------- +// Length parsing — spacing, radii. Everything lands in px (rem/em at the 16px root) +// so the base-unit inference sees one scale. +// --------------------------------------------------------------------------- + +const LEN_RE = /(-?\d*\.?\d+)(px|rem|em)\b/g; + +/** Absolute px lengths inside one CSS value string (zero and non-lengths skipped). */ +function parseLengths(value) { + const out = []; + for (const [, n, unit] of String(value).matchAll(LEN_RE)) { + const px = Math.abs(+n) * (unit === "px" ? 1 : 16); + if (px > 0) out.push(Math.round(px * 100) / 100); + } + return out; +} + +const cssValues = (text, propRe) => [...text.matchAll(propRe)].map((m) => m[1]); + +// The leading class keeps `scroll-padding`, `--m-4` etc. from matching. +const SPACING_PROP_RE = /(?:^|[;{\s"'])(?:margin|padding)(?:-[a-z-]+)?\s*:\s*([^;}]+)/gi; +const GAP_PROP_RE = /(?:^|[;{\s"'])(?:row-gap|column-gap|gap)\s*:\s*([^;}]+)/gi; +const RADIUS_PROP_RE = + /(?:^|[;{\s"'])border(?:-(?:top|bottom|start|end)-(?:left|right)?)?-radius\s*:\s*([^;}]+)/gi; +const SHADOW_PROP_RE = /(?:^|[;{\s"'])box-shadow\s*:\s*([^;}]+)/gi; +const FONT_PROP_RE = /(?:^|[;{\s"'])font-family\s*:\s*([^;}]+)/gi; + +// Tailwind spacing utilities: p-4/mx-2/gap-6/space-y-8 → n×4 px (p-px → 1px). The +// lookbehind stops `top-4` matching as `p-4`. +const TW_SPACE_RE = + /(? Math.abs(s - Math.round(s / u) * u); + +/** + * Infer the spacing base unit: residual(u) = Σ distance-to-nearest-multiple-of-u. + * Residuals are monotone in divisibility (every multiple of 8 is one of 4 and 2), so + * a bare argmin always degenerates to the smallest candidate — instead take the + * LARGEST base that explains the data within half a pixel per value, falling back to + * the true argmin when nothing fits cleanly. + * @param {number[]} values @param {number[]} [candidates] + * @returns {number|null} null when there are no values to infer from + */ +export function inferSpacingBase(values, candidates = [2, 4, 8]) { + if (!values.length) return null; + const residual = (u) => values.reduce((sum, s) => sum + offBase(s, u), 0); + let best = null; + for (const u of [...candidates].sort((a, b) => a - b)) + if (residual(u) <= 0.5 * values.length) best = u; + if (best !== null) return best; + return candidates.reduce((arg, u) => (residual(u) < residual(arg) ? u : arg), candidates[0]); +} + +/** Fraction of values sitting on multiples of `base` within ε (vacuously 1). */ +export function onScaleFraction(values, base, epsilon = 0.5) { + if (!values.length || !base) return 1; + return values.filter((s) => offBase(s, base) <= epsilon).length / values.length; +} + +// --------------------------------------------------------------------------- +// The fingerprint. +// --------------------------------------------------------------------------- + +/** + * @typedef {{h:number,s:number,l:number}} Hsl + * @typedef {{palette:Hsl[], paletteSize:number, hueBuckets:number[], spacing:number[], + * spacingBase:number|null, spacingOnScale:number, fontFamilies:string[], + * radii:number[], radiusLevels:number, shadowLevels:number}} Fingerprint + */ + +// Below this saturation a color is a neutral: it carries no hue identity, so it must +// not vote in the hue histogram (an ink-and-paper UI is not "red" at h=0). +const NEUTRAL_S = 10; + +const sortNum = (a, b) => a - b; +const uniqSorted = (arr) => [...new Set(arr)].sort(sortNum); + +/** + * Extract the design fingerprint from raw CSS / JSX / Tailwind-class text. Pure and + * deterministic — the same text always yields the same vector (it becomes a + * content-addressed ledger claim, so this is a protocol requirement, not a nicety). + * @param {string} text + * @returns {Fingerprint} + */ +export function fingerprintText(text) { + const t = String(text); + + const seen = new Set(); + /** @type {Hsl[]} */ + const palette = []; + for (const c of parseColors(t)) { + const key = `${c.h},${c.s},${c.l}`; + if (!seen.has(key)) { + seen.add(key); + palette.push(c); + } + } + palette.sort((a, b) => a.h - b.h || a.s - b.s || a.l - b.l); + const hueBuckets = new Array(12).fill(0); + for (const c of palette) if (c.s >= NEUTRAL_S) hueBuckets[Math.floor((c.h % 360) / 30)]++; + + const spacingRaw = [ + ...cssValues(t, SPACING_PROP_RE).flatMap(parseLengths), + ...cssValues(t, GAP_PROP_RE).flatMap(parseLengths), + ...[...t.matchAll(TW_SPACE_RE)].map(([, n]) => (n === "px" ? 1 : +n * 4)).filter(Boolean), + ]; + const spacing = uniqSorted(spacingRaw); + const spacingBase = inferSpacingBase(spacing); + const spacingOnScale = onScaleFraction(spacing, spacingBase); + + const fontFamilies = [ + ...new Set([ + // Only the FIRST family in a stack — the intended face; the rest are fallbacks. + ...cssValues(t, FONT_PROP_RE) + .map((v) => + String(v.split(",")[0]) + .trim() + .replace(/^["']|["']$/g, "") + .toLowerCase(), + ) + .filter((f) => f && !f.includes("(")), // var()/env() indirections carry no face + ...[...t.matchAll(TW_FONT_RE)].map(([, k]) => TW_FONT_STACK[k]), + ]), + ].sort(); + + const radii = uniqSorted([ + // 999+px pill radii normalize to one "full" level — 9999 vs 99999 is not a choice. + ...cssValues(t, RADIUS_PROP_RE) + .flatMap(parseLengths) + .map((r) => (r >= 999 ? 9999 : r)), + ...[...t.matchAll(TW_ROUNDED_RE)] + .map(([, size]) => (size === undefined ? 4 : TW_ROUNDED_PX[size])) + .filter((r) => r > 0), + ]); + + const shadows = new Set([ + ...cssValues(t, SHADOW_PROP_RE) + .map((v) => v.trim().replace(/\s+/g, " ")) + .filter((v) => v !== "none"), + ...[...t.matchAll(TW_SHADOW_RE)] + .map(([, size]) => `tw:${size ?? "base"}`) + .filter((s) => s !== "tw:none"), + ]); + + return { + palette, + paletteSize: palette.length, + hueBuckets, + spacing, + spacingBase, + spacingOnScale, + fontFamilies, + radii, + radiusLevels: radii.length, + shadowLevels: shadows.size, + }; +} + +/** + * Fingerprint a set of files as ONE vector (a design system is a property of the + * whole surface, not any single file). Unreadable files are skipped; the file list + * is sorted first so argument order can never change the vector. + * @param {string} root @param {string[]} files + * @returns {Fingerprint} + */ +export function fingerprintFiles(root, files) { + const texts = []; + for (const f of [...files].sort()) { + try { + texts.push(readFileSync(isAbsolute(f) ? f : join(root, f), "utf8")); + } catch {} + } + return fingerprintText(texts.join("\n")); +} + +// --------------------------------------------------------------------------- +// Slop distance — the shipped generic-template signature set. Each entry is the +// measurable footprint of a recognizable "AI default" look; being NEAR one of these +// is the failure. Curated, versioned, extensible (spec §2). +// --------------------------------------------------------------------------- + +/** + * @typedef {{id:string, why:string, hues:number[], spacingBase:number, + * spacingCount:number, fontCount:number, radii:number[], radiusLevels:number, + * shadowLevels:number}} GenericSignature + */ + +/** @type {GenericSignature[]} */ +export const GENERIC_SIGNATURES = [ + { + id: "tailwind-default", + why: "the untouched Tailwind starter: blue-500/indigo-500 accents, flat 8px-everything spacing (the p-2/p-4/p-8 trio), one sans stack, rounded-xl on every card, one soft shadow", + hues: [217, 239], // blue-500 #3b82f6 ≈ h217, indigo-500 #6366f1 ≈ h239 + spacingBase: 8, + spacingCount: 3, // 8/16/32 — a flat scale with no rhythm + fontCount: 1, + radii: [12], // rounded-xl + radiusLevels: 1, + shadowLevels: 1, + }, + { + id: "bootstrap-default", + why: "stock Bootstrap: the #0d6efd primary (h≈216), 1rem spacers halving to 8px steps, one system stack, uniform 6px --bs-border-radius, one shadow", + hues: [216], + spacingBase: 8, + spacingCount: 4, // the $spacer/2 ladder: 8/16/24/48 + fontCount: 1, + radii: [6], + radiusLevels: 1, + shadowLevels: 1, + }, + { + id: "ai-landing-gradient", + why: "the canonical AI landing page: violet→purple gradient hero (h 258–271), airy uniform 8px spacing, rounded-2xl cards, two layered soft shadows", + hues: [258, 271], // violet-500 #8b5cf6, purple-500 #a855f7 + spacingBase: 8, + spacingCount: 3, + fontCount: 1, + radii: [16], // rounded-2xl + radiusLevels: 1, + shadowLevels: 2, + }, +]; + +const round3 = (x) => Math.round(x * 1000) / 1000; +const hueDist = (a, b) => { + const d = Math.abs(a - b) % 360; + return Math.min(d, 360 - d); +}; +const chromaticHues = (fp) => fp.palette.filter((c) => c.s >= NEUTRAL_S).map((c) => c.h); + +/** Tolerate a raw ledger body (or partial vector) anywhere a fingerprint is read. */ +const asFp = (fp) => ({ + palette: fp?.palette ?? [], + paletteSize: fp?.paletteSize ?? 0, + hueBuckets: fp?.hueBuckets ?? new Array(12).fill(0), + spacing: fp?.spacing ?? [], + spacingBase: fp?.spacingBase ?? null, + spacingOnScale: fp?.spacingOnScale ?? 1, + fontFamilies: fp?.fontFamilies ?? [], + radii: fp?.radii ?? [], + radiusLevels: fp?.radiusLevels ?? 0, + shadowLevels: fp?.shadowLevels ?? 0, +}); + +// Per-feature distances (each in [0,1]) from a fingerprint to one generic signature. +// Only features the input actually exhibits are compared — an all-CSS-variables file +// with no radii must not be judged on radii it doesn't have. +function sigFeatures(fp, sig) { + const out = []; + if (fp.paletteSize > 0) { + const hues = chromaticHues(fp); + // A neutral-only palette is maximally far from a blue-band template (grayscale is + // a deliberate stance, e.g. brutalist), hence d=1 rather than "incomparable". + const d = hues.length + ? hues.reduce((s, h) => s + Math.min(...sig.hues.map((z) => hueDist(h, z))), 0) / + hues.length / + 180 + : 1; + out.push({ feature: "palette", d: round3(d) }); + } + if (fp.spacing.length) { + const baseD = fp.spacingBase === sig.spacingBase ? 0 : 1; + const divD = Math.min(1, Math.abs(fp.spacing.length - sig.spacingCount) / 8); + out.push({ feature: "spacing", d: round3((baseD + divD) / 2) }); + } + if (fp.fontFamilies.length) + out.push({ + feature: "type", + d: round3(Math.min(1, Math.abs(fp.fontFamilies.length - sig.fontCount) / 2)), + }); + if (fp.radii.length) { + const near = Math.min( + ...fp.radii.map((r) => Math.min(...sig.radii.map((z) => Math.min(1, Math.abs(r - z) / 16)))), + ); + const lev = Math.min(1, Math.abs(fp.radiusLevels - sig.radiusLevels) / 3); + out.push({ feature: "shape", d: round3((near + lev) / 2) }); + } + // Zero shadows on a UI that exhibits other features IS signal (flat ≠ template + // soft-shadow); but on an empty vector it would be judging nothing, so gate on + // having at least one other measurable feature. + if (out.length || fp.shadowLevels > 0) + out.push({ + feature: "elevation", + d: round3(Math.min(1, Math.abs(fp.shadowLevels - sig.shadowLevels) / 3)), + }); + return out; +} + +/** + * Nearest generic signature with its per-feature breakdown (what uiGate turns into + * actionable violations). Null when the input had nothing measurable. + * @param {Fingerprint} fingerprint + * @returns {{id:string, why:string, distance:number, features:{feature:string,d:number}[]}|null} + */ +export function nearestGeneric(fingerprint) { + const fp = asFp(fingerprint); + let best = null; + for (const sig of GENERIC_SIGNATURES) { + const features = sigFeatures(fp, sig); + if (!features.length) continue; + const distance = round3(features.reduce((s, f) => s + f.d, 0) / features.length); + if (!best || distance < best.distance) best = { id: sig.id, why: sig.why, distance, features }; + } + return best; +} + +/** Normalized distance (0..1) to the NEAREST generic-template signature. Low = slop. + * An empty/unmeasurable vector returns 1 — nothing measurable is not generic. */ +export function slopDistance(fingerprint) { + return nearestGeneric(fingerprint)?.distance ?? 1; +} + +// --------------------------------------------------------------------------- +// Conformance — distance from the project's own design system. +// --------------------------------------------------------------------------- + +const jaccardDist = (a, b) => { + const A = new Set(a); + const B = new Set(b); + const uni = new Set([...A, ...B]).size; + if (!uni) return 0; + return 1 - [...A].filter((x) => B.has(x)).length / uni; +}; + +// Per-feature distances between two fingerprints; comparable features only. +function conformFeatures(a, b) { + const out = []; + if (a.paletteSize || b.paletteSize) { + const ha = chromaticHues(a); + const hb = chromaticHues(b); + let d; + if (!ha.length && !hb.length) + d = 0; // two neutral-only systems agree + else if (!ha.length || !hb.length) + d = 1; // chromatic vs grayscale — different worlds + else { + // Symmetric mean nearest-hue distance (an averaged Hausdorff): every hue in + // each palette must have a home in the other. + const dir = (xs, ys) => + xs.reduce((s, h) => s + Math.min(...ys.map((z) => hueDist(h, z))), 0) / xs.length; + d = (dir(ha, hb) + dir(hb, ha)) / 2 / 180; + } + out.push({ feature: "palette", d: round3(d) }); + } + if (a.spacing.length && b.spacing.length) { + // Base disagreement in octaves (2 vs 8 is worse than 4 vs 8) blended with how + // much of the OUTPUT sits off the PROJECT's base — the actionable half. + const baseD = Math.min(1, Math.abs(Math.log2((a.spacingBase || 1) / (b.spacingBase || 1))) / 2); + const scaleD = 1 - onScaleFraction(a.spacing, b.spacingBase ?? undefined); + out.push({ feature: "spacing", d: round3((baseD + scaleD) / 2) }); + } + if (a.fontFamilies.length || b.fontFamilies.length) + out.push({ feature: "type", d: round3(jaccardDist(a.fontFamilies, b.fontFamilies)) }); + if (a.radii.length || b.radii.length) { + const lev = Math.min(1, Math.abs(a.radiusLevels - b.radiusLevels) / 3); + out.push({ feature: "shape", d: round3((jaccardDist(a.radii, b.radii) + lev) / 2) }); + } + out.push({ + feature: "elevation", + d: round3(Math.min(1, Math.abs(a.shadowLevels - b.shadowLevels) / 3)), + }); + return out; +} + +/** + * Normalized distance (0..1) between a fingerprint and the project's. High = the + * output ignored the system the codebase already has. + * @param {Fingerprint} fingerprint @param {Fingerprint} projectFp + */ +export function conformance(fingerprint, projectFp) { + const feats = conformFeatures(asFp(fingerprint), asFp(projectFp)); + return feats.length ? round3(feats.reduce((s, f) => s + f.d, 0) / feats.length) : 0; +} + +// --------------------------------------------------------------------------- +// The gate — actionable violations, never a bare score (spec §2: each failing +// feature maps to a concrete edit). +// --------------------------------------------------------------------------- + +/** Default thresholds; taste profiles override these in P8 once fixtures show separation. */ +export const UI_GATE_DEFAULTS = { tauSlop: 0.25, tauConform: 0.5 }; + +const SLOP_HINTS = { + palette: (fp, near) => + `palette hues sit in the ${near.id} band — pick a brand hue (or the project's accent) outside it`, + spacing: (fp) => + `spacing is the uniform ${fp.spacingBase ?? 8}px template rhythm — use a deliberate scale (e.g. the project's 4-based scale with real jumps)`, + type: () => + "a single default font stack is the strongest template tell — use the project's faces or a deliberate pairing", + shape: () => + "one uniform large radius on everything reads as template — commit to few deliberate radii (0 counts)", + elevation: () => + "the one-soft-shadow-on-every-card look is generic — flatten, or define explicit elevation steps", +}; + +const CONFORM_HINTS = { + palette: (fp, proj) => + `output hues [${chromaticHues(proj).join(", ") || "neutral-only"}] are the project's — reuse those accents instead of [${chromaticHues(fp).join(", ") || "none"}]`, + spacing: (fp, proj) => + `the project spacing base is ${proj.spacingBase}px — put values on that scale (output inferred base ${fp.spacingBase}px)`, + type: (fp, proj) => + `output fonts [${fp.fontFamilies.join(", ")}] don't match the project's [${proj.fontFamilies.join(", ")}] — use the project stacks`, + shape: (fp, proj) => + `project radii are [${proj.radii.join(", ")}] — use those levels, not [${fp.radii.join(", ")}]`, + elevation: (fp, proj) => + `project uses ${proj.shadowLevels} shadow level(s), output uses ${fp.shadowLevels} — match the project's elevation system`, +}; + +/** + * The two-sided quality gate: PASS iff slop ≥ tauSlop AND (when a project + * fingerprint exists) conform ≤ tauConform. Violations name the driving feature and + * a concrete edit; because each per-feature distance is in [0,1], a failing mean + * always has at least one failing feature — a FAIL can never arrive hint-less. + * @param {Fingerprint} fingerprint + * @param {{projectFp?:Fingerprint|null, tauSlop?:number, tauConform?:number}} [opts] + * @returns {{pass:boolean, slop:number, conform:number|null, + * violations:{feature:string, detail:string, hint:string}[]}} + */ +export function uiGate(fingerprint, opts = {}) { + const { projectFp = null, tauSlop, tauConform } = { ...UI_GATE_DEFAULTS, ...opts }; + const fp = asFp(fingerprint); + const violations = []; + const near = nearestGeneric(fp); + const slop = near?.distance ?? 1; + if (near && slop < tauSlop) { + for (const f of near.features) + if (f.d < tauSlop) + violations.push({ + feature: f.feature, + detail: `${f.feature} is Δ${f.d} from the "${near.id}" template (need ≥ ${tauSlop} overall)`, + hint: SLOP_HINTS[f.feature](fp, near), + }); + } + let conform = null; + if (projectFp) { + const proj = asFp(projectFp); + const feats = conformFeatures(fp, proj); + conform = feats.length ? round3(feats.reduce((s, f) => s + f.d, 0) / feats.length) : 0; + if (conform > tauConform) { + for (const f of feats) + if (f.d > tauConform) + violations.push({ + feature: f.feature, + detail: `${f.feature} is Δ${f.d} from the project fingerprint (need ≤ ${tauConform} overall)`, + hint: CONFORM_HINTS[f.feature](fp, proj), + }); + } + } + return { pass: violations.length === 0, slop, conform, violations }; +} + +// --------------------------------------------------------------------------- +// Scale-conformance checks — the ASSERTABLE_CHECKS extension (spec §4). Same shape +// as uicheck's list: deterministic, per-fingerprint, pass|fail + fix hint. The ids +// mirror the entries added to uicheck.ASSERTABLE_CHECKS (drift-tested). +// --------------------------------------------------------------------------- + +/** A design system uses FEW levels, deliberately — these caps encode that. */ +export const SCALE_CHECK_DEFAULTS = { + epsilon: 0.5, // px tolerance for "on the base scale" (sub-pixel = rounding noise) + minOnScale: 0.9, // ≥90% of spacing values must sit on the base + maxRadiusLevels: 3, + maxShadowLevels: 3, + maxPalette: 8, // distinct normalized colors — beyond this it's pixel-soup, not a palette +}; + +/** + * Run the deterministic scale checks over a fingerprint. + * @param {Fingerprint} fingerprint + * @param {{base?:number|null, epsilon?:number, minOnScale?:number, maxRadiusLevels?:number, + * maxShadowLevels?:number, maxPalette?:number}} [opts] + * `base` = the DECLARED design-system base; defaults to the inferred one. + * @returns {{id:string, pass:boolean, detail:string, hint:string}[]} + */ +export function scaleChecks(fingerprint, opts = {}) { + const fp = asFp(fingerprint); + const o = { ...SCALE_CHECK_DEFAULTS, base: fp.spacingBase, ...opts }; + const off = o.base ? fp.spacing.filter((s) => offBase(s, o.base) > o.epsilon) : []; + const onFrac = fp.spacing.length ? 1 - off.length / fp.spacing.length : 1; + return [ + { + id: "spacing-scale", + pass: onFrac >= o.minOnScale, + detail: `${Math.round(onFrac * 100)}% of ${fp.spacing.length} spacing value(s) on the ${o.base ?? "(none)"}px base (ε ${o.epsilon}px)`, + hint: off.length ? `move ${off.join(", ")}px onto the ${o.base}px scale` : "", + }, + { + id: "radius-levels", + pass: fp.radiusLevels <= o.maxRadiusLevels, + detail: `${fp.radiusLevels} distinct radius level(s) (max ${o.maxRadiusLevels})`, + hint: + fp.radiusLevels > o.maxRadiusLevels + ? `collapse [${fp.radii.join(", ")}] to ≤${o.maxRadiusLevels} deliberate levels` + : "", + }, + { + id: "shadow-levels", + pass: fp.shadowLevels <= o.maxShadowLevels, + detail: `${fp.shadowLevels} distinct shadow level(s) (max ${o.maxShadowLevels})`, + hint: + fp.shadowLevels > o.maxShadowLevels + ? `define ≤${o.maxShadowLevels} elevation steps and reuse them` + : "", + }, + { + id: "palette-size", + pass: fp.paletteSize <= o.maxPalette, + detail: `${fp.paletteSize} distinct color(s) (max ${o.maxPalette})`, + hint: + fp.paletteSize > o.maxPalette + ? "consolidate to design tokens — a palette is a decision, not an accumulation" + : "", + }, + ]; +} + +// --------------------------------------------------------------------------- +// The project fingerprint claim — v_proj lives in the PCM ledger so it is shared +// with the team and updated by the same evidence rules as everything else. +// --------------------------------------------------------------------------- + +/** + * Extract the project fingerprint from `files` and store it as a `fingerprint` + * claim. Content-addressed: the same UI surface mints the same id on every machine, + * so teammates converge on one claim instead of duplicating. + * @param {string} root @param {string[]} files @param {{t?:number}} [opts] + * @returns {{ok:true, id:string, existed:boolean, fingerprint:Fingerprint}|{ok:false, reason:string}} + */ +export function mintProjectFingerprint(root, files, { t = 0 } = {}) { + const fingerprint = fingerprintFiles(root, files); + const minted = mintClaim({ + kind: "fingerprint", + body: fingerprint, + scope: { level: "repo" }, + provenance: { agent: "uicheck", author: gitAuthor() }, + t, + }); + if (!minted.ok) return { ok: false, reason: "reason" in minted ? minted.reason : "mint failed" }; + const dir = repoLedger(root); + const put = putClaim(dir, minted.claim); + if (!put.ok) return { ok: false, reason: put.reason ?? "putClaim failed" }; + reindex(dir, t); + return { ok: true, id: minted.claim.id, existed: Boolean(put.existed), fingerprint }; +} + +/** + * The stored project fingerprint (latest live `fingerprint` claim), or null on a + * greenfield repo — the gate then runs slop-only (spec §2). + * @param {string} root + * @returns {Fingerprint|null} + */ +export function loadProjectFingerprint(root) { + const live = loadClaims(repoLedger(root)).filter((c) => c.kind === "fingerprint" && !c.tombstone); + if (!live.length) return null; + live.sort((a, b) => (b.provenance?.t ?? 0) - (a.provenance?.t ?? 0) || (a.id < b.id ? -1 : 1)); + return live[0].body; +} diff --git a/test/uifingerprint.test.js b/test/uifingerprint.test.js new file mode 100644 index 0000000..6a36abc --- /dev/null +++ b/test/uifingerprint.test.js @@ -0,0 +1,245 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { loadClaims, repoLedger } from "../src/ledger_store.js"; +import { ASSERTABLE_CHECKS } from "../src/uicheck.js"; +import { + conformance, + fingerprintFiles, + fingerprintText, + GENERIC_SIGNATURES, + inferSpacingBase, + loadProjectFingerprint, + mintProjectFingerprint, + nearestGeneric, + onScaleFraction, + scaleChecks, + slopDistance, + UI_GATE_DEFAULTS, + uiGate, +} from "../src/uifingerprint.js"; + +const CLI = fileURLToPath(new URL("../src/cli.js", import.meta.url)); +const runCli = (args, cwd) => spawnSync("node", [CLI, ...args], { cwd, encoding: "utf8" }); +const tmp = () => mkdtempSync(join(tmpdir(), "forge-uifp-")); + +// The generic-template look: default-Tailwind blue/indigo, flat 8px spacing, one +// font, uniform rounded-xl, one soft shadow — must FAIL the gate. +const GENERIC_CSS = ` +.card { background: #ffffff; color: #3b82f6; border: 1px solid #6366f1; + padding: 16px; margin: 8px; gap: 32px; + font-family: Inter, sans-serif; + border-radius: 12px; + box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1); } +.hero { background: #6366f1; padding: 32px 16px; border-radius: 12px; } +`; + +// A distinctive custom system: warm ink/paper neutrals + a red accent, 4-based +// spacing with real jumps, two deliberate faces, near-square corners, no shadows — +// must PASS. +const CUSTOM_CSS = ` +:root { --ink: #1c1b18; --paper: #f5f1e8; --accent: #e63946; } +h1 { font-family: "Fraunces", serif; margin: 4px 0 20px; } +body { font-family: "Atkinson Hyperlegible", sans-serif; color: #1c1b18; + background: #f5f1e8; padding: 12px 28px; } +.note { border-radius: 2px; gap: 44px; border: 2px solid #e63946; } +`; + +const TW_JSX = `export const Card = () => ( +
    + +
    );`; + +test("fingerprintText: CSS fixture — palette normalized to HSL, spacing/type/shape extracted", () => { + const fp = fingerprintText(GENERIC_CSS); + // #fff, #3b82f6, #6366f1 (deduped), rgba(0,0,0) from the shadow. + assert.equal(fp.paletteSize, 4); + assert.ok( + fp.palette.some((c) => c.h === 217 && c.s === 91), + "blue-500 hex lands on h217", + ); + assert.ok( + fp.palette.some((c) => c.h === 239), + "indigo-500 hex lands on h239", + ); + assert.equal(fp.hueBuckets[7], 2, "both chromatic hues fall in the 210–239 bin"); + assert.deepEqual(fp.spacing, [8, 16, 32]); + assert.equal(fp.spacingBase, 8); + assert.equal(fp.spacingOnScale, 1); + assert.deepEqual(fp.fontFamilies, ["inter"]); + assert.deepEqual(fp.radii, [12]); + assert.equal(fp.radiusLevels, 1); + assert.equal(fp.shadowLevels, 1); +}); + +test("fingerprintText: Tailwind-class JSX — classes map to px / hues without any CSS", () => { + const fp = fingerprintText(TW_JSX); + assert.deepEqual(fp.spacing, [8, 16, 24, 32], "p-4/m-2/gap-6/px-8/py-2 → n×4 px"); + assert.equal(fp.spacingBase, 8); + assert.ok( + fp.palette.some((c) => c.h === 217), + "bg-blue-500 → h217", + ); + assert.ok( + fp.palette.some((c) => c.s === 0 && c.l === 100), + "text-white → neutral white", + ); + assert.deepEqual(fp.fontFamilies, ["sans-serif"]); + assert.deepEqual(fp.radii, [12], "rounded-xl → 12px"); + assert.equal(fp.shadowLevels, 1, "shadow-lg twice is ONE level"); +}); + +test("fingerprintText is deterministic (same text → deep-equal vector)", () => { + assert.deepEqual(fingerprintText(CUSTOM_CSS), fingerprintText(CUSTOM_CSS)); +}); + +test("inferSpacingBase: largest base that fits wins; off-scale falls back to argmin", () => { + assert.equal(inferSpacingBase([8, 16, 32]), 8); + assert.equal(inferSpacingBase([4, 12, 20]), 4, "multiples of 4 but not 8 → 4, not 2"); + assert.equal(inferSpacingBase([5, 13, 21]), 2, "nothing fits cleanly → smallest residual"); + assert.equal(inferSpacingBase([]), null); + assert.equal(onScaleFraction([8, 16, 31], 8), 2 / 3); +}); + +test("slopDistance: the generic fixture sits ON a signature; the custom one is far", () => { + const generic = fingerprintText(GENERIC_CSS); + const custom = fingerprintText(CUSTOM_CSS); + assert.ok(slopDistance(generic) < UI_GATE_DEFAULTS.tauSlop, "generic is inside the slop radius"); + assert.equal(nearestGeneric(generic)?.id, "tailwind-default"); + assert.ok(slopDistance(custom) >= UI_GATE_DEFAULTS.tauSlop, "custom clears the slop radius"); + assert.equal(slopDistance({}), 1, "nothing measurable is not generic"); +}); + +test("uiGate: known-generic FAILS with named, actionable violations", () => { + const gate = uiGate(fingerprintText(GENERIC_CSS)); + assert.equal(gate.pass, false); + const byFeature = Object.fromEntries(gate.violations.map((v) => [v.feature, v])); + assert.ok(byFeature.palette, "the default-Tailwind palette is named"); + assert.match(byFeature.palette.hint, /brand hue/); + assert.ok(byFeature.spacing); + assert.match(byFeature.spacing.hint, /8px/); + for (const v of gate.violations) assert.ok(v.hint.length > 0, "every violation carries a fix"); +}); + +test("uiGate: distinctive custom fixture PASSES (slop-only and vs its own system)", () => { + const custom = fingerprintText(CUSTOM_CSS); + assert.equal(uiGate(custom).pass, true); + const gate = uiGate(custom, { projectFp: custom }); + assert.equal(gate.pass, true); + assert.equal(gate.conform, 0, "a fingerprint conforms perfectly to itself"); +}); + +test("conformance: flags divergence from a provided project fingerprint", () => { + const generic = fingerprintText(GENERIC_CSS); + const project = fingerprintText(CUSTOM_CSS); + assert.equal(conformance(project, project), 0); + assert.ok(conformance(generic, project) > UI_GATE_DEFAULTS.tauConform); + const gate = uiGate(generic, { projectFp: project }); + assert.equal(gate.pass, false); + const type = gate.violations.find((v) => v.feature === "type" && /project/.test(v.detail)); + assert.ok(type, "font divergence from the project is a named violation"); + assert.match(type.hint, /fraunces/); +}); + +test("scaleChecks: off-scale spacing, level sprawl, and palette bloat all fail with hints", () => { + const messy = { + palette: [], + paletteSize: 12, + spacing: [8, 16, 31], + spacingBase: 8, + fontFamilies: [], + radii: [1, 2, 3, 4, 6], + radiusLevels: 5, + shadowLevels: 5, + }; + const byId = Object.fromEntries(scaleChecks(messy).map((c) => [c.id, c])); + assert.equal(byId["spacing-scale"].pass, false); + assert.match(byId["spacing-scale"].hint, /31/); + assert.equal(byId["radius-levels"].pass, false); + assert.equal(byId["shadow-levels"].pass, false); + assert.equal(byId["palette-size"].pass, false); + assert.ok(scaleChecks(fingerprintText(CUSTOM_CSS)).every((c) => c.pass)); +}); + +test("scaleChecks ids are wired into uicheck's ASSERTABLE_CHECKS (no drift)", () => { + const assertable = new Set(ASSERTABLE_CHECKS.map((c) => c.id)); + for (const c of scaleChecks(fingerprintText(GENERIC_CSS))) + assert.ok(assertable.has(c.id), `${c.id} missing from ASSERTABLE_CHECKS`); + assert.ok(assertable.has("contrast"), "the existing WCAG surface is untouched"); +}); + +test("generic signatures document their rationale", () => { + for (const sig of GENERIC_SIGNATURES) { + assert.ok(sig.why.length > 20, `${sig.id} needs a why`); + assert.ok(sig.hues.length >= 1); + } +}); + +test("mintProjectFingerprint: claim lands in the ledger; id stable for the same inputs", () => { + const root = tmp(); + writeFileSync(join(root, "app.css"), CUSTOM_CSS); + const a = mintProjectFingerprint(root, ["app.css"], { t: 1 }); + assert.equal(a.ok, true); + if (!a.ok) return; + assert.equal(a.existed, false); + const b = mintProjectFingerprint(root, ["app.css"], { t: 2 }); + if (!b.ok) assert.fail(b.reason); + else { + assert.equal(b.id, a.id, "content-addressed: same UI surface → same claim id"); + assert.equal(b.existed, true); + } + const claims = loadClaims(repoLedger(root)).filter((c) => c.kind === "fingerprint"); + assert.equal(claims.length, 1); + assert.equal(claims[0].provenance.agent, "uicheck"); + assert.deepEqual(loadProjectFingerprint(root), a.fingerprint); +}); + +test("fingerprintFiles: argument order never changes the vector; missing files are skipped", () => { + const root = tmp(); + writeFileSync(join(root, "a.css"), GENERIC_CSS); + writeFileSync(join(root, "b.jsx"), TW_JSX); + assert.deepEqual( + fingerprintFiles(root, ["a.css", "b.jsx", "ghost.css"]), + fingerprintFiles(root, ["b.jsx", "a.css"]), + ); +}); + +test("cli: legacy `uicheck ` and new `uicheck contrast ` both work", () => { + const cwd = tmp(); + const legacy = runCli(["uicheck", "#000000", "#ffffff"], cwd); + assert.equal(legacy.status, 0); + assert.match(legacy.stdout, /21:1/); + const named = runCli(["uicheck", "contrast", "#000000", "#ffffff"], cwd); + assert.equal(named.status, 0); + assert.match(named.stdout, /21:1/); +}); + +test("cli: `uicheck design` exits 1 on the generic fixture, names the fixes", () => { + const cwd = tmp(); + writeFileSync(join(cwd, "generic.css"), GENERIC_CSS); + const r = runCli(["uicheck", "design", "generic.css"], cwd); + assert.equal(r.status, 1); + assert.match(r.stdout, /slop distance/); + assert.match(r.stdout, /fix:/); + assert.match(r.stdout, /FAIL/); +}); + +test("cli: fingerprint --mint stores the project claim; design then gates against it", () => { + const cwd = tmp(); + writeFileSync(join(cwd, "app.css"), CUSTOM_CSS); + const mint = runCli(["uicheck", "fingerprint", "app.css", "--mint", "--json"], cwd); + assert.equal(mint.status, 0); + const out = JSON.parse(mint.stdout); + assert.equal(out.minted.ok, true); + assert.ok(out.fingerprint.paletteSize > 0); + const design = runCli(["uicheck", "design", "app.css", "--json"], cwd); + assert.equal(design.status, 0, design.stdout); + const gate = JSON.parse(design.stdout); + assert.equal(gate.pass, true); + assert.equal(gate.hasProjectFingerprint, true); + assert.equal(gate.conform, 0); +}); From b92ee4526dd01430e040a23f761bd506a80c7cdf Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 17:27:11 +0000 Subject: [PATCH 5/5] polish: make forge dash pass its own UI gate; fix unused param MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The P6 gate judged the P7 dashboard 'too close to the tailwind-default template' (slop distance 0.091 < 0.25): cool blue-gray neutrals and a single soft glow. Warmed the dark neutrals to the brand ember's hue and flattened the glow — slop distance now 0.43, gate PASSES. The dashboard is the reference customer of the taste discipline, and the gate catching it first is the discipline working. Known P6 limitation surfaced by this: the CSS extractor doesn't resolve var() indirection, so spacing tokens defined once as custom properties count as a single value — noted for the taste-profile follow-up. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Fc2MvWJbQ1cyNQ76ymv4hs --- src/dash.html | 18 +++++++++--------- src/uifingerprint.js | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/dash.html b/src/dash.html index 2f187c7..4d09d65 100644 --- a/src/dash.html +++ b/src/dash.html @@ -9,13 +9,13 @@ own — not framework blue), a 4px-based spacing scale, exactly two radius levels, system sans + one mono. Palette matches the landing page = close to home. */ :root { - --bg: #12131a; - --panel: #1b1d27; - --panel-2: #21242f; - --line: #2b2e3b; - --fg: #ecedf3; - --muted: #9a9dae; - --faint: #6b6e7e; + --bg: #171310; + --panel: #201a15; + --panel-2: #272019; + --line: #372c22; + --fg: #f2ede7; + --muted: #a99e90; + --faint: #7d7263; --ember: #f26430; --ember-wash: rgba(242, 100, 48, 0.14); --r-s: 4px; /* radius level 1: chips, bars, inputs */ @@ -40,12 +40,12 @@ position: sticky; top: 0; z-index: 5; display: flex; align-items: center; gap: var(--s3); padding: var(--s3) var(--s6); - background: rgba(18, 19, 26, 0.85); backdrop-filter: blur(8px); + background: rgba(23, 19, 16, 0.85); backdrop-filter: blur(8px); border-bottom: 1px solid var(--line); } .spark { width: 8px; height: 8px; border-radius: var(--r-s); - background: var(--ember); box-shadow: 0 0 10px var(--ember); + background: var(--ember); } .brand { font-family: var(--mono); font-size: 13px; letter-spacing: 0.02em; } .brand b { color: var(--ember); font-weight: 600; } diff --git a/src/uifingerprint.js b/src/uifingerprint.js index e14068c..1e5914b 100644 --- a/src/uifingerprint.js +++ b/src/uifingerprint.js @@ -506,7 +506,7 @@ export function conformance(fingerprint, projectFp) { export const UI_GATE_DEFAULTS = { tauSlop: 0.25, tauConform: 0.5 }; const SLOP_HINTS = { - palette: (fp, near) => + palette: (_fp, near) => `palette hues sit in the ${near.id} band — pick a brand hue (or the project's accent) outside it`, spacing: (fp) => `spacing is the uniform ${fp.spacingBase ?? 8}px template rhythm — use a deliberate scale (e.g. the project's 4-based scale with real jumps)`,