From 9615e258906be144844bc5d491106b468eacfaad Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 15:35:32 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20substrate-v2=20plan=20=E2=80=94=20compl?= =?UTF-8?q?ete=20the=20whitepaper=20via=20the=20Proof-Carrying=20Memory=20?= =?UTF-8?q?protocol?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nine specs (docs/plans/substrate-v2/00-08) closing every remaining gap between the cognitive-substrate whitepaper and the shipped code, plus the owner-named pains (cost, reuse, team memory, incomplete context, generated-UI quality): - 01 PCM protocol: content-addressed claims, Beta-posterior confidence from independent oracles only, Eq. 3 retrieval, ilm->fahm->hikma layering - 02 team memory: git-native CRDT ledger (set-union semilattice merge) - 03 reuse cache: proof-carrying artifact cache (MinHash/LSH near-match, atlas revalidation, outcome-driven demotion) - 04 context assembly: token-budget knapsack + set-cover completeness gate - 05 cost model: measured-vs-target composition toward ~90%, P8 harness - 06 faculties/mechanisms: AST-tier atlas + mandatory gate, imagination dry-run (test set cover + worktree sandbox), oracle cascade, M3 partition planning, M4 CUSUM drift, M5 footprint metric, M6 checkpoint cadence, outcome-calibrated routing - 07 UI quality gate: design fingerprints, slop-distance + conformance - 08 forge dash: local dashboard over the ledger and metrics ADR-0005 (selective optional deps with stdlib fallbacks, supersedes 0001's blanket rule), ADR-0006 (PCM as the one store), ROADMAP P1-P8, CHANGELOG. Docs only; no runtime changes (222/222 tests green). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Fc2MvWJbQ1cyNQ76ymv4hs --- CHANGELOG.md | 11 + ROADMAP.md | 43 +++- docs/adr/0005-allow-runtime-dependencies.md | 41 ++++ docs/adr/0006-proof-carrying-memory.md | 39 ++++ docs/plans/substrate-v2/00-overview.md | 87 ++++++++ docs/plans/substrate-v2/01-pcm-protocol.md | 193 ++++++++++++++++++ docs/plans/substrate-v2/02-team-memory.md | 103 ++++++++++ docs/plans/substrate-v2/03-reuse-cache.md | 108 ++++++++++ .../plans/substrate-v2/04-context-assembly.md | 131 ++++++++++++ docs/plans/substrate-v2/05-cost-model.md | 90 ++++++++ .../06-faculties-and-mechanisms.md | 161 +++++++++++++++ docs/plans/substrate-v2/07-ui-quality-gate.md | 125 ++++++++++++ docs/plans/substrate-v2/08-dashboard-ux.md | 55 +++++ 13 files changed, 1177 insertions(+), 10 deletions(-) create mode 100644 docs/adr/0005-allow-runtime-dependencies.md create mode 100644 docs/adr/0006-proof-carrying-memory.md create mode 100644 docs/plans/substrate-v2/00-overview.md create mode 100644 docs/plans/substrate-v2/01-pcm-protocol.md create mode 100644 docs/plans/substrate-v2/02-team-memory.md create mode 100644 docs/plans/substrate-v2/03-reuse-cache.md create mode 100644 docs/plans/substrate-v2/04-context-assembly.md create mode 100644 docs/plans/substrate-v2/05-cost-model.md create mode 100644 docs/plans/substrate-v2/06-faculties-and-mechanisms.md create mode 100644 docs/plans/substrate-v2/07-ui-quality-gate.md create mode 100644 docs/plans/substrate-v2/08-dashboard-ux.md diff --git a/CHANGELOG.md b/CHANGELOG.md index e8cf19d..05601c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,17 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Documentation +- **Substrate v2 plan: the whitepaper, completed (`docs/plans/substrate-v2/`).** Nine specs + + two ADRs mapping every remaining paper faculty/mechanism to a concrete algorithm, unified + by the **Proof-Carrying Memory (PCM) protocol**: every stored unit (lesson, fact, cached + artifact, graph edge, design fingerprint, diagnosis) becomes a content-addressed claim whose + confidence is a decayed Beta posterior over independent-oracle outcomes — retrieval implements + the paper's Eq. 3, team memory is a conflict-free CRDT ledger merged over git, code reuse is a + proof-carrying artifact cache, context assembly is a token-budget knapsack with a set-cover + completeness gate, and generated-UI quality is a measurable slop-distance/conformance gate. + ADR-0005 relaxes the zero-dependency rule to selective optional deps with stdlib fallbacks; + ADR-0006 converges all persistence on the PCM ledger. `ROADMAP.md` now carries the P1–P8 + phase plan. Docs only — no runtime behavior changes. - **Visual flow diagrams in the entry-point docs.** A "one source → every tool + pre-action gate" mermaid in `README.md` and a "your day with Forge" loop in `ONBOARDING.md` (alongside the propose→verify diagram in the substrate README) — making the model easier to grasp at a glance, diff --git a/ROADMAP.md b/ROADMAP.md index 224af05..c34ecf2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -13,19 +13,42 @@ governor (`forge cost`), spec-as-contract drift (`forge spec`), goal-drift check [docs/cognitive-substrate/](./docs/cognitive-substrate/). See [CHANGELOG.md](./CHANGELOG.md) `[Unreleased]`. -## Next -- **Substrate calibration fixtures** — expand assumption/routing/impact fixtures beyond - the original paper prototypes while keeping research-edge claims advisory. -- **AST-backed impact** — an optional exact call graph behind the regex-approximate atlas - (weighed against the zero-runtime-dependency constraint). -- **Testing** — Playwright MCP/agents opt-in + a coverage Stop-hook gate. -- **MCP hygiene** — enforce the ~6-server cap + a registry resolver in `forge doctor`. +## Next — Substrate v2 (the whitepaper, completed) +The full plan lives in [docs/plans/substrate-v2/](./docs/plans/substrate-v2/00-overview.md): +every paper faculty and mechanism mapped to an algorithm, unified by the +**Proof-Carrying Memory (PCM) protocol** (ADR-0006) — every stored thing is a claim that +carries its evidence, earns confidence only from independent oracles, and merges across +teammates conflict-free (git-native CRDT ledger). + +Phases (dependency-ordered; acceptance gates in the overview): +- **P1 Ledger core** — claim store, Beta-posterior confidence, Eq. 3 retrieval; + cortex + recall migrate onto claim kinds. +- **P2 Team memory** — `.forge/ledger/` union-merge over git; `forge ledger`. +- **P3 Reuse cache** — proof-carrying artifact cache (`forge reuse`): verified code is + served with its evidence, revalidated against the atlas, demoted by outcomes. +- **P4 Context assembly** — knapsack selection under a token budget + a set-cover + completeness gate that *computes* what's missing (`forge context`). +- **P5 Loop closure** — outcome write-back band; doom-loop diagnosis + escalation; + imagination dry-run (impacted-test set cover + worktree sandbox); AST-tier atlas with + a mandatory impact gate; M3 partition planning, M4 CUSUM drift, M5 footprint metric, + M6 checkpoint cadence; outcome-calibrated routing. +- **P6 UI quality gate** — design fingerprints, slop-distance + project-conformance + scoring, scale/palette conformance checks (`forge uicheck` v2, machine-readable taste). +- **P7 `forge dash`** — local dashboard over the ledger, cost stages, cache, blast radius. +- **P8 Evaluation** — measured per-stage cost report (target ~90 % composed; only + measured numbers get claimed — see [05-cost-model.md](./docs/plans/substrate-v2/05-cost-model.md)). + +Carried forward from the previous roadmap into these phases: calibration fixtures (P8), +AST-backed impact (P5), Playwright testing (P6), MCP hygiene in `forge doctor` (P7). ## Later / exploring -- Optional vector-memory backend (Hindsight / Mem0) wiring. +- Optional embedding/vector backends for retrieval and cache near-match (ADR-0005 tier). - `forge verify` independent-review wired into CI with provenance gating. - Formal/semantic verification — documented as out-of-scope for now. +- Parametric learning channel (LoRA distillation) — deliberately out of scope (ADR-0006). ## Non-goals -Adding runtime dependencies. Reimplementing mature tools (skills installer, memory engine, -SDD framework, SAST) — we wire the best existing ones. Bundling a whole IDE. +Unbounded dependency growth — deps are selective, optional, and always backed by a +stdlib fallback path (ADR-0005). Reimplementing mature tools (skills installer, +subagent orchestration, SDD framework, SAST) — we wire the best existing ones. Bundling +a whole IDE. A hosted memory server — git *is* the sync (see 02-team-memory.md). diff --git a/docs/adr/0005-allow-runtime-dependencies.md b/docs/adr/0005-allow-runtime-dependencies.md new file mode 100644 index 0000000..0a08679 --- /dev/null +++ b/docs/adr/0005-allow-runtime-dependencies.md @@ -0,0 +1,41 @@ +# ADR 0005: Allow selective runtime dependencies + +- Status: accepted (supersedes the blanket rule in ADR-0001) +- Date: 2026-07-07 + +## Context +ADR-0001 committed the CLI to zero production dependencies, sized for a config emitter +plus advisory gates. Substrate v2 (docs/plans/substrate-v2/) adds capabilities where the +stdlib ceiling has real costs: AST-accurate impact analysis (TypeScript compiler API), +higher-recall semantic matching for the reuse cache and retrieval (embedding backends), +fast ledger indexing at monorepo scale (SQLite), and the UI visual loop (Playwright). +Writing these from scratch would be reinventing mature tools — the exact waste +ARCHITECTURE.md's reuse ledger forbids. Meanwhile the symlink install channel that +motivated ADR-0001 still exists and must not break. + +## Decision +Runtime dependencies are permitted under four conditions, each enforced in review: + +1. **Core stays stdlib.** Every substrate feature has a pure-JS/stdlib reference path + (MinHash instead of embeddings, JSON-file scan instead of SQLite, regex atlas tier + instead of AST). Dependencies buy *better*, never *at all*. +2. **Graceful absence.** A missing optional dependency degrades to the stdlib path with + a `forge doctor` note — never a crash. The symlink/plugin channels ship the stdlib + paths and remain dependency-free. +3. **Named tiers.** `dependencies` stays empty; enhancements live in + `optionalDependencies` or documented opt-in installs, each mapped to the feature it + unlocks. +4. **Vetting.** Each new dependency passes `forge scan`-grade review (provenance, + install scripts, transitive count) and is recorded in the reuse ledger with its + rationale. + +## Consequences +- (+) AST-tier atlas, embedding retrieval, SQLite indexes, and the Playwright visual + loop become buildable without forking the architecture. +- (+) The zero-dep guarantee survives where it matters (plugin/symlink channels, `npx` + cold start) because it is now a per-path property, not a package-wide one. +- (−) Two code paths per enhanced feature — the fallback must be tested as a first-class + path (CI runs the suite with optional deps absent *and* present). +- (−) Supply-chain surface grows; bounded by conditions 3–4. +- CI's "dependencies is empty" assertion is kept; a new assertion checks every + optionalDependency has a registered fallback test. diff --git a/docs/adr/0006-proof-carrying-memory.md b/docs/adr/0006-proof-carrying-memory.md new file mode 100644 index 0000000..7798874 --- /dev/null +++ b/docs/adr/0006-proof-carrying-memory.md @@ -0,0 +1,39 @@ +# ADR 0006: Proof-Carrying Memory is the one store + +- Status: accepted +- Date: 2026-07-07 + +## Context +The whitepaper's sharpest novelty is validity-anchored memory: stored knowledge scored by +whether external oracles later confirmed it (Eq. 3's `val` term), never by the model's +say-so. v0.4 approximates this in two parallel stores with different rules — cortex +lessons (α/β evidence, decay) and recall facts (plain files, no validity) — and the +planned subsystems (reuse cache, context summaries, design fingerprints, diagnoses, +team sharing) would each need their own persistence, trust, and merge story. Divergent +stores mean divergent trust semantics and no coherent team-sync path. + +## Decision +All substrate persistence converges on one protocol: the **Proof-Carrying Memory (PCM) +claim ledger** (docs/plans/substrate-v2/01-pcm-protocol.md). Every stored unit is a +content-addressed claim carrying provenance and a grow-only evidence set; confidence is +a decayed Beta posterior over independent-oracle outcomes; retrieval implements the +paper's Eq. 3; team sharing is the ledger's CRDT set-union merge over git +(02-team-memory.md). Cortex lessons and recall facts become claim kinds behind their +existing CLIs; new subsystems (artifacts, edges, fingerprints, diagnoses, summaries, +decisions) mint claims from day one. The parametric learning channel (LoRA distillation, +paper §7.2) remains explicitly out of scope. + +## Consequences +- (+) One trust model everywhere: anything the substrate believes can answer "why?" + with machine-readable evidence, and anything contradicted by outcomes demotes itself — + including cached code and context summaries, not just lessons. +- (+) Team memory falls out of the storage choice (semilattice merge ⇒ conflict-free by + construction) instead of being a bolted-on sync feature. +- (+) The write-back band (Eq. 2) has one place to write, so outcome-validated learning + covers every subsystem at once. +- (−) A migration: lessons/recall move onto adapters; their tests must pass unchanged + (the P1 acceptance gate) and the old on-disk stores need a one-shot import. +- (−) The ledger becomes load-bearing infrastructure — its canonicalization and merge + invariants require property-level tests, a higher bar than the stores it replaces. +- (−) Repo history carries the ledger; bounded by canonical-JSON compactness, sharding, + and the attic-prune policy. diff --git a/docs/plans/substrate-v2/00-overview.md b/docs/plans/substrate-v2/00-overview.md new file mode 100644 index 0000000..3fef997 --- /dev/null +++ b/docs/plans/substrate-v2/00-overview.md @@ -0,0 +1,87 @@ +# Substrate v2 — completing the whitepaper, and the Proof-Carrying Memory protocol + +> Status: **P0 — specification** · Owner: forgekit core · Companion to +> [docs/cognitive-substrate/](../../cognitive-substrate/) (the paper this plan completes) + +ForgeKit v0.4 implements roughly half of the committed whitepaper — the two prototyped +mechanisms (M1 routing, M2 assumption gate), an approximate impact atlas, and an advisory +memory (cortex + recall). This plan closes the rest, and it closes the pains the paper's +build-opportunity map (§10) ranks highest: validity-anchored memory, outcome-validated +learning, team/shared memory, code reuse, context completeness, doom-loop diagnosis — plus +two pains named by the project owner that the paper only gestures at: **generated-UI +template convergence** and a **~90 % cost-reduction target**. + +One idea unifies all of it: the **Proof-Carrying Memory (PCM) protocol** +([01-pcm-protocol.md](./01-pcm-protocol.md)). Every unit of knowledge the system holds — +a lesson, a cached code artifact, a dependency edge, a design fingerprint, a diagnosis — +becomes a *claim* that carries its own evidence, earns confidence only from independent +oracles (tests, typecheck, CI, human accept/revert), decays without review, and merges +across teammates without conflicts. This takes the paper's sharpest novelty — the `val` +term of Eq. 3, "validity-anchored memory" (§11) — and turns it from one scoring term into +the storage, trust, and wire protocol for the whole substrate. + +## 1. Gap analysis — paper vs. `src/` + +| Paper capability (§10 map) | ForgeKit v0.4 | Gap | Closed by | +|---|---|---|---| +| M1 routing + M2 assumption gate (opp. #1) | `src/route.js`, `src/preflight.js` — 62 % measured saving | ✅ shipped; residue: outcome-calibrated weights | [06](./06-faculties-and-mechanisms.md) §7 | +| Impact oracle, mandatory gate (opp. #3) | `src/atlas.js` regex graph; gate opt-in (`FORGE_ENFORCE=1`) | precision; gate not mandatory | [06](./06-faculties-and-mechanisms.md) §1 | +| Validity-anchored memory (opp. #2, Eq. 3) | cortex confidence exists (`src/lessons.js` already keeps α/β evidence) | no Eq. 3 retrieval, no forget/consolidate policy, flat store | [01](./01-pcm-protocol.md) | +| Outcome-validated learning (opp. #4, Eq. 2) | outcomes update lessons, but not *the memories that informed an action* | write-back band incomplete | [01](./01-pcm-protocol.md) §6, [06](./06-faculties-and-mechanisms.md) §6 | +| Team / shared memory | single-repo file recall (`src/recall.js`, `src/brain.js`) | no merge semantics, no attribution | [02](./02-team-memory.md) | +| Code reuse / generation cache | `reuse-first` skill is prose only | no artifact cache | [03](./03-reuse-cache.md) | +| Context compression / completeness | none — "incomplete context" is unaddressed | whole module missing | [04](./04-context-assembly.md) | +| Doom-loop root cause (opp. #5) | `doom-loop.sh` detects; no diagnosis, no escalation | diagnosis + escalation | [06](./06-faculties-and-mechanisms.md) §5 | +| Imagination (faculty, §3) | atlas traversal only — no dry-run of consequences | test selection + sandbox | [06](./06-faculties-and-mechanisms.md) §2 | +| M3/M4/M5/M6 (decomposition, drift, lean, inline verify) | `scope.js`/`anchor.js`/`lean.js`/`verify.js` heuristics | each gets its algorithm | [06](./06-faculties-and-mechanisms.md) §3–§6 | +| Generated-UI quality (owner pain; M5-shaped) | `src/uicheck.js` WCAG contrast only; taste is prose | anti-template gate | [07](./07-ui-quality-gate.md) | +| Cost to ~90 % (owner target) | routing alone: 62 % measured | cache + context + gate stages unmeasured | [05](./05-cost-model.md) | +| ForgeKit's own UX | CLI only | `forge dash` dashboard | [08](./08-dashboard-ux.md) | + +## 2. The 11-capability master table + +Every faculty (paper §3) and mechanism (paper §6) with the math, algorithm, or data +structure this plan assigns it. Nothing is left as prose-only discipline. + +| # | Capability | Mechanism | Math / algorithm / data structure | Spec | +|---|---|---|---|---| +| 1 | Memory (faculty) | PCM claim ledger; Eq. 3 retrieval; ʿilm→fahm→ḥikma layers | Content-addressed (Merkle-keyed) claim store; Beta(α,β) posterior with exponential decay; MinHash sketches; consolidation = union-find clustering over Jaccard ≥ τ, promoting episodes → patterns → decision rules | [01](./01-pcm-protocol.md) | +| 2 | Learning (faculty) | Outcome write-back band (Eq. 2): oracle results update the `val` of every claim that informed the action | Bayesian evidence update; rubric-weight calibration by logistic regression over outcome claims (the paper's own "learn the rubric weights" note, §9.3) | [01](./01-pcm-protocol.md) §6 | +| 3 | Imagination (faculty) | Consequence simulator `g` (paper Eq. 4): blast radius → impacted-test selection → sandboxed dry-run | Reverse-dependency traversal with hop-decay; test selection as bipartite set cover (greedy ln n-approx); dry-run result becomes evidence on the prediction claim | [06](./06-faculties-and-mechanisms.md) §2 | +| 4 | Self-correction (faculty) | External-oracle cascade, never self-prompting (paper §3 honest negative, C12) | Cost-ordered oracle chain (types → impacted tests → independent reviewer); verdict requires ≥ 2 signals external to fθ | [06](./06-faculties-and-mechanisms.md) §4 | +| 5 | Impact-awareness (faculty) | Atlas hardened; pre-edit gate becomes **mandatory** (hook-enforced) | Incremental dep graph keyed by file content hash; reverse-edge index → O(deg) "who depends on X" | [06](./06-faculties-and-mechanisms.md) §1 | +| 6 | M1 routing | Shipped; add auditable per-task rubric surface + outcome-calibrated weights | Additive transparent rubric; escalation only on verified failure; online weight calibration from ledger | [06](./06-faculties-and-mechanisms.md) §7 | +| 7 | M2 assumption gate | Shipped; questions become **computed missing-sets**, not pattern matches | `s(x) < τ` halt rule + set difference `R(edit) \ covered(selection)` as the question generator | [04](./04-context-assembly.md) §4 | +| 8 | M3 decomposition | Automatic partition-boundary detection (the paper's flagged residue, §5.3) | Constrained graph partition on the task-dependency graph: greedy modularity / min-cut, each part's working set ≤ window budget | [06](./06-faculties-and-mechanisms.md) §3 | +| 9 | M4 goal-anchoring | Continuous drift control, not a static anchor (§5.4) | Drift `D(y_t,g) = 1 − sim(goal, rolling summary)`; **CUSUM control chart** triggers mandatory re-anchor | [06](./06-faculties-and-mechanisms.md) §5 | +| 10 | M5 anti-over-engineering | `src/lean.js` footprint made a defined metric (§5.5) | `φ(y) − φ*(x)` over files/abstractions/LOC; MDL tie-break: smallest description that passes the oracle | [06](./06-faculties-and-mechanisms.md) §6 · [07](./07-ui-quality-gate.md) for UI | +| 11 | M6 inline verification | Checkpoint scheduling during generation (§5.6) | Optimal-stopping threshold: check when hazard × tokens-at-risk > check cost → deterministic cadence per tier | [06](./06-faculties-and-mechanisms.md) §6 | + +## 3. Phase roadmap + +Phases are dependency-ordered; each has an acceptance gate. P1 is the keystone — every +later phase stores its state as PCM claims. + +| Phase | Delivers | Depends on | Acceptance | +|---|---|---|---| +| **P0** (this PR) | Specs 00–08, ADR-0005, ADR-0006, ROADMAP update | — | Docs merged; referenced paths resolve; no source changes | +| **P1 Ledger core** | `src/ledger.js` (claim store, canonical hashing, Beta confidence, Eq. 3 retrieval, decay/prune); migrate `src/lessons.js` + `src/lessons_store.js` + `src/recall.js` onto claim kinds | P0 | All existing cortex/recall tests green on the new store; property tests: id stability, decay monotonicity | +| **P2 Team sync** | `.forge/ledger/` git layout, union-merge driver, `forge ledger merge\|verify\|blame` | P1 | Three-way merge fuzz: any interleaving of two ledgers converges byte-identically (semilattice test) | +| **P3 Reuse cache** | `forge reuse` — fingerprint, exact/near lookup, atlas revalidation, eviction | P1 | Cache hit returns artifact + evidence; stale-interface artifact refused; hit/miss metrics emitted | +| **P4 Context assembly** | `forge context` — candidate scoring, knapsack selection, required-set completeness gate; wired into `src/substrate.js` + hooks | P1 | Gate emits computed missing-set on incomplete context; token budget never exceeded | +| **P5 Loop closure** | Outcome write-back band; doom-loop diagnosis + escalation; imagination dry-run; M3/M4/M5/M6 extensions | P1, P4 | Revert/test outcomes visibly move `val` of informing claims; repeated failure signature halts with a diagnosis claim | +| **P6 UI quality gate** | `forge uicheck` v2: design fingerprints, slop distance, scale conformance; machine-readable taste constraints | P1 | Known-template fixture flagged; project-conformant fixture passes; zero LLM calls in the gate | +| **P7 Dashboard** | `forge dash` — local server + self-contained HTML over `.forge/` stores | P1–P6 (reads their stores) | Renders ledger, cost meter, cache rate, blast radius offline | +| **P8 Evaluation** | Extend `src/eval.js`: cost-stage measurement, cache-hit-rate harness, honest report | P3–P5 | A measured (not asserted) end-to-end cost figure per stage, published in reports/ | + +## 4. Honesty register (the paper's own discipline, applied to this plan) + +- **Measured:** 62 % routing saving (paper §9, live tokens); atlas recall/precision method + (paper §8). Everything else in [05-cost-model.md](./05-cost-model.md) is a **target** until + P8 measures it — the ~90 % figure is a composition argument, not a result. +- **Solved-elsewhere, not rebuilt:** subagent orchestration (M3's mechanics), model + gateways (M1's plumbing) — per paper §10 "do not rebuild". +- **Research-edge, shipped as advisory first:** consolidation quality (ʿilm→fahm promotion), + M4 drift similarity, M6 hazard estimates. They enter as advisory signals and only become + blocking gates once P8 gives them fixtures — the same guard-over-prose discipline as + ADR-0003. diff --git a/docs/plans/substrate-v2/01-pcm-protocol.md b/docs/plans/substrate-v2/01-pcm-protocol.md new file mode 100644 index 0000000..1c7adf5 --- /dev/null +++ b/docs/plans/substrate-v2/01-pcm-protocol.md @@ -0,0 +1,193 @@ +# 01 — The Proof-Carrying Memory (PCM) protocol + +> Normative spec for the claim ledger every substrate-v2 subsystem stores its state in. +> Extends the paper's Eq. 3 `val` term ("validity-anchored memory", §7.1, §11) into a full +> storage / trust / merge protocol. Implemented in P1 as `src/ledger.js`. + +## 0. Design axioms + +1. **Confidence is earned, never asserted.** Only *independent oracles* — test runs, + typecheck, CI, a human accept or revert — may move a claim's confidence. The model's + self-assessment never does (paper C12: LLM evaluators favor their own generations; + §7.2's write-back rule). This is already cortex's invariant in `src/lessons.js` + ("a lesson is trusted because it keeps being RE-CONFIRMED"); PCM generalizes it to + every stored thing. +2. **Every claim carries its proof.** A claim is inseparable from its evidence set — + provenance, oracle outcomes, artifact references (commit SHA, test-run id). A consumer + can always ask *why should I believe this?* and get a machine-readable answer. (The + name is a deliberate riff on proof-carrying code: the artifact travels with the reason + to trust it.) +3. **Merge must be conflict-free by construction.** Team memory dies the day two + teammates' stores need manual conflict resolution. All mutable state is therefore + modeled as grow-only sets whose merge is set union — a join-semilattice + (see [02-team-memory.md](./02-team-memory.md)). +4. **Forgetting is a formula.** The paper's ḥifẓ/murājaʿa policy (§7.2) — retention + through review, decay without it — is implemented as time-decayed evidence weight, + not a cron job that deletes old files. + +## 1. The claim + +``` +claim := { + v: 1, // protocol version + id: sha256(canonical(kind, body, scope)), + kind: "lesson" | "fact" | "artifact" | "edge" | "fingerprint" + | "diagnosis" | "decision" | "summary" | "outcome", + body: , // the content itself + scope: { repo?, dir?, symbol? }, // where it applies (mirrors SCOPE_WEIGHT in lessons.js) + provenance: { author, agent?, model?, tier?, task? , t }, + evidence: [ outcome... ], // grow-only, see §3 + tombstone?: { author, reason, t }, // retraction marker (grow-only too) +} +``` + +- `id` is content-addressed over `(kind, body, scope)` **only** — not provenance, not + evidence — so two teammates who independently learn the same lesson produce the *same + claim id* and their evidence merges instead of duplicating. +- **Canonicalization:** JSON with lexicographically sorted keys, no insignificant + whitespace, NFC-normalized strings, numbers in shortest round-trip form. The canonical + bytes are what is hashed and what is stored. (Stability of `id` under re-serialization + is a P1 property test.) +- `body` schemas per kind are versioned in `source/pcm-kinds.json`; unknown kinds are + preserved verbatim (forward compatibility) but never retrieved. + +### Claim kinds and what they replace + +| kind | body (essentials) | Replaces / powers | +|---|---|---| +| `lesson` | rule text, trigger context | cortex lessons (`src/lessons.js`) | +| `fact` | name, text | recall facts (`src/recall.js`) | +| `artifact` | spec fingerprint, code ref (path@commit or inline), declared interface | reuse cache ([03](./03-reuse-cache.md)) | +| `edge` | from-symbol, to-symbol, relation | atlas graph overlay — verified edges outrank regex guesses | +| `fingerprint` | design-token vector | UI quality gate ([07](./07-ui-quality-gate.md)) | +| `diagnosis` | failure signature, root-cause note, tried-fixes | doom-loop ([06](./06-faculties-and-mechanisms.md) §5) | +| `decision` | choice, alternatives, rationale | ADR-style team decisions, speclock | +| `summary` | source range, compressed text, coverage set | context assembly ([04](./04-context-assembly.md)) | +| `outcome` | oracle, result, refs | the write-back band itself (§6) | + +## 2. The oracle taxonomy + +An oracle is a source of confirm/contradict signals *external to fθ*. Each has a base +weight `w ∈ (0,1]` — its prior reliability — extending the signal table already shipped +in `src/lessons.js` (`SIGNALS` S1–S7): + +| oracle | example | w | family | +|---|---|---|---| +| `human.revert` | git revert / checkout of the change (S6) | 1.0 | human | +| `human.accept` | PR merged, explicit approval | 0.9 | human | +| `test.run` | impacted test suite pass/fail (S1) | 0.8 | outcome | +| `ci.run` | pipeline result | 0.8 | outcome | +| `typecheck` | tsc/compiler on affected files (S7) | 0.6 | outcome | +| `graph.reval` | artifact interface still resolves in atlas | 0.5 | structural | +| `behavioral` | re-edit/thrash signals (S2, S3) | 0.3 | behavioral | + +Rules inherited from cortex, now protocol law: **injection/retrieval is never +confirmation**; a `behavioral` signal can never move confidence alone (needs a second +family — the noisy-OR cross-family gate of `scoreMistake()`). + +## 3. Evidence and confidence + +``` +outcome := { oracle, result: "confirm" | "contradict", w, ref, author, t } +``` + +`ref` points at a verifiable artifact (commit SHA, test-run id, CI URL) — evidence +without a ref is rejected at append time. Evidence records are immutable and +content-hashed; the evidence *set* is grow-only. + +**Confidence** is a Beta posterior with exponential time decay. With prior `Beta(1,1)`, +half-life `T` (default 45 days), `λ = ½`, and `Δt` the age of each outcome: + +``` +val(c) = (1 + Σ_confirms wᵢ·λ^(Δtᵢ/T)) / (2 + Σ_all wⱼ·λ^(Δtⱼ/T)) +``` + +Properties (each a P1 unit test): + +- A fresh claim has `val = 0.5` — exactly `newLesson()`'s starting confidence today. +- Unreviewed claims decay **toward 0.5** (the prior), not toward 0: an old, once-confirmed + claim becomes *uncertain*, not *false*. This is murājaʿa — review restores weight. +- Contradictions weigh in the denominator only, so wrong claims die faster than right + ones grow (the asymmetry `src/lessons.js` already encodes). +- `val` is a pure function of the evidence set ⇒ identical after any merge order (needed + for CRDT convergence in [02](./02-team-memory.md)). + +**Pruning:** a claim is *dormant* below `val < 0.35` (never retrieved, kept for audit) +and *pruned* (moved to `ledger/attic/`) when dormant for > 2·T with no new evidence, or +when tombstoned. Nothing is silently deleted — the attic is the audit trail (paper §5, +17:36: "you will be questioned about all these"). + +## 4. Retrieval — paper Eq. 3, implemented + +``` +score(x, c) = σ( a·rel(x,c) + b·rec(c) + g·val(c) ) // paper §7.1 Eq. 3 +``` + +- `rel(x,c)` — cheap path: Jaccard similarity of MinHash sketches (k = 128 hashes, + 4-token shingles over normalized text; sketches stored on the claim, so comparison is + O(k)). Optional dep path (ADR-0005): pluggable embedding backend behind the same + interface; falls back to MinHash offline. +- `rec(c) = λ^(Δt/T)` — same decay clock as confidence. +- `val(c)` — §3 above. **This term is the paper's load-bearing addition** — memories + pruned by ground truth, not by the model's say-so. +- Default weights `a = 0.55, b = 0.15, g = 0.30`; stored in `source/substrate.json`, + calibrated in P8 by logistic regression on retrieval-outcome pairs (did an injected + claim get confirmed or contradicted downstream?). +- Scope multiplier: reuse `SCOPE_WEIGHT` (symbol 1.0 > dir 0.8 > repo 0.6 > global 0.4). + +## 5. The three layers (ʿilm → fahm → ḥikma, paper §5 & §7.2) + +A flat store collapses fact and understanding; the paper demands layers. In PCM the +layers are **claim kinds plus a promotion rule**, not separate databases: + +| layer | claims | produced by | +|---|---|---| +| **ʿilm** (raw knowledge) | `outcome`, `fact`, `edge`, episode-grade `lesson` | hooks, oracles, direct capture | +| **fahm** (patterns) | consolidated `lesson`, `artifact`, `fingerprint`, `summary` | **consolidation** (below) | +| **ḥikma** (decision support) | `decision`, `diagnosis`, calibrated rubric weights | promotion + human ratification | + +**Consolidation (the murājaʿa job, runs at session end — extends +`src/cortex_distill.js`):** + +1. Cluster ʿilm-layer claims by near-duplicate similarity: union-find over pairs with + Jaccard(sketch) ≥ τ_c (default 0.7). Union-find gives O(n α(n)) clustering without a + pairwise matrix blowup because candidate pairs come from MinHash banding (LSH). +2. Each cluster with ≥ 3 members and mean `val` ≥ 0.6 is distilled into one fahm-layer + claim whose evidence set is the **union** of members' evidence (id changes because the + body changed; member claims get a `consolidated-into` pointer and go dormant). +3. Fahm claims that survive ≥ 2·T with `val` ≥ 0.8 and a human `decision` reference are + eligible for ḥikma promotion — surfaced in `forge dash` for one-click ratification, + never auto-promoted (stewardship boundary, 33:72). + +## 6. The write-back band (paper Eq. 2, §7.2) — learning without touching θ + +The substrate becomes `(y_t, M_{t+1}) = F(x_t, M_t; fθ)` only if outcomes actually flow +back into `M`. Protocol: + +1. **Attribution at injection:** whenever the substrate injects claims into context + (retrieval, cache hit, context assembly), it records the set `informed(action) = + {claim ids}` in the session trace (`.forge/trace/`). +2. **Outcome capture:** existing guards (`session-learner.sh`, `cortex_hook.js`) plus new + CI/test adapters emit `outcome` claims with refs. +3. **Propagation:** each outcome appends evidence to every claim in `informed(action)` — + confirm on success signals, contradict on revert/fail — weighted by the oracle table. + *This is the missing loop:* today a revert teaches the lesson store; after P5 it also + demotes the cached artifact that was reused, the summary that hid the relevant detail, + and the edge that mispredicted impact. +4. **Parametric channel** (paper §7.2 channel 2 — LoRA distillation) stays **out of + scope**: deliberate, offline, and not needed for any P1–P8 gate. Recorded here so the + omission is a decision, not a gap. + +## 7. Module plan (P1) + +- `src/ledger.js` — pure core: canonicalize, hash, append-evidence, `val()`, `score()`, + MinHash sketch, consolidation clustering. No fs. (Mirrors the `lessons.js` / + `lessons_store.js` purity split.) +- `src/ledger_store.js` — file layout + atomic writes ([02](./02-team-memory.md) owns the + on-disk format). +- Migration: `lessons` and `recall` become thin adapters minting `lesson`/`fact` claims; + their public CLI (`forge cortex`, `forge remember`, `forge recall`) is unchanged; their + existing tests must pass unmodified against the adapter (the P1 acceptance gate). + `SECRET_RE` refusal moves down into `ledger.js` so *no* claim kind can store a secret. +- Optional index (ADR-0005): SQLite mirror for O(log n) retrieval on big ledgers; + JSON-file scan remains the reference implementation and offline fallback. diff --git a/docs/plans/substrate-v2/02-team-memory.md b/docs/plans/substrate-v2/02-team-memory.md new file mode 100644 index 0000000..beb090f --- /dev/null +++ b/docs/plans/substrate-v2/02-team-memory.md @@ -0,0 +1,103 @@ +# 02 — Team & shared memory: a git-native CRDT ledger + +> How teammates share one substrate memory with zero sync infrastructure and zero merge +> conflicts. Implements the paper's Memory faculty residual gap — "no standard, +> tool-agnostic, durable memory layer that persists project knowledge … across sessions, +> tools, and **teammates**" (ecosystem map, Memory row). Phase P2. + +## 0. Why git, why CRDT + +Teams already share exactly one replicated, offline-capable, access-controlled store: the +repository. Any bespoke memory server re-solves auth, backup, and offline for worse +adoption. So the ledger lives *in the repo*, and the only hard problem left is merging — +which we make impossible to get wrong by choosing state that merges as **set union**. + +**The math.** Let a ledger state be `S = (C, E, D)`: the set of claims, the set of +evidence records, the set of tombstones — all grow-only, all content-addressed. Define +merge `S₁ ⊔ S₂ = (C₁∪C₂, E₁∪E₂, D₁∪D₂)`. Since `(P(X), ∪)` is a join-semilattice and a +product of semilattices is a semilattice, `⊔` is **commutative, associative, and +idempotent**. Therefore any two replicas that have seen the same updates converge to the +same state regardless of merge order or repetition — the standard state-based CRDT +(CvRDT) convergence argument. Derived values (`val`, retrieval scores) are pure functions +of `S`, so they converge too. There is nothing to resolve, ever. + +What makes this legal is a PCM protocol rule ([01](./01-pcm-protocol.md)): claims are +**immutable by id** and every mutable thing (evidence, tombstones, consolidation +pointers) is an add-only record. "Update" = append; "delete" = tombstone. + +## 1. On-disk layout + +``` +.forge/ledger/ + claims//.json # canonical claim body+provenance, immutable + evidence/.log # one canonical-JSON outcome per line, append-only + attic/ # pruned claims (audit trail, not retrieved) + LEDGER.md # generated human index (like recall's MEMORY.md) +``` + +- One **file per claim**: two teammates minting different claims touch different paths — + git merges trivially. +- Same claim independently minted by both (same content ⇒ same id ⇒ same path, byte-equal + canonical file): identical content merges clean. +- **Evidence logs** are the only files two people append to concurrently. Git's default + line-merge already unions non-overlapping appends; we declare it explicitly: + +``` +# .gitattributes (emitted by forge init) +.forge/ledger/evidence/*.log merge=union +``` + + Line order may differ between replicas — harmless, because `val` is order-independent + (a sum) and lines are deduplicated by content hash on read. `forge ledger verify` + re-canonicalizes: sorts by `(t, hash)`, drops duplicate lines, checks every line parses + and every ref format is valid. Run as a pre-commit guard (`global/guards/`) so the + committed state is always normal-form. + +## 2. CLI surface (P2) + +| command | does | +|---|---| +| `forge ledger merge` | semilattice merge of two ledger trees (used as a git merge driver for pathological cases, and for importing a ledger from another repo/branch) | +| `forge ledger verify` | normal-form check + schema + secret scan; CI-friendly exit code | +| `forge ledger blame ` | provenance + evidence history of a claim — who learned it, what confirmed it | +| `forge ledger stats` | counts by kind/layer, val distribution, decay report | + +## 3. Trust, attribution, and scope + +- `provenance.author` (git identity) rides on every claim and every evidence record. +- **Per-author trust weight** `u(author) ∈ [0.5, 1]` multiplies evidence weight `w`. + It is itself computed from the ledger: the historical confirm rate of that author's + claims (Beta posterior again — authors whose claims keep being contradicted contribute + less). Bootstrap value 1.0; floor 0.5 so no teammate is silenced. +- Sharing boundary: `.forge/ledger/` is repo-scoped and travels with the repo. The + existing global store (`~/.forge`) remains personal; `forge brain` continues to inline + a *curated* subset into AGENTS.md for agents that read nothing else. Promotion from + personal → repo ledger is an explicit `forge remember --share` act, never automatic. +- **Secrets:** `SECRET_RE` refusal (now in `ledger.js`) applies at mint *and* + `forge ledger verify` re-scans on commit — defense in depth matching + `secret-redact.sh`. + +## 4. Retraction and disputes + +- Retraction = tombstone claim (grow-only set `D`): the claim stops being retrieved + everywhere after sync, but history remains auditable in the attic. +- Disagreement between teammates is *not* a merge conflict — it is two evidence streams + on one claim. If Alice's tests confirm what Bob's revert contradicts, `val` settles + near the weighted evidence balance and the claim shows as **contested** in + `forge dash` (val ∈ [0.4, 0.6] with ≥ 1 contradiction), prompting a human `decision` + claim — the stewardship boundary, not an algorithmic override. + +## 5. Failure modes considered + +| risk | mitigation | +|---|---| +| Ledger bloat in git history | claims are small canonical JSON; evidence logs compact; attic prunable via `git rm` without breaking ids (content-addressed, no positional refs) | +| Two authors, same lesson, different wording | MinHash consolidation ([01](./01-pcm-protocol.md) §5) clusters them; union evidence | +| Malicious/poisoned claim from a fork | evidence requires verifiable refs; `val` starts at 0.5 and only local oracles raise it; `forge scan`-style provenance gating for imported ledgers | +| Monorepo scale | id-prefix sharding (256 dirs); optional SQLite read index (ADR-0005) | + +## 6. Explicit non-goals (P2) + +No sync server, no real-time channel, no cross-repo global team memory — git push/pull +*is* the sync. If a later phase wants real-time, the CRDT property means a dumb +last-writer-wins file relay suffices; nothing in this design would change. diff --git a/docs/plans/substrate-v2/03-reuse-cache.md b/docs/plans/substrate-v2/03-reuse-cache.md new file mode 100644 index 0000000..f7c9a93 --- /dev/null +++ b/docs/plans/substrate-v2/03-reuse-cache.md @@ -0,0 +1,108 @@ +# 03 — The proof-carrying reuse cache + +> "Reuse already-generated code" as a deterministic system, not a skill exhortation. +> Turns `global/tools/reuse-first/` from prose into the advisory layer over a real cache. +> Addresses the duplication trend the paper cites (C4 — direction confirmed, multiplier +> disputed) and is the largest single lever in the cost model +> ([05](./05-cost-model.md)). Phase P3. + +## 0. The idea + +Every artifact the agent generates and *verifies* (tests passed, human accepted) becomes +an `artifact` claim in the ledger — code that travels with its proof. Before generating +anything, the substrate asks the cache: *have we, or any teammate, already built this?* +A hit returns the artifact **with its evidence attached** and skips generation entirely; +via the team ledger ([02](./02-team-memory.md)), one teammate's verified work is +everyone's cache hit. + +Why this beats naive LLM response caching: responses are keyed to prompts (brittle, +provider-side, trustless). PCM artifacts are keyed to *normalized task specs + codebase +context*, survive rewording (near-match), and are only served while their proof still +holds (revalidation + `val` floor). A stale or discredited artifact silently stops being +served — the cache prunes itself by ground truth. + +## 1. Artifact claim body + +``` +artifact.body := { + spec: normalized task specification text, + sketch: MinHash sketch of spec (for near-match), + slice: sha256 of the atlas graph slice the artifact touches, // context key + interface: [ exported symbols + signatures the artifact declares ], + deps: [ symbols the artifact requires from the codebase ], + code: { path, commit } | { inline }, // committed code by ref, snippets inline + lang, kind: "function" | "module" | "component" | "config" | "test", +} +``` + +## 2. Fingerprinting + +**Normalization** strips volatility so the same task fingerprints identically across +sessions and teammates: lowercase, collapse whitespace, drop punctuation-only tokens, +replace literal identifiers/paths/numbers with typed placeholders (`⟨ident⟩`, `⟨path⟩`, +`⟨num⟩`), sort simple constraint clauses. (Deterministic, pure — a P3 unit-test surface; +same spirit as `src/preflight.js`'s lexical feature extraction.) + +**Two keys per artifact:** + +- `exact = sha256(normalized spec ‖ slice)` — O(1) lookup for the literal repeat. +- `sketch = MinHash_k(shingles₄(normalized spec))`, k = 128 — near-match. `E[|sketch + match|/k] = Jaccard(A,B)`, so sketch agreement is an unbiased Jaccard estimator with + standard error `≈ √(J(1−J)/k)` ≤ 0.045 — accurate enough to threshold at τ = 0.8. + LSH banding (16 bands × 8 rows) finds candidates without scanning: collision + probability `1−(1−J⁸)¹⁶` — ≈ 0.96 at J = 0.8, ≈ 0.17 at J = 0.5 — a sharp cliff + exactly where we want it. + +## 3. The lookup ladder + +``` +reuse(x): + 1. exact hit (same spec, same slice) → serve, cost ≈ 0 + 2. near hit (Jaccard ≥ 0.8, compatible slice) → REVALIDATE, then serve-with-diff + 3. adapt hit (Jaccard ≥ 0.6) → inject artifact as context ("start + from this verified code"), generate + the delta only — cheaper prompt, + strong anchor against re-invention + 4. miss → generate; on verification, mint + the artifact claim (cache fill) +``` + +**Serving conditions (all three, always):** + +1. `val(artifact) ≥ 0.6` — the proof-carrying floor; contradicted artifacts stop serving + automatically via the write-back band. +2. **Revalidation against the current atlas:** every symbol in `deps` still resolves + (`atlas.has()`), and nothing in the current graph shadows `interface`. A cache serving + code whose dependencies vanished is worse than a miss. Structural revalidation emits a + `graph.reval` outcome (w = 0.5) — so even serving keeps evidence fresh. +3. Secret scan on serve (paranoia; artifacts were already scanned at mint). + +**After serving:** the reuse event enters `informed(action)` — if the reused code is then +reverted, the artifact's `val` drops (write-back band, [01](./01-pcm-protocol.md) §6). +The cache is thus self-correcting end-to-end: filled by verification, served by proof, +demoted by outcomes, evicted by decay. + +**Eviction** = the ledger's standard policy (dormant < 0.35, attic after 2·T) — no +separate LRU machinery. + +## 4. Where it hooks in + +- `substrateCheck()` (`src/substrate.js`) gains a reuse stage between preflight and + route: a hit short-circuits generation; an adapt-hit rewrites the prompt. +- `forge reuse query|stats|mint` CLI for inspection and manual seeding. +- The `reuse-first` skill is rewritten to *call* the cache and explain its answer — + advisory prose backed by a deterministic lookup (guard-over-prose, ADR-0003). +- Metrics per lookup (`hit_exact | hit_near | hit_adapt | miss`, tokens saved estimate) + → `.forge/metrics.jsonl` → [05-cost-model.md](./05-cost-model.md) and `forge dash`. + +## 5. Honest limits + +- Near-match on MinHash is lexical: "sort users by signup date" vs "order accounts by + registration time" may score < 0.8 and fall to adapt-tier or miss. The optional + embedding backend (ADR-0005) raises near-hit recall; MinHash stays the offline floor. +- The `slice` key makes hits context-sensitive: a utility generated for one dependency + neighborhood won't exact-hit in a different one (by design — that's how stale context + is excluded), which caps the hit rate on fast-moving code. Adapt-tier exists precisely + to salvage value there. +- Cache hit *rate* is workload-dependent; the 90 % cost story only cites what P8 + measures. diff --git a/docs/plans/substrate-v2/04-context-assembly.md b/docs/plans/substrate-v2/04-context-assembly.md new file mode 100644 index 0000000..a0a6fad --- /dev/null +++ b/docs/plans/substrate-v2/04-context-assembly.md @@ -0,0 +1,131 @@ +# 04 — Context assembly: knapsack selection + a set-cover completeness gate + +> Fixes the "incomplete context" failure from the root: what goes into the window becomes +> a budgeted optimization, and *whether the context is sufficient* becomes a computed +> set, not a feeling. Supplies the paper's P3 remedy at the system level and gives M2's +> questions a derivation. Phase P4, wired into `src/substrate.js`. + +## 0. The two failures this kills + +1. **Over-stuffing:** everything competes for the window on equal terms (paper P3 — + "bounded, undifferentiated context"); MCP setups alone can eat most of it before work + begins (C11, vendor-reported). Result: the relevant fact is present but drowned. +2. **Under-supplying:** the agent edits a symbol without the callers, the test, or the + team lesson that would have changed the plan — then "assumes" (M2's root failure, + C1/C2). Today nothing *computes* what was missing; the user finds out from the bug. + +Both are the same problem: context selection has no objective function and no +sufficiency predicate. We give it both. + +## 1. Candidate items + +For a task `x` targeting symbols `S` (parsed by `src/scope.js` + atlas query), gather +candidates `I`: + +| source | items | +|---|---| +| atlas slice | definitions + reverse-dependency neighborhood of `S` (`impact()` output) | +| ledger | top-scoring claims per Eq. 3 (`lesson`, `fact`, `decision`, `diagnosis` in scope) | +| reuse cache | adapt-tier artifacts ([03](./03-reuse-cache.md)) | +| files | source spans for `S` and its direct callers; relevant test files | +| summaries | `summary` claims covering cold ranges (below) | + +Each item `i` carries: token cost `tᵢ` (measured, not guessed — `str.length/3.6` +heuristic calibrated in P8), score `sᵢ`, and a **coverage set** `cov(i)` — which required +entities it satisfies. + +## 2. Selection — a budgeted knapsack + +Given window budget `B` (per-tool cap from `source/substrate.json`, minus fixed +overhead): + +``` +maximize Σ sᵢ·xᵢ +subject to Σ tᵢ·xᵢ ≤ B, xᵢ ∈ {0,1} +``` + +`sᵢ` = Eq. 3 score ([01](./01-pcm-protocol.md) §4) × source prior (atlas slice and +required files outrank nice-to-have lessons). Solved greedily by density `sᵢ/tᵢ` with the +best-single-item fallback — the classic ½-approximation to 0/1 knapsack; optimality is +not the point, *having an objective* is. Deterministic, O(n log n), zero model calls. + +Two refinements: + +- **Diminishing returns within a source:** the marginal value of the 4th lesson is below + the 1st. Score the j-th item taken from one source as `sᵢ·δ^(j−1)` (δ = 0.7) — a lazy + submodular discount evaluated during the greedy pass. +- **Compression ladder:** every file/graph item exists at up to three granularities — + full span, signature-only, one-line `summary` claim — with costs `t⁽⁰⁾ > t⁽¹⁾ > t⁽²⁾`. + The greedy pass may *downgrade* an item's granularity instead of dropping it, provided + the granularity still covers the entities the gate needs (below). This is the paper's + "context compression" made explicit: compression is a *lossy move with a known coverage + cost*, chosen by the optimizer, never silently by scroll-off. + +## 3. The completeness gate — set cover, inverted + +Define the **required-knowledge set** for an edit — computed, not vibes: + +``` +R(edit) = defs(S) // the symbols being changed + ∪ blast₁(S) // direct dependents (atlas reverse edges, hop 1) + ∪ tests(S) // tests covering S (atlas contains/test edges) + ∪ contracts(S) // types/interfaces S implements + ∪ lessons*(S) // scope-matching lessons with val ≥ 0.8 +``` + +The gate over a selection `X`: + +``` +missing = R(edit) \ ⋃_{i∈X} cov(i) +gate: missing = ∅ → proceed | missing ≠ ∅ → resolve(missing) +``` + +`resolve(missing)`, in order: + +1. **Auto-fetch** — entity exists in repo/ledger: add it (re-running the knapsack with it + pinned; something optional gets downgraded/dropped to make room). +2. **Ask** — entity is unknowable from the repo (an unstated requirement, an ambiguous + target): emit it as an M2 clarifying question. **This upgrades `src/preflight.js`:** + questions are now *derived from a set difference* — "I cannot see the callers of + `validatePayment` in `billing/`; is the EU flow in scope?" — instead of + pattern-matched from lexical cues. The paper's `s(x) < τ` halt rule keeps its + role for lexical under-specification; the missing-set adds structural + under-supply. +3. **Block** — under `FORGE_ENFORCE=1`, an edit whose `R` was never covered is refused + the same way the impact gate refuses (17:36 — no action without knowledge). + +Coverage bookkeeping is exact for graph/file items (`cov` = the entities the span +contains, from atlas) and declared for claims (`scope`). `R ⊆ cov(X)` is a set inclusion +check — O(|R|) with a hash set. + +## 4. Session dynamics + +- **Re-assembly triggers:** target set `S` changes; M4 drift alarm fires + ([06](./06-faculties-and-mechanisms.md) §5); context usage crosses 70 % (pre-empting + the "compression wipes the anchor" decay the paper documents in M4). On re-assembly, + the goal claim and `R`-covering items are pinned — fluff is what gets evicted, never + the anchor. +- **Write-back:** the selection set is recorded in `informed(action)`; if the task + outcome contradicts (revert/fail), the *selection itself* is evidence — items that were + present get nothing, but a post-hoc diagnosis that names an entity absent from `cov(X)` + mints a `diagnosis` claim that grows `R`'s recipe for that scope. The gate learns what + "complete" means per codebase. + +## 5. CLI + integration + +- `forge context ` — prints the assembled selection: items, granularity, tokens, + scores, covered vs. required entities, and the missing-set if any. `--json` for hooks. +- `substrateCheck()` stage order becomes: preflight(lexical) → **context-assemble + + completeness gate** → reuse → route → impact → lean → verify plan. +- The ambient hook (`substrateContext()`) injects the *rendered selection*, replacing + ad-hoc "read these files" advisories. + +## 6. Honest limits + +- `R(edit)` inherits atlas's regex-approximation: over-approximate on dependents (safe — + more required knowledge), potentially blind where regex misses an edge. The AST-backed + atlas upgrade ([06](./06-faculties-and-mechanisms.md) §1) tightens both at once. +- Greedy ½-approximation can leave budget value on the table; irrelevant next to the + failure mode it replaces (no objective at all). +- `lessons*` selection uses `val`, so a young team ledger under-supplies at first — the + gate's floor (`defs ∪ blast₁ ∪ tests`) is structural and needs no history. diff --git a/docs/plans/substrate-v2/05-cost-model.md b/docs/plans/substrate-v2/05-cost-model.md new file mode 100644 index 0000000..5c0aadc --- /dev/null +++ b/docs/plans/substrate-v2/05-cost-model.md @@ -0,0 +1,90 @@ +# 05 — The cost model: the path to ~90 %, stated honestly + +> The owner's target is ~90 % cost reduction. This doc gives the composition argument for +> why the target is reachable, the exact formula, which factors are **measured** today +> versus **targets**, and the P8 harness that will replace targets with measurements. +> Discipline per the paper (§4, C6): a number is an assumption until measured. + +## 1. The stage model + +A task's cost passes through independent multiplicative stages: + +``` +C = C₀ · (1 − g·h_gate) · (1 − h_cache·σ_cache) · (1 − ρ_ctx) · (1 − r_route) +``` + +| factor | stage | meaning | status | +|---|---|---|---| +| `h_gate` | M2 gate | fraction of requests halted as under-specified (spend ≈ 0 generation tokens; `g` ≈ their share of would-have-been spend) | mechanism **measured** in paper §9 (9/9 halts, zero gen tokens); rate is workload-dependent | +| `h_cache` | reuse ([03](./03-reuse-cache.md)) | hit rate; `σ_cache` = avg saving per hit (≈ 1.0 exact, ≈ 0.85 near, ≈ 0.5 adapt) | **target** — P8 measures | +| `ρ_ctx` | assembly ([04](./04-context-assembly.md)) | input-token reduction from knapsack + compression ladder vs. read-everything baseline | **target** — P8 measures | +| `r_route` | M1 routing | tier selection saving on remaining generation | **0.62 measured live** (paper §9, real tokens, real ladder) | + +Secondary effects deliberately **excluded** from the multiplication (they'd double-count +or are unpriceable now): doom-loop halts (avoided thrash loops), M5 lean (fewer generated +tokens), avoided-rework from the completeness gate (C2's "almost right" loop). These are +tracked in P8 as separate counters, reported alongside — upside, not arithmetic. + +## 2. What the target requires + +With routing fixed at its measured 0.62, reaching 90 % total requires the *other* stages +to jointly remove ≈ 74 % of the remaining cost: + +``` +(1 − h_cache·σ) · (1 − ρ_ctx) · (1 − g·h_gate) ≤ 0.10 / 0.38 ≈ 0.263 +``` + +Example compositions that satisfy it: + +| scenario | h_cache·σ | ρ_ctx | g·h_gate | total reduction | +|---|---|---|---|---| +| repeat-heavy team (CRUD, components, migrations) | 0.55 | 0.35 | 0.10 | **90.2 %** | +| moderate reuse | 0.40 | 0.30 | 0.10 | 85.6 % | +| cold start (fresh repo, empty ledger) | 0.05 | 0.25 | 0.05 | 74.3 % | + +Read the table honestly: **~90 % is credible on repeat-heavy team workloads once the +ledger is warm, and is not credible cold.** The cache factor dominates, and it *grows* +with team adoption (every teammate's verified artifact is everyone's hit — +[02](./02-team-memory.md)) and with time (ledger accumulation). The floor — routing + +assembly + gate alone — is ≈ 75 %, already substantial. + +## 3. Measurement plan (P8) + +**Instrumentation** — every stage emits one line to `.forge/metrics.jsonl`: + +``` +{ t, task, stage: "gate|cache|context|route|generate|verify", + tokens_in, tokens_out, tier, outcome, saved_estimate, ref } +``` + +Written by the existing guard layer (`cost-budget.sh` already meters spend; it gains +stage tags), `substrateCheck()`, and the reuse/context modules. `forge cost` learns a +`--stages` report; `forge dash` charts it. + +**Harness** — extend `src/eval.js` (which already does precision/recall for impact): + +1. **Replay corpus:** N ≥ 100 real tasks captured from session traces (spec + repo state + ref + outcome), stratified: repeat-heavy / mixed / cold. +2. **Paired runs:** baseline (always-premium, read-everything, no cache) vs. substrate, + same tasks — the paper §9 methodology (identical tokens repriced) extended to all four + stages, so every saving is arithmetic on measured tokens, never an estimate. +3. **Correctness guard:** a saving only counts if the external verifier passes the output + (paper §9.3's rule: "routing down only counts as a win if the cheap tier is still + correct" — applied to every stage; a cache hit that gets reverted is a *negative* + entry). +4. **Report:** per-stage factors with confidence intervals → `reports/cost-eval.md`; + the README claim gets updated to whatever the harness measured, with the workload + caveat attached. Until then the README may say "62 % measured (routing); ~90 % + composed target" — never "90 % achieved". + +## 4. Cost of the substrate itself + +The overhead side of the ledger, counted against the savings in P8: + +- Deterministic stages (gate, knapsack, cache lookup, atlas query) are CPU-cheap and + token-free — the paper's atlas figures (1.9k-node graph in 91 ms, sub-ms queries) + bound the latency class. +- Injected context (claims, blast radius, checkpoints) *spends* tokens to save tokens; + the assembly budget `B` caps it structurally, and `ρ_ctx` is measured net of it. +- The opt-in LLM adjudication layer (`FORGE_LLM=1`) is priced per call in + `model_tiers.json` and appears as its own metrics stage. diff --git a/docs/plans/substrate-v2/06-faculties-and-mechanisms.md b/docs/plans/substrate-v2/06-faculties-and-mechanisms.md new file mode 100644 index 0000000..1d28cb3 --- /dev/null +++ b/docs/plans/substrate-v2/06-faculties-and-mechanisms.md @@ -0,0 +1,161 @@ +# 06 — Faculties & mechanisms: the algorithms for everything else + +> Per-capability specs for the paper points not owned by docs 01–05: impact hardening, +> imagination, self-correction, M3 decomposition, M4 drift, M5 lean, M6 inline +> verification, M1 calibration, doom-loop diagnosis. Each names its algorithm, data +> structure, and the `src/` module it extends. Phase P5 (plus the atlas work in P1/P5). + +## §1 Impact-awareness — harden the atlas, make the gate mandatory + +**Today:** `src/atlas.js` builds a regex-approximate symbol graph (honest, conservative); +`impact()` traverses reverse edges with hop-decay; the gate only blocks under +`FORGE_ENFORCE=1`. + +**Plan:** + +- **Graph:** keep the incremental design (nodes keyed by file content hash — unchanged + files never re-parsed, paper §8.1) and add an **AST tier** where a parser is cheap + (TS/JS via the TypeScript compiler API — legal under ADR-0005; Python via `ast` as in + the paper's prototype). Regex tier remains the universal fallback; each edge records + its tier so confidence decay can discount regex edges (`w_edge`: ast 1.0, regex 0.7, + verified-by-outcome 1.2 via `edge` claims — the ledger overlay). +- **Data structure:** adjacency + **reverse-edge index** (symbol → dependents) for O(deg) + blast queries; per-file symbol table for incremental invalidation. +- **Mandatory gate:** the PreToolUse guard blocks file-mutating tool calls whose target + symbols have no computed blast radius this session (ships enforcing by default in P5 — + the paper's opp. #3 is "mandatory, hook-enforced", and ADR-0003 says guards, not + prose). Escape hatch: `FORGE_GATE=advisory` env for opt-out, inverting today's opt-in. +- **Accuracy loop:** each predicted blast radius becomes a claim; post-change test/CI + results confirm or contradict it (write-back band) — `forge eval` gets live + precision/recall over time instead of fixture-only (extends `evalImpact()` in + `src/eval.js`). + +## §2 Imagination — the consequence simulator g (paper Eq. 4) + +The paper: `ĉ = g(a, C)` predicting `{broken call sites, type errors, failing tests}` — +partly exact (static), partly cheap simulation (sandboxed tests). Atlas gives the static +half; P5 adds the simulation half: + +1. **Impacted-test selection.** Build the bipartite cover relation `covers(test, symbol)` + from atlas (`contains` edges of test files + call edges into `S ∪ blast(S)`). Choosing + the cheapest test set covering the blast radius is **weighted set cover** (weight = + measured test duration); greedy gives the classic `ln n`-approximation and is exact on + the small instances that matter. Output: a minimal dry-run suite instead of "run + everything" (minutes → seconds — what makes pre-action simulation affordable at all). +2. **Sandboxed dry-run.** Apply the proposed diff in an ephemeral worktree + (`git worktree` — the same isolation the ecosystem already trusts for fan-out), run + typecheck + the selected suite, discard. Result = a `prediction` outcome: the + *imagined* consequence, now with evidence, attached to the plan before any real edit. +3. **Cost control:** dry-run only when the impact gate ranks the edit risky — + `risk = Σ_{n∈blast} conf(n)·w(kind_n)` above threshold, or blast crosses a package + boundary. Trivial edits skip simulation (M1's proportionality principle). + +This is the paper's open build target ("no general, reusable such component exists", §3 +Imagination) scoped to where code makes it tractable: exact structure + cheap oracle. + +## §3 M3 — automatic decomposition boundaries + +The paper marks M3 solved (subagents, worktrees) *except* "choosing the partition +boundary is still a human heuristic" (§5.3). The residue, formalized: + +- Build the **task-dependency graph**: nodes = subtasks from `src/scope.js` + decomposition, weighted edges = shared working-set mass `w(u,v) = Σ_{e∈R(u)∩R(v)} + tokens(e)` (required-knowledge sets from [04](./04-context-assembly.md) — already + computed). +- Partition to minimize cut weight subject to each part's working set fitting the window + budget: `min Σ_cut w(u,v)` s.t. `tokens(⋃_{u∈Cᵢ} R(u)) ≤ B` — constrained graph + partitioning. At session scale (≤ ~20 subtasks) **greedy modularity merging** + (agglomerate while budget holds, never merge across a zero-density cut) is sufficient; + no METIS needed. +- Output: `forge scope --plan` renders the partition — "fork these 2 subtasks (disjoint + working sets); keep these 3 sequential (78 % shared context)" — consumable by + subagent/worktree tooling. ForgeKit does **not** re-implement orchestration (paper §10: + do not rebuild). + +## §4 Self-correction — the external-oracle cascade + +Formalizes `src/verify.js` + crew reviewers per the paper's honest negative (§3: intrinsic +self-correction unreliable, C12: self-preference): + +- **Cascade, cost-ordered:** typecheck (seconds) → impacted tests (§2's selected suite) → + independent reviewer (`global/crew/independent-reviewer.md`, *different model tier* + than the generator, per C12) → human. +- **Verdict rule:** an action is *verified* only with ≥ 2 confirming signals from + families external to fθ (type/test/human) — the same cross-family gate as + `scoreMistake()`. A lone LLM reviewer approval is advisory, never sufficient. +- Each cascade stage emits `outcome` claims → the write-back band. Escalation to a + costlier stage only on a *verified* failure of the cheaper one (M1's clause). + +## §5 M4 goal-drift + doom-loop root cause (the two "am I still on track?" controls) + +**M4 — CUSUM drift control** (extends `src/anchor.js`): + +- Goal `g` is minted as a pinned claim at task start. Drift signal per checkpoint: + `Dₜ = 1 − sim(g, rolling summary of recent actions)` — sim = MinHash Jaccard (embedding + backend optional), summary from the trace of file targets + stated step intents. +- A raw threshold on `Dₜ` is noisy (single checkpoint can legitimately explore). Use a + **one-sided CUSUM control chart**: `Cₜ = max(0, Cₜ₋₁ + Dₜ − k)`; alarm at `Cₜ > h` + (defaults k = 0.35, h = 1.0 — calibrated in P8 fixtures). CUSUM detects *sustained* + small drift with provably minimal detection delay for a given false-alarm rate + (classical sequential analysis) — exactly the "decaying anchor" failure the paper + documents, caught early. +- Alarm action: re-inject the goal claim, force context re-assembly with the goal pinned + ([04](./04-context-assembly.md) §4), and require an explicit "resume / re-scope" + decision. Never silent. + +**Doom-loop diagnosis** (extends `global/guards/doom-loop.sh`, paper opp. #5): + +- **Failure signature** `= sha256(error class ‖ normalized message ‖ file ‖ symbol)` + (message normalized: strip line numbers, addresses, timestamps). Ring buffer of recent + signatures in the session trace. +- Same signature `k = 3` times ⇒ **thrash**, not progress: halt the retry loop, mint a + `diagnosis` claim (signature, attempted fixes so far, blast context), escalate one + model tier *with the diagnosis as the prompt's head* — the escalation is + diagnosis-carrying, not "try again but more expensive". +- Diagnosis claims are team-shared via the ledger: teammate hits the same signature next + week, the diagnosis is retrieved at score-time (Eq. 3 rel on the signature). The doom + loop becomes a one-per-team event instead of one-per-session. + +## §6 M5 lean + M6 inline verification (the two "how much is enough?" controls) + +**M5 — footprint metric** (extends `src/lean.js`, which already parses diff footprints): + +- Define `φ(y) = (files touched, new exported abstractions, net LOC)` — measured by + `parseDiffFootprint()`. Estimate the task-implied minimum `φ*(x)` from the reuse cache + and ledger: the footprint distribution of *verified* artifacts for tasks in the same + fingerprint neighborhood (median of near-matches). Flag when + `φ(y) ≫ φ*(x)` component-wise (default: any component > 2× the neighborhood median + and above `assessFootprint()`'s absolute floors). +- Tie-break rule, MDL-flavored: between candidate solutions that pass the same oracle, + prefer minimal description length (fewest new abstractions, then fewest LOC). Advisory + in P5; enforcing only for the UI gate ([07](./07-ui-quality-gate.md)) where the slop + correlation is strongest. + +**M6 — checkpoint scheduling** (extends `src/verify.js` + PostToolUse hooks): + +- The question "when to check?" is optimal stopping: insert a checkpoint when the + expected loss of continuing-while-wrong exceeds the check's cost: + `p_err(t) · tokens_at_risk(t) · c_tok > c_check`. With hazard `p_err` per step + estimated from ledger outcome history per tier (haiku ≫ opus), this yields a + **deterministic cadence**: check every `n* = ⌈c_check / (p_err·c_tok·s̄)⌉` meaningful + steps — smaller `n*` for cheaper tiers and riskier blast radii. No magic constants; + every input is measured or priced. +- A "check" = the cheapest cascade stage that can falsify the current step (usually + typecheck on touched files) surfacing a one-line human-checkable claim — the paper's + interpretive checkpoint (§5.6), priced so it can afford to run *during* generation. + +## §7 M1 — outcome-calibrated routing (closing the paper's own caveat) + +Paper §9.3: "a production version would … likely learn the rubric weights rather than +hand-set them." With the ledger, the training signal exists: + +- Every routing decision + external-verifier outcome is an `outcome` claim + (`tier, rubric features, pass/fail/escalated`). +- Periodically fit logistic regression `P(pass | tier, features)` (pure-JS IRLS on a few + hundred rows — no dependency needed) and choose the cheapest tier with + `P(pass) ≥ 0.9`; hand-set rubric weights remain the cold-start prior and the + explanation surface (the rubric stays additive and auditable — transparency is the + feature, per §9.1). +- The auditable per-task surface ships regardless: `forge route --explain` prints every + feature, its points, the tier, and the calibrated `P(pass)` — the "thin transparency + layer" the paper says is the only thing left to build for M1 (§5.1). diff --git a/docs/plans/substrate-v2/07-ui-quality-gate.md b/docs/plans/substrate-v2/07-ui-quality-gate.md new file mode 100644 index 0000000..1a14d27 --- /dev/null +++ b/docs/plans/substrate-v2/07-ui-quality-gate.md @@ -0,0 +1,125 @@ +# 07 — The generated-UI quality gate: custom design, not templates + +> Owner-named pain: AI-generated UI converges on the same generic template — the modal +> design is the max-likelihood completion, the same statistical failure shape as M5 +> over-engineering (paper §5.5: "the maximum-likelihood objective favors the elaborate +> completion"; here it favors the *median* one). ForgeKit's taste layer +> (`global/taste/*.md`, `ui-workflow`, `design-md`) is prose; prose loses to gradients. +> This spec makes taste **measurable and enforceable** — guard-over-prose (ADR-0003) +> applied to design. Phase P6, extending `src/uicheck.js`. + +## 0. Two-sided objective + +Generated UI fails in two directions at once, so the gate scores two distances: + +1. **Too close to the slop centroid** — the generic-AI look (default framework palette, + uniform 8px spacing, one font, gradient hero, uniform rounded-xl cards). +2. **Too far from the project's own design system** — ignores the tokens, scales, and + voice the codebase already has. + +Good output is *far from generic and close to home*. Both are geometric statements once +UI is embedded as a feature vector — so embed it. + +## 1. The design fingerprint (deterministic feature extraction) + +`fingerprint(ui) → v ∈ ℝᵈ`, extracted by parsing CSS/JSX/Tailwind classes — pure static +analysis, zero LLM calls, zero screenshots needed for the core (same discipline as +`uicheck`'s WCAG math): + +| group | features | +|---|---| +| **color** | palette as OKLCH points; palette size; hue histogram (12 bins); chroma mean/max; neutral-to-accent ratio; specific-hex flags for framework defaults | +| **space** | the observed spacing multiset; inferred base unit `u*` (argmin Σ min_k \|s − k·u\| over candidate units — approximate-GCD by residual minimization); scale ratio fit (geometric vs. linear); % of values on-scale | +| **type** | font-stack set; distinct sizes; modular-scale ratio fit `r*` (least-squares on log sizes); weight distribution | +| **shape** | radius multiset + entropy (uniform-radius = template tell); border/shadow token count; shadow-elevation levels | +| **layout** | container widths; grid/flex ratio; density (elements/viewport heuristic); hero/card/section pattern counts | + +Normalization per feature (z-score against the corpus, below) so distances are +scale-free. The **project fingerprint** `v_proj` is extracted the same way from the +repo's existing UI + design tokens (tailwind config, CSS custom properties, theme files) +and stored as a `fingerprint` claim in the ledger — shared with the team, updated by the +same evidence rules as everything else ([01](./01-pcm-protocol.md)). + +## 2. Slop distance and conformance + +Ship a small corpus of **generic-template fingerprints** `Z = {z₁…z_m}` (extracted from +the recognizable default outputs: untouched framework starters, the canonical +AI-landing-page look, default shadcn/Tailwind palettes — a curated fixture set in +`source/slop-corpus.json`, versioned and extensible). Then: + +``` +slop(v) = min_j ‖v − z_j‖₂ (distance to nearest generic centroid) +conform(v) = ‖v − v_proj‖₂ (distance from the project's own system) + +gate: PASS iff slop(v) ≥ τ_s ∧ conform(v) ≤ τ_c +``` + +- Weighted L2 — color and space dominate (they carry the "looks generic" signal); + weights + thresholds live in the taste profile (§3) and get P8 fixtures. +- Greenfield repo (no `v_proj`): conformance term is replaced by the *chosen taste + profile's* target region (§3), so the gate still has a "home" to pull toward. +- Failure output is **actionable, not a score**: the top-k features driving each + violation — "palette is default-Tailwind blue (slop Δ 0.04 on color); spacing is + uniform 8px (scale entropy 0); project uses 1.25-ratio type scale, output uses one + size" — each mapped to a concrete edit. + +## 3. Taste profiles become constraint sets + +Each `global/taste/.md` (brutalist, corporate, editorial, minimalist, playful) +gains a machine-readable sibling `.json`: + +``` +{ "palette": { "max_hues": 3, "chroma": [0.02, 0.12], "neutrals": "warm" }, + "space": { "scale": "geometric", "ratio": [1.4, 1.6], "base": [4, 8] }, + "type": { "scale_ratio": [1.2, 1.333], "max_families": 2 }, + "shape": { "radius_levels": [0, 2], "shadow_levels": [0, 1] }, + "gate": { "tau_slop": 0.35, "tau_conform": 0.5, "feature_weights": {…} } } +``` + +The prose file keeps the *why* and the vocabulary (it still steers generation); the JSON +is what the gate checks (it catches what steering missed). `forge taste` learns +`--check `. + +## 4. Perceptual constraint checks (uicheck v2) + +Deterministic per-file assertions extending `ASSERTABLE_CHECKS` in `src/uicheck.js` +(alongside the existing WCAG contrast math): + +- **Scale conformance:** every spacing/size value on the declared scale within ε + (flags the pixel-soup that reads as "off" without anyone knowing why). +- **Palette bounds:** hue count ≤ max; min pairwise ΔE (OKLCH) between accents — no + five-blues; chroma within profile range. +- **Consistency entropies:** radius/shadow/weight entropy under threshold (a design + system uses few levels, deliberately). +- **Contrast:** existing WCAG checks, unchanged. + +All emit `pass|fail + fix hint`, `--json` for hooks; the PostToolUse guard runs them on +edited UI files (advisory by default, enforcing under `FORGE_ENFORCE=1`, same ladder as +the other gates). + +## 5. The visual loop (the only non-static part) + +Static analysis can't see rendered composition. Under ADR-0005, the +`frontend-verifier` crew agent gets Playwright: + +- Screenshot changed routes/components at 2 viewports; attach to the review. +- The *reviewer* (different model than the generator — C12 discipline, + [06](./06-faculties-and-mechanisms.md) §4) judges against the taste profile's prose + + the fingerprint report, and its accept/reject is an `outcome` claim on the component's + `artifact` — so reused UI components carry visual-review evidence too + ([03](./03-reuse-cache.md)). +- Optional visual regression: pixel-diff against the last accepted screenshot of the same + component (stored ref in the artifact claim), flagging unintended drift. + +## 6. Why this can work (and its limits) + +The mechanism is honest about what it is: **fingerprint geometry catches the measurable +signature of genericness** — the features that make everyone say "that looks +AI-generated" are, concretely, default palettes, flat spacing, single-size type, uniform +radii, and those are all extractable. It cannot measure *beauty*; it measures *distance +from generic* and *distance from your system*, which is the tractable 80 % of the +complaint. The subjective residue stays with the visual-loop reviewer and the human — +the gate's job is to stop the template from ever reaching them. Thresholds start +advisory and only become enforcing once the P8 fixture set (known-slop vs. known-custom +UI corpus) shows separation, same as every other research-edge gate +([00-overview.md](./00-overview.md) §4). diff --git a/docs/plans/substrate-v2/08-dashboard-ux.md b/docs/plans/substrate-v2/08-dashboard-ux.md new file mode 100644 index 0000000..4e766e9 --- /dev/null +++ b/docs/plans/substrate-v2/08-dashboard-ux.md @@ -0,0 +1,55 @@ +# 08 — `forge dash` and the UX layer + +> ForgeKit's own user experience: a local dashboard that makes the substrate's state — +> memory, cost, cache, blast radius, drift — visible and steerable. The substrate's +> decisions are auditable by design (paper §5, 17:36's audit trail; §9.1 "a transparent +> rubric … the user can see and override"); this is the surface where a human actually +> sees them. Phase P7. + +## 0. Shape + +- `forge dash [--port 4242]` — Node `http` server (stdlib), serving one self-contained + HTML file (inline CSS/JS, no CDN — the `landing/index.html` precedent) plus small JSON + endpoints that read `.forge/` stores. Localhost-only by default; read-mostly (the two + writes below are explicit POSTs). +- Works offline, no build step, no framework. It must pass its own gate: uicheck v2 + + a taste profile ([07](./07-ui-quality-gate.md)) — the dashboard is the reference + customer of the UI quality gate. + +## 1. Panels → data contracts + +| panel | shows | reads | +|---|---|---| +| **Ledger** | claims by kind/layer (ʿilm/fahm/ḥikma), val sparkline per claim (evidence history re-scored over time), contested claims (val ∈ [0.4,0.6] with contradictions), attic | `.forge/ledger/` | +| **Cost** | spend by stage (gate/cache/context/route/generate/verify), measured savings vs. baseline, budget meter (the statusline meter, expanded) | `.forge/metrics.jsonl` | +| **Reuse** | hit rate (exact/near/adapt/miss), tokens saved, top-served artifacts with their evidence, artifacts nearing the val floor | metrics + ledger `artifact` claims | +| **Impact** | atlas blast-radius explorer: pick a symbol → dependents graph (SVG, hop-decay shading like paper Fig. 4), edge tier (ast/regex/verified) | `.forge/atlas.json` | +| **Session** | goal claim, CUSUM drift chart with alarm threshold, doom-loop signatures + diagnoses, M6 checkpoint log | `.forge/trace/` | +| **Team** | claims by author, per-author trust weight, recently merged teammate evidence, promotion queue | ledger provenance | + +Every number links to its provenance (claim id → `forge ledger blame` view) — no +unexplained scores anywhere in the UI, matching the CLI's rubric-transparency rule. + +## 2. The two writes + +1. **Ratify** — promote a fahm claim to ḥikma ([01](./01-pcm-protocol.md) §5): one click, + mints a `decision` claim with the human as author. Promotion is human-only by design. +2. **Retract** — tombstone a claim with a reason. Both append-only, so the dashboard + can never corrupt the ledger; both are also available as CLI (`forge ledger ratify| + retract`) so the dashboard is a convenience, never a requirement. + +## 3. CLI polish (same phase, small) + +- Uniform table/color renderer for all `forge` list outputs (one module, respects + `NO_COLOR` and `--json` everywhere — several commands already do; make it the rule). +- `forge doctor` gains substrate-v2 checks: ledger normal-form, merge driver installed, + atlas tier coverage, metrics file writable. +- Every gate refusal (impact/context/uicheck) prints the same three-part shape: + *what was blocked · the computed reason (the missing-set / signature / feature deltas) · + the exact command to inspect it* — refusals teach, not just deny. + +## 4. Non-goals + +No hosted app, no auth, no multi-user server, no websockets. The ledger is the shared +state; the dashboard is a local lens on it. If two teammates both run `forge dash`, they +see the same truth after `git pull` — that is the sync story, and it is enough.