diff --git a/CHANGELOG.md b/CHANGELOG.md index 05dd0cc..a60133c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,48 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added + +- **Optional embeddings tier** (`src/embed.js`, ADR-0005; ROADMAP "Next"): set + `FORGE_EMBED=cmd:` (stdin/stdout JSON protocol — any local model or script) + or `FORGE_EMBED=http:` (OpenAI-compatible, `$FORGE_EMBED_MODEL` / + `$FORGE_EMBED_KEY`, key never logged) and `forge reuse query` + `forge ledger query` + replace the MinHash `rel` term with embedding cosine (near/adapt ≥ 0.85/0.7 — a + higher bar than Jaccard's 0.8/0.6 to match dense cosine's noise floor), fixing the + documented weak spot on very short specs. Vectors are disk-cached + (`.forge/embed-cache.jsonl`, content-hash keyed, corrupt-tolerant, truncate-oldest); + both commands print the backend that served (`sim: minhash` / `sim: embed(cmd)`); + any provider failure degrades silently to MinHash. `dependencies` stays empty — + the tier is configuration, not a package; the pure ledger core never imports it. +- **`forge uicheck visual `** — the Playwright visual loop + (07-ui-quality-gate §5): renders the page headless at two viewports, fingerprints + the **computed** styles of every visible element (what the cascade and runtime + theming actually painted, with used `auto`-margins and never-painted UA noise + filtered out), and runs the identical `design` gate over that rendered vector — + screenshots land in `.forge/ui/`. Playwright stays an optional tier (ADR-0005): + `package.json` gains no dependency, absence degrades to a "skipped (no browser + runtime)" note with exit 0 (`npm i -D playwright-core` or `FORGE_PLAYWRIGHT=…` to + enable), and non-loopback http(s) targets are refused by default (`--remote` to + override) — a gate that fetches arbitrary URLs is an exfiltration hazard. + +### Changed + +- **Ledger read-path flip (P2).** Reads are now a merged view (legacy ∪ ledger) via the + new `src/ledger_read.js`, so teammate knowledge that arrives with `forge ledger merge` + actually reaches injection and retrieval: cortex lesson surfaces + (`lessonsForContext`, `startupBlock`, `summary`, the substrate advisory and routing + past-mistake density) map ledger `lesson` claims onto the legacy lesson shape with an + evidence-derived status (tombstoned → retired, val ≥ 0.6 → active, val < 0.45 with a + contradiction → quarantined, else candidate), and fact surfaces (`recall list`/ + `MEMORY.md`, brain's `AGENTS.md` index) include live ledger `fact` claims — always + deduped by legacy id/slug with the local file winning, and best-effort (a missing or + corrupt ledger degrades to legacy-only). Write paths (`recordMistake`'s + confirm-vs-create lookup, `recordContradiction`, `applyDistillation`) deliberately + keep reading the legacy store they edit; convergence comes from content-addressed + claim ids. `reconcileFacts` now only tombstones locally-authored claims, so a merged + teammate fact survives `forge recall consolidate`. Legacy formats are still written — + full retirement is the next step. + ## [0.6.0] - 2026-07-07 ### Changed diff --git a/README.md b/README.md index 7ee6b08..1e38046 100644 --- a/README.md +++ b/README.md @@ -166,8 +166,9 @@ Forge states its own ceiling everywhere. In short: **guards reduce, don't elimin the "ignored my rules" problem; `recall`/`cortex` are file memory, **not** weight-level learning; the `atlas`/`impact` graph is regex-approximate (conservative, not a sound call graph — the impact numbers above are n = 6 hand-labeled cases on one JavaScript -repo); the substrate's rubrics are heuristic; `forge reuse`'s near-match is weak on very -short specs; and `forge cost --stages` reports **measured stages only** — a stage with +repo); the substrate's rubrics are heuristic; the MinHash near-match is weak on very short +specs (an optional embeddings backend — `FORGE_EMBED` — lifts this; MinHash stays the +zero-dependency default); and `forge cost --stages` reports **measured stages only** — a stage with no events says "no data", never a default. What's *asserted* is safe to gate on (repo grounding, graph traversal, routing arithmetic, test commands); everything else is *advisory*. **Tests and human corrections always win.** Full list: diff --git a/ROADMAP.md b/ROADMAP.md index 2580d04..1496233 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -26,14 +26,19 @@ confidence only from independent oracles, and merges across teammates conflict-f (git-native CRDT ledger). ## Next -- **Ledger read-path flip** — the ledger is the convergent *write* store today while - the legacy stores (`lessons/`, `recall`, `brain`) still serve reads; flip reads to be - ledger-first, then retire the legacy formats. +- **Legacy store retirement** — the read-path flip has shipped: every read surface + (cortex injection/status, the substrate advisory, routing, `recall list`, brain's + AGENTS.md index) is now a merged view (legacy ∪ ledger) via `src/ledger_read.js`, + so teammate knowledge from `forge ledger merge` reaches injection. The legacy + formats (`lessons/*.md`, recall/brain fact files) are still written as the canonical + local state; the remaining step is retiring them so the ledger is the only store. - **Embeddings tier** — optional vector backend (ADR-0005 dependency tier, stdlib fallback kept) for Eq. 3 retrieval and `forge reuse` near-match, where MinHash is weak on short specs. -- **Playwright loop** — drive a real browser in the UI gate: screenshot + interaction - checks feeding `forge uicheck` verdicts back as oracle evidence on design claims. +- **Playwright loop** — shipped as `forge uicheck visual` (rendered computed-style + fingerprint through the design gate + 2-viewport screenshots, optional-tier + playwright); still open: interaction checks and feeding verdicts back as oracle + evidence on design claims. - **Advisory → gated promotions** — outcome-calibrated routing weights, consolidation promotion (ʿilm→fahm), M6 hazard estimates: advisory today, become blocking only once fixtures measure them (overview §4 honesty register). diff --git a/docs/GUIDE.md b/docs/GUIDE.md index 9f3dc7b..bde0809 100644 --- a/docs/GUIDE.md +++ b/docs/GUIDE.md @@ -330,10 +330,12 @@ holds** — confidence above the 0.6 floor and every declared dependency still i ```console $ forge reuse query "debounce user input before firing search" + sim: minhash NEAR hit (similarity 0.87) — module at src/lib/debounce.js claim 9c41d2ab77e0 — `forge ledger blame 9c41d2ab` for its proof $ forge reuse query "quantum blockchain" + sim: minhash miss — nothing verified matches; generate, then `forge reuse mint` it ``` @@ -348,7 +350,32 @@ $ forge reuse mint "debounce helper for search input" --file src/lib/debounce.js `forge reuse stats` shows lookups by outcome + estimated tokens saved (from `.forge/metrics.jsonl`). Honest limit: the MinHash near-match is weak on very short -specs — a few words hash to too few shingles to rank reliably. +specs — a few words hash to too few shingles to rank reliably. The optional embeddings +tier below lifts exactly this. + +### `FORGE_EMBED` — the optional embeddings tier (ADR-0005) + +MinHash is the always-working, zero-dependency default. Set `FORGE_EMBED` and +`forge reuse query` + `forge ledger query` swap the lexical similarity for embedding +cosine — no new flags, the env var is the switch, and every query prints which backend +served it (`sim: minhash` / `sim: embed(cmd)` / `sim: embed(http)`): + +- `FORGE_EMBED=cmd:` — the universal escape hatch (any local model, any + script): forge writes `{"texts":[...]}` to its stdin and reads + `{"vectors":[[...]]}` from its stdout. +- `FORGE_EMBED=http:` (or a bare `https://` URL) — OpenAI-compatible + `POST {input, model: $FORGE_EMBED_MODEL}` with + `Authorization: Bearer $FORGE_EMBED_KEY` (the key is passed via environment only — + never logged, never in argv). + +Thresholds move with the scale: near/adapt are cosine ≥ 0.85/0.7 instead of Jaccard's +0.8/0.6, because dense cosines have a much higher noise floor (unrelated sentences +commonly score 0.4–0.6) while unrelated shingle sets sit near 0. Vectors are cached in +`.forge/embed-cache.jsonl` (content-hash keyed, corrupt-tolerant, size-capped +truncate-oldest) so repeated queries never re-pay the provider. Any provider failure — +crash, timeout, garbage output — silently degrades to the MinHash path +(`FORGE_EMBED_TIMEOUT_MS` caps the wait, default 15000). Per ADR-0005: +`dependencies` stays empty; this tier is configuration, not a package. ### `forge context ""` — budgeted assembly + completeness gate @@ -424,7 +451,8 @@ tree (your uncommitted changes wouldn't be in the run); commit/stash first or pa ### `forge uicheck` — deterministic UI checks -Three subcommands, all static parsing — no LLM, no screenshots. +Four subcommands: three are static parsing — no LLM, no screenshots — and `visual` +optionally drives a real browser. **`contrast `** — exact WCAG math, asserted, never guessed (bare `forge uicheck ` still works): @@ -457,6 +485,32 @@ Failures are actionable per-feature edits, never a bare score. Honest limit: the fingerprint doesn't resolve CSS `var()` indirection yet — fully tokenized palettes are partially invisible to it. +**`visual [--taste ] [--json] [--remote]`** — the Playwright +visual loop: renders the page headless at two viewports (1280×800, 390×844), +fingerprints the **computed** styles of every visible element — what the cascade, +`var()` resolution, and runtime theming actually produced — and runs the exact same +design gate as `design` (exit 1 on fail). Screenshots land in `.forge/ui/` for human +review. Playwright is an *optional tier* (ADR-0005): `package.json` stays +dependency-free; without a browser runtime the command prints a "skipped (no browser +runtime)" note and exits 0 — enable it with `npm i -D playwright-core` or point +`FORGE_PLAYWRIGHT` at an existing install (e.g. +`FORGE_PLAYWRIGHT=/path/to/node_modules/playwright-core`). Security default: http(s) +targets must be loopback (`localhost`, `127.*`, `[::1]`) — fetching arbitrary URLs is +an exfiltration hazard, so non-local URLs are refused unless you pass `--remote`. + +```console +$ forge uicheck visual src/dash.html +Forge uicheck visual — rendered fingerprint + design gate + + rendered: file:///…/src/dash.html (80 visible element style(s)) + screenshots: .forge/ui/dash-1280x800.png, .forge/ui/dash-390x844.png + slop distance: 0.516 (need ≥ 0.25 — farther from generic is better) + ... + ✓ spacing-scale: 100% of 6 spacing value(s) on the 4px base (ε 0.5px) + + ✓ PASS +``` + ### `forge dash [--port N]` — the local dashboard A read-only lens over `.forge/` — stdlib `node:http`, localhost-only, one @@ -683,7 +737,8 @@ Edit `brand.json` (`FORGE_BRAND`), the `bin` key in `package.json`, and `name` i - **The atlas graph is regex-approximate** — conservative, not a sound call graph; dynamic dispatch and generated code can be missed. - **`forge reuse`'s MinHash near-match is weak on very short specs** — a few words hash - to too few shingles to rank reliably; write a sentence, not a keyword. + to too few shingles to rank reliably; write a sentence, not a keyword — or configure + the optional `FORGE_EMBED` embeddings tier, which replaces exactly this term. - **The UI fingerprint doesn't resolve CSS `var()` indirection yet** — a fully tokenized palette is partially invisible to the design gate. - **`forge cost --stages` reports measured stages only** — a stage with no events says diff --git a/docs/plans/substrate-v2/07-ui-quality-gate.md b/docs/plans/substrate-v2/07-ui-quality-gate.md index 1a14d27..527a996 100644 --- a/docs/plans/substrate-v2/07-ui-quality-gate.md +++ b/docs/plans/substrate-v2/07-ui-quality-gate.md @@ -99,6 +99,10 @@ the other gates). ## 5. The visual loop (the only non-static part) +> Status: shipped as `forge uicheck visual ` (`src/uivisual.js`) — rendered +> computed-style fingerprint through the same gate, 2-viewport screenshots to +> `.forge/ui/`, optional-tier playwright with graceful absence (ADR-0005). + Static analysis can't see rendered composition. Under ADR-0005, the `frontend-verifier` crew agent gets Playwright: diff --git a/src/brain.js b/src/brain.js index 1e7999c..6887cf6 100644 --- a/src/brain.js +++ b/src/brain.js @@ -5,11 +5,16 @@ // by construction: the inlined index is capped; overflow stays in fact files, never // silently truncated the way Claude's native 200-line MEMORY.md is (#39811). import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { add as recallAdd, list as recallList } from "./recall.js"; +import { dirname, join } from "node:path"; +import { ledgerFacts, mergeFactSlugs } from "./ledger_read.js"; +import { listStored, add as recallAdd } from "./recall.js"; export const brainStore = (targetRoot = process.cwd()) => join(targetRoot, ".forge", "brain"); +// Brain facts shadow into the REPO ledger (.forge/ledger — the sibling of this store, +// see `forge remember`), so that is where merged teammate facts arrive. +const brainLedger = (store) => join(dirname(store), "ledger"); + /** Store one fact (secret-refused by recall) and rebuild the inlined index. */ export function remember(store, name, body) { const res = recallAdd(store, name, body); @@ -17,7 +22,8 @@ export function remember(store, name, body) { return res; } -export const list = recallList; +/** Merged read (P2 read flip): file facts ∪ live repo-ledger facts, file wins on slug. */ +export const list = (store) => mergeFactSlugs(listStored(store), brainLedger(store)); const gistOf = (text) => text @@ -26,7 +32,9 @@ const gistOf = (text) => .map((l) => l.trim()) .find(Boolean) || ""; -/** Build the capped, cliff-safe index inlined into AGENTS.md. Overflow → a pointer. */ +/** Build the capped, cliff-safe index inlined into AGENTS.md. Overflow → a pointer. + * Merged view (P2 read flip): teammate facts that arrived in the repo ledger via + * `forge ledger merge` join the index; a local file wins on name collision. */ export function buildIndex(store, { capItems = 120 } = {}) { const factsDir = join(store, "facts"); const facts = existsSync(factsDir) @@ -34,17 +42,25 @@ export function buildIndex(store, { capItems = 120 } = {}) { .filter((f) => f.endsWith(".md")) .sort() : []; + const entries = facts.map((file) => ({ + name: file.replace(/\.md$/, ""), + gist: gistOf(readFileSync(join(factsDir, file), "utf8")), + })); + const seen = new Set(entries.map((e) => e.name)); + for (const f of ledgerFacts(brainLedger(store))) { + if (seen.has(f.slug)) continue; + seen.add(f.slug); + entries.push({ name: f.slug, gist: gistOf(f.text) }); + } + entries.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)); const rows = []; let overflow = 0; - for (const file of facts) { + for (const e of entries) { if (rows.length >= capItems) { overflow += 1; continue; } - const name = file.replace(/\.md$/, ""); - rows.push( - `- **${name}** — ${gistOf(readFileSync(join(factsDir, file), "utf8")).slice(0, 140)}`, - ); + rows.push(`- **${e.name}** — ${e.gist.slice(0, 140)}`); } const indexed = rows.length; // count real facts before adding the overflow pointer if (overflow) diff --git a/src/cli.js b/src/cli.js index 7a5b5da..18de17e 100755 --- a/src/cli.js +++ b/src/cli.js @@ -34,7 +34,7 @@ const COMMANDS = { imagine: "consequence simulation — predicted breaks + the minimal dry-run test suite for a task", lean: "scope-minimality (M5) — measure the diff's footprint vs what the task asked for", uicheck: - "deterministic UI checks — contrast · fingerprint · design ", + "deterministic UI checks — contrast · fingerprint · design · visual ", dash: "local dashboard over the ledger, metrics, and blast radius", brand: "print the active brand token map", }; @@ -341,16 +341,25 @@ async function run(argv) { return; } const { retrieve, claimText } = await import("./ledger.js"); + // The embeddings tier (ADR-0005) is assembled HERE, not in ledger.js — the pure + // core stays provider-free. No FORGE_EMBED (or a failing provider) → sim is + // null and retrieval is the stock MinHash path. + const { claimSim, simLabel } = await import("./embed.js"); const claims = ls.loadClaims(dir); - const ranked = retrieve(q, claims, { nowDay, budget: 8 }); + const sim = claimSim(root, q, claims, claimText); + const ranked = retrieve(q, claims, { nowDay, budget: 8, sim }); if (json) return console.log( JSON.stringify( - ranked.map((r) => ({ id: r.claim.id, kind: r.claim.kind, score: r.score })), + { + sim: simLabel(sim), + results: ranked.map((r) => ({ id: r.claim.id, kind: r.claim.kind, score: r.score })), + }, null, 2, ), ); + console.log(` sim: ${simLabel(sim)}`); if (!ranked.length) return console.log(" no matching live claims"); for (const r of ranked) console.log( @@ -408,17 +417,25 @@ async function run(argv) { if (json) return console.log( JSON.stringify( - { tier: r.tier, artifact: r.artifact?.id, jaccard: r.jaccard, reasons: r.reasons }, + { + tier: r.tier, + artifact: r.artifact?.id, + jaccard: r.jaccard, + similarity: r.similarity, + sim: r.sim, + reasons: r.reasons, + }, null, 2, ), ); + console.log(` sim: ${r.sim}`); if (r.tier === "miss") { console.log(" miss — nothing verified matches; generate, then `forge reuse mint` it"); } else { const a = r.artifact; console.log( - ` ${r.tier.toUpperCase()} hit (similarity ${(r.jaccard ?? 1).toFixed(2)}) — ${a.body.form}${a.body.code?.path ? ` at ${a.body.code.path}` : ""}`, + ` ${r.tier.toUpperCase()} hit (similarity ${(r.similarity ?? r.jaccard ?? 1).toFixed(2)}) — ${a.body.form}${a.body.code?.path ? ` at ${a.body.code.path}` : ""}`, ); console.log( ` claim ${a.id.slice(0, 12)} — \`forge ledger blame ${a.id.slice(0, 8)}\` for its proof`, @@ -1047,6 +1064,70 @@ async function run(argv) { } if (cmd === "uicheck") { const sub = argv[1]; + if (sub === "visual") { + // The Playwright visual loop (spec §5): render in a real browser, fingerprint + // the COMPUTED styles, run the same design gate. Playwright is an optional + // tier (ADR-0005) — its absence is a note and exit 0, never a failure. + const { visualGate } = await import("./uivisual.js"); + const args = argv.slice(2); + const tasteIdx = args.indexOf("--taste"); + const tasteArg = tasteIdx >= 0 ? (args.splice(tasteIdx, 2)[1] ?? null) : null; + const json = args.includes("--json"); + const remote = args.includes("--remote"); + const targets = args.filter((a) => !a.startsWith("--")); + if (targets.length !== 1 || (tasteIdx >= 0 && !tasteArg)) { + console.error( + `usage: ${BRAND.cli} uicheck visual [--taste ] [--json] [--remote]`, + ); + process.exitCode = 1; + return; + } + const r = await visualGate(targets[0], { taste: tasteArg, remote, root: process.cwd() }); + if (!r.ok) { + const reason = "reason" in r ? r.reason : "visual gate failed"; + if ("skipped" in r && r.skipped) { + // Graceful absence (ADR-0005): a missing optional tier is not a failure. + if (json) console.log(JSON.stringify({ skipped: true, reason }, null, 2)); + else { + console.log(`${BRAND.brand} uicheck visual — skipped (no browser runtime)\n`); + console.log(` ${reason}`); + console.log( + " enable it: npm i -D playwright-core (or point FORGE_PLAYWRIGHT at an existing install, e.g. FORGE_PLAYWRIGHT=/path/to/node_modules/playwright-core)", + ); + } + return; // exit 0 — the static gate still stands + } + console.error(reason); + process.exitCode = 1; + return; + } + if (json) { + const { ok: _ok, fail: _fail, ...body } = r; + console.log(JSON.stringify(body, null, 2)); + } else { + console.log(`${BRAND.brand} uicheck visual — rendered fingerprint + design gate\n`); + console.log(` rendered: ${r.url} (${r.elements} visible element style(s))`); + console.log(` screenshots: ${r.screenshots.join(", ")}`); + if (r.taste) console.log(` taste: ${r.taste} (thresholds from its profile)`); + console.log( + ` slop distance: ${r.slop} (need ≥ ${r.tauSlop} — farther from generic is better)`, + ); + console.log( + r.hasProjectFingerprint + ? ` conformance: ${r.conform} (need ≤ ${r.tauConform} — closer to the project system is better)` + : ` conformance: (no project fingerprint claim — slop-only; mint one: \`${BRAND.cli} uicheck fingerprint --mint\`)`, + ); + for (const v of r.violations) console.log(`\n ✗ ${v.detail}\n fix: ${v.hint}`); + console.log(""); + for (const c of r.checks) + console.log( + ` ${c.pass ? "✓" : "✗"} ${c.id}: ${c.detail}${c.pass || !c.hint ? "" : `\n fix: ${c.hint}`}`, + ); + console.log(`\n ${r.fail ? "✗ FAIL" : "✓ PASS"}`); + } + if (r.fail) process.exitCode = 1; + return; + } if (sub === "fingerprint" || sub === "design") { const ui = await import("./uifingerprint.js"); // `--taste ` (design only) takes a VALUE — splice it out before the diff --git a/src/cortex.js b/src/cortex.js index 4c7ae63..904d1a6 100644 --- a/src/cortex.js +++ b/src/cortex.js @@ -5,6 +5,7 @@ // without any hook wiring. import { recordLessonEvent, supersedeLessonClaim } from "./ledger_bridge.js"; +import { mergedLessons } from "./ledger_read.js"; import { confidenceOf, confirm, @@ -74,6 +75,13 @@ export function recordMistake(root, { signals, context, nowDay, episodeId, disti const accumulated = fires && p >= 0.4 && recurredWeak; if (!strong && !accumulated) return { action: "logged", p, fires }; + // Deliberately LEGACY-ONLY (not mergedLessons): this lookup decides confirm-vs-create + // for a LOCAL write. A teammate's merged ledger claim has no local file, and letting it + // match would swallow a fresh local mistake into confirm() — which edits a legacy file + // that doesn't exist — and hang the shadow evidence ref (#n) off counters this + // repo never owned. Creating locally is correct AND convergent: the new lesson's shadow + // claim content-addresses to the teammate's claim, so both sides' evidence lands on ONE + // claim at the next `forge ledger merge`. const existing = matchingLessons(load(root), context)[0]; if (existing) { const c = confirm(existing, nowDay); @@ -85,8 +93,8 @@ export function recordMistake(root, { signals, context, nowDay, episodeId, disti }, }; if (!save(root, updated).ok) return { action: "refused", p, fires }; - // Shadow the confirmation into the PCM ledger (P1 bridge — legacy store stays the - // read path; best-effort by design, never blocks the hook). The evidence counter + // Shadow the confirmation into the PCM ledger (best-effort by design, never blocks + // the hook — reads are merged via ledger_read.js since P2). The evidence counter // rides in the ref because episode ids reset per session (ep_m0_…) — without it, // two same-day sessions confirming via the same file would hash identically and // the second real confirmation would be silently deduped away. @@ -144,6 +152,9 @@ export function recordContradiction(root, { context, nowDay, episodeId }) { context, day: nowDay, }); + // Legacy-only for the same reason as recordMistake: contradict() saves the legacy + // file, so only lessons that HAVE one are targets. A teammate's claim still converges + // — the same reversal, recurring locally, mints the local twin whose evidence merges. const targets = matchingLessons(load(root), context, ["active", "quarantined"]); const results = targets.map((l) => { const updated = contradict(l, nowDay); @@ -163,14 +174,17 @@ export function recordContradiction(root, { context, nowDay, episodeId }) { return { action: "contradicted", results }; } -/** The injection block for the current context — what a SessionStart/PreToolUse hook emits. */ +/** The injection block for the current context — what a SessionStart/PreToolUse hook emits. + * Reads the MERGED view (P2 read flip): teammate lessons that arrived via + * `forge ledger merge` inject alongside local ones. */ export function lessonsForContext(root, context, opts = {}) { - return selectForInjection(load(root), context, opts); + return selectForInjection(mergedLessons(root, opts.nowDay ?? 0), context, opts); } -/** Repo-wide top active lessons — what a SessionStart hook injects (no file context yet). */ +/** Repo-wide top active lessons — what a SessionStart hook injects (no file context yet). + * Merged view: a teammate's outcome-confirmed lesson surfaces here too. */ export function startupBlock(root, nowDay = 0, budget = 8) { - const active = load(root).filter((l) => l.status === "active"); + const active = mergedLessons(root, nowDay).filter((l) => l.status === "active"); if (!active.length) return ""; const ranked = active .map((l) => ({ lesson: l, conf: confidenceOf(l, nowDay) })) @@ -194,6 +208,7 @@ export function startupBlock(root, nowDay = 0, budget = 8) { * refused — e.g. the distilled text tripped secret-refusal). */ export function applyDistillation(root, lessonId, distilled) { if (!distilled) return false; + // Legacy-only read: distillation EDITS the legacy file, so only file-backed lessons apply. const lesson = load(root).find((l) => l.id === lessonId); if (!lesson) return false; const updated = { @@ -214,9 +229,9 @@ export function cortexBlock(targetRoot = process.cwd()) { return startupBlock(targetRoot, Math.floor(Date.now() / 86400000)); } -/** Auditable snapshot for `forge cortex status`. */ +/** Auditable snapshot for `forge cortex status` — merged view, like every read surface. */ export function summary(root, nowDay = 0) { - const lessons = load(root); + const lessons = mergedLessons(root, nowDay); const by = (s) => lessons.filter((l) => l.status === s).length; return { total: lessons.length, diff --git a/src/embed.js b/src/embed.js new file mode 100644 index 0000000..935de71 --- /dev/null +++ b/src/embed.js @@ -0,0 +1,244 @@ +// forge embed — the OPTIONAL embeddings tier (ROADMAP "Next", ADR-0005). MinHash is +// weak on very short specs (a few words hash to too few shingles — the documented +// honest limit); a configured embedding provider replaces the lexical `rel` term in +// Eq. 3 retrieval and the reuse cache's near-match with semantic cosine similarity. +// +// ADR-0005 compliance, condition by condition: +// 1. Core stays stdlib — this file is node stdlib only (spawnSync + the provider's +// own process); ledger.js never imports it. MinHash remains the reference path. +// 2. Graceful absence — no FORGE_EMBED, a crashed provider, a timeout, or garbage +// output all yield `null`, and every caller keeps the MinHash path unchanged. +// 3. Named tiers — `dependencies` stays empty; the provider is an opt-in env config +// (`FORGE_EMBED`), not a package at all. +// 4. Vetting — no third-party code runs unless the USER points FORGE_EMBED at it. +// +// Providers (FORGE_EMBED): +// cmd: spawn it, write {"texts":[..]} to stdin, read +// {"vectors":[[..]]} from stdout. The universal escape hatch: +// any local model, any script, any language. +// http: OpenAI-compatible POST {input, model: $FORGE_EMBED_MODEL} +// with Authorization: Bearer $FORGE_EMBED_KEY. The fetch runs +// in a spawned node child (global fetch, node ≥20 stdlib) so +// this module stays synchronous like every other forge stage. +// The key travels only via the child's environment — it is +// never logged, never put in argv. +import { spawnSync } from "node:child_process"; +import { + appendFileSync, + existsSync, + mkdirSync, + readFileSync, + statSync, + writeFileSync, +} from "node:fs"; +import { join } from "node:path"; +import { contentHash } from "./util.js"; + +/** Hard ceilings — a misbehaving provider must never hang or flood a forge command. */ +const DEFAULT_TIMEOUT_MS = 15000; +const MAX_TEXT_CHARS = 4000; // specs/claim texts are short; embed a stable prefix +const MAX_INPUT_BYTES = 2 * 1024 * 1024; +const MAX_OUTPUT_BYTES = 32 * 1024 * 1024; +/** Cache cap — see appendCache for the truncate-oldest policy. */ +export const CACHE_MAX_BYTES = 4 * 1024 * 1024; + +export const embedCachePath = (root = process.cwd()) => join(root, ".forge", "embed-cache.jsonl"); + +// Memoized per env value (not per process) so tests and long-lived callers that flip +// FORGE_EMBED re-resolve, while hot paths pay the parse once. +let memo = { env: /** @type {string|undefined} */ (undefined), provider: null }; + +/** + * Resolve the provider from FORGE_EMBED. Unset/unknown scheme → null (callers keep + * MinHash — the always-working default). + * @returns {{kind:"cmd", cmd:string}|{kind:"http", url:string}|null} + */ +export function getProvider() { + const env = process.env.FORGE_EMBED ?? ""; + if (memo.env === env) return memo.provider; + /** @type {any} */ + let provider = null; + if (env.startsWith("cmd:") && env.length > 4) provider = { kind: "cmd", cmd: env.slice(4) }; + else if (/^https?:\/\//.test(env)) provider = { kind: "http", url: env }; + else if (env.startsWith("http:") && env.length > 5) + provider = { kind: "http", url: env.slice(5) }; + memo = { env, provider }; + return provider; +} + +/** Cosine similarity in [-1,1]; zero-norm or empty vectors → 0 (never NaN). */ +export function cosine(a, b) { + const n = Math.min(a?.length ?? 0, b?.length ?? 0); + if (!n) return 0; + let dot = 0; + let na = 0; + let nb = 0; + for (let i = 0; i < n; i++) { + dot += a[i] * b[i]; + na += a[i] * a[i]; + nb += b[i] * b[i]; + } + return na && nb ? dot / Math.sqrt(na * nb) : 0; +} + +const isVec = (v) => Array.isArray(v) && v.length > 0 && v.every((x) => Number.isFinite(x)); + +/** Cache keys include the provider identity + model — switching backends must never + * serve another model's vectors. */ +const providerId = (p) => + `${p.kind}:${p.kind === "cmd" ? p.cmd : p.url}:${process.env.FORGE_EMBED_MODEL ?? ""}`; + +/** Corrupt-line-tolerant cache reader (same discipline as metrics.js — a bad line is + * skipped, never fatal). @returns {Map} */ +function readCache(root) { + const out = new Map(); + const path = embedCachePath(root); + if (!existsSync(path)) return out; + try { + for (const line of readFileSync(path, "utf8").split("\n")) { + if (!line.trim()) continue; + try { + const e = JSON.parse(line); + if (typeof e.k === "string" && isVec(e.v)) out.set(e.k, e.v); + } catch {} + } + } catch {} + return out; +} + +/** Append new vectors; when the file outgrows CACHE_MAX_BYTES, rewrite it keeping the + * NEWEST lines that fit in half the cap (truncate-oldest — embeddings are + * re-derivable, so losing an old entry only costs one re-embed, never correctness). + * Best-effort: a cache write failure must never fail the query it was accelerating. */ +function appendCache(root, entries) { + if (!entries.length) return; + const path = embedCachePath(root); + try { + mkdirSync(join(root, ".forge"), { recursive: true }); + appendFileSync(path, `${entries.map((e) => JSON.stringify(e)).join("\n")}\n`); + if (statSync(path).size > CACHE_MAX_BYTES) { + const lines = readFileSync(path, "utf8").split("\n").filter(Boolean); + const keep = []; + let bytes = 0; + for (let i = lines.length - 1; i >= 0; i--) { + bytes += lines[i].length + 1; + if (bytes > CACHE_MAX_BYTES / 2) break; + keep.unshift(lines[i]); + } + writeFileSync(path, keep.length ? `${keep.join("\n")}\n` : ""); + } + } catch {} +} + +// Runs in a spawned node child: reads {url, model, texts} from stdin, POSTs the +// OpenAI-compatible payload, prints {vectors}. The Authorization header is built from +// the CHILD's env — the key never appears in argv, logs, or error output. +const HTTP_CHILD = `let raw="";process.stdin.on("data",(d)=>{raw+=d;});process.stdin.on("end",async()=>{try{const{url,model,texts}=JSON.parse(raw);const headers={"content-type":"application/json"};if(process.env.FORGE_EMBED_KEY)headers.authorization="Bearer "+process.env.FORGE_EMBED_KEY;const res=await fetch(url,{method:"POST",headers,body:JSON.stringify(model?{input:texts,model}:{input:texts})});if(!res.ok){console.error("embed: http "+res.status);process.exit(1);}const data=await res.json();process.stdout.write(JSON.stringify({vectors:(data.data??[]).map((d)=>d.embedding)}));}catch(e){console.error("embed: request failed");process.exit(1);}});`; + +/** One provider round-trip. Any failure (spawn error, non-zero exit, timeout, bad + * JSON, wrong count, malformed vectors) → null. Never throws, never logs the key. */ +function callProvider(provider, texts) { + const timeout = Number(process.env.FORGE_EMBED_TIMEOUT_MS) || DEFAULT_TIMEOUT_MS; + const input = + provider.kind === "cmd" + ? JSON.stringify({ texts }) + : JSON.stringify({ url: provider.url, model: process.env.FORGE_EMBED_MODEL, texts }); + if (input.length > MAX_INPUT_BYTES) return null; + try { + const r = + provider.kind === "cmd" + ? spawnSync(provider.cmd, { + shell: true, + input, + encoding: "utf8", + timeout, + maxBuffer: MAX_OUTPUT_BYTES, + }) + : spawnSync(process.execPath, ["-e", HTTP_CHILD], { + input, + encoding: "utf8", + timeout, + maxBuffer: MAX_OUTPUT_BYTES, + }); + if (r.error || r.status !== 0 || !r.stdout) return null; + const vectors = JSON.parse(r.stdout).vectors; + if (!Array.isArray(vectors) || vectors.length !== texts.length) return null; + return vectors.map((v) => (isVec(v) ? v : null)); + } catch { + return null; + } +} + +/** + * Embed texts through the configured provider, via the content-hash-keyed disk cache + * (`.forge/embed-cache.jsonl` under `root`) so repeated retrieval doesn't re-pay the + * provider. Returns an array aligned with `texts` where an entry may be null (that + * one text couldn't be embedded — per-candidate MinHash fallback), or null overall + * when no provider is configured / nothing could be embedded. + * @param {string[]} texts + * @param {{root?: string}} [opts] + * @returns {(number[]|null)[]|null} + */ +export function embed(texts, { root = process.cwd() } = {}) { + const provider = getProvider(); + if (!provider || !Array.isArray(texts) || !texts.length) return null; + const pid = providerId(provider); + const trimmed = texts.map((t) => String(t).slice(0, MAX_TEXT_CHARS)); + const keys = trimmed.map((t) => contentHash(`${pid}\n${t}`)); + const cache = readCache(root); + const out = keys.map((k) => cache.get(k) ?? null); + + const missIdx = out.flatMap((v, i) => (v === null ? [i] : [])); + if (missIdx.length) { + const missTexts = [...new Set(missIdx.map((i) => trimmed[i]))]; + const vecs = callProvider(provider, missTexts); + if (vecs) { + const byText = new Map(missTexts.map((t, i) => [t, vecs[i]])); + const fresh = new Map(); + for (const i of missIdx) { + const v = byText.get(trimmed[i]) ?? null; + out[i] = v; + if (v) fresh.set(keys[i], v); + } + appendCache( + root, + [...fresh].map(([k, v]) => ({ k, v })), + ); + } + } + return out.some(Boolean) ? out : null; +} + +/** + * Build the optional `sim` function callers inject into ledger.retrieve/score and + * reuse.lookup (those cores stay embed-free). Query + every candidate text go through + * ONE embed() call (cache-deduped); the returned sim maps a claim to + * cosine(query, claim) or null when that claim's vector is missing (per-candidate + * MinHash fallback). Returns null when no provider is configured, the candidate list + * is empty, or the query itself couldn't be embedded — the caller's MinHash path is + * then byte-for-byte unchanged. + * @param {string} root repo root (hosts the disk cache) + * @param {string} query + * @param {any[]} claims + * @param {(claim:any)=>string} textOf + * @returns {((query:any, claim:any)=>number|null)|null} + */ +export function claimSim(root, query, claims, textOf) { + if (!getProvider() || !claims.length) return null; + const texts = [String(query), ...claims.map((c) => String(textOf(c)))]; + const vectors = embed(texts, { root }); + const qv = vectors?.[0]; + if (!qv) return null; + const byText = new Map(); + for (let i = 1; i < texts.length; i++) if (vectors[i]) byText.set(texts[i], vectors[i]); + return (_query, claim) => { + const v = byText.get(String(textOf(claim))); + return v ? cosine(qv, v) : null; + }; +} + +/** The explainability line every query prints: which similarity backend served it. */ +export function simLabel(sim) { + const p = getProvider(); + return sim && p ? `embed(${p.kind})` : "minhash"; +} diff --git a/src/ledger.js b/src/ledger.js index 293e66d..3e587c0 100644 --- a/src/ledger.js +++ b/src/ledger.js @@ -332,22 +332,50 @@ const sketchOf = (claim) => (claim._sketch ??= sketch(claimText(claim))); * Eq. 3 retrieval score (paper §7.1): σ(a·rel + b·rec + g·val) × scope weight. * `query` may be a string or a precomputed sketch. The `g·val` term is the protocol's * load-bearing addition — outcome-confirmed claims outrank merely-recent ones. + * + * `sim` (optional) replaces the lexical `rel` term with a caller-supplied similarity + * (the ADR-0005 embeddings tier — built by callers from embed.js; this pure core + * NEVER imports a provider). It returns a cosine in [-1,1] or null; null (or any + * non-finite value) falls back to MinHash Jaccard per claim, and negatives clamp to 0 + * — "anti-similar" is just irrelevant, never a penalty below unrelated. + * @param {*} query + * @param {any} claim + * @param {{nowDay?:number, weights?:typeof EQ3_WEIGHTS, sim?:(query:any, claim:any)=>number|null}} [opts] */ -export function score(query, claim, { nowDay = 0, weights = EQ3_WEIGHTS } = {}) { - const qs = Array.isArray(query) ? query : sketch(query); - const rel = jaccard(qs, sketchOf(claim)); +export function score(query, claim, { nowDay = 0, weights = EQ3_WEIGHTS, sim } = {}) { + let rel = null; + if (sim) { + const s = sim(query, claim); + if (typeof s === "number" && Number.isFinite(s)) rel = Math.max(0, Math.min(1, s)); + } + if (rel === null) { + const qs = Array.isArray(query) ? query : sketch(query); + rel = jaccard(qs, sketchOf(claim)); + } const x = weights.a * rel + weights.b * rec(claim, nowDay) + weights.g * val(claim, nowDay); const sigma = 1 / (1 + Math.exp(-x)); const scopeW = SCOPE_WEIGHT[claim.scope?.level] ?? 0.5; return sigma * scopeW; } -/** Rank live (non-dormant, non-tombstoned) claims for a query; caps at `budget`. */ -export function retrieve(query, claims, { nowDay = 0, budget = 12, weights = EQ3_WEIGHTS } = {}) { - const qs = sketch(String(query)); +/** Rank live (non-dormant, non-tombstoned) claims for a query; caps at `budget`. + * Optional `sim` as in score() — the caller-built embedding similarity; the query + * string (not the sketch) is what a sim sees. + * @param {*} query + * @param {any[]} claims + * @param {{nowDay?:number, budget?:number, weights?:typeof EQ3_WEIGHTS, + * sim?:((query:any, claim:any)=>number|null)|null}} [opts] */ +export function retrieve( + query, + claims, + { nowDay = 0, budget = 12, weights = EQ3_WEIGHTS, sim } = {}, +) { + const q = String(query); + const qs = sketch(q); + const boundSim = sim ? (_qs, c) => sim(q, c) : undefined; return claims .filter((c) => !c.tombstone && !isDormant(c, nowDay)) - .map((c) => ({ claim: c, score: score(qs, c, { nowDay, weights }) })) + .map((c) => ({ claim: c, score: score(qs, c, { nowDay, weights, sim: boundSim }) })) .sort((a, b) => b.score - a.score || (a.claim.id < b.claim.id ? -1 : 1)) .slice(0, budget); } diff --git a/src/ledger_bridge.js b/src/ledger_bridge.js index 95ab66a..b8144ef 100644 --- a/src/ledger_bridge.js +++ b/src/ledger_bridge.js @@ -1,12 +1,13 @@ // forge ledger bridge — the P1 migration seam between the legacy stores (lessons/*.md, -// recall facts) and the PCM ledger. Legacy files stay the READ path (every existing -// test, hook, and guard keeps working unchanged); the ledger shadows every write as -// the new canonical, and `forge ledger import` back-fills pre-ledger history. P2 flips -// the read path once merge/verify tooling lands. Spec: docs/plans/substrate-v2/01-pcm-protocol.md §7. +// recall facts) and the PCM ledger. The ledger shadows every write as the new +// canonical, and `forge ledger import` back-fills pre-ledger history. P2 (shipped — +// see ledger_read.js) flipped READS to a merged view (legacy ∪ ledger); the legacy +// files remain the canonical LOCAL state until full retirement. +// Spec: docs/plans/substrate-v2/01-pcm-protocol.md §7. // // Every entry point here is BEST-EFFORT by design: the legacy store is still -// canonical in P1, so a bridge failure returns {ok:false} and must never break a -// hook or CLI write that already succeeded. +// canonical for local state, so a bridge failure returns {ok:false} and must never +// break a hook or CLI write that already succeeded. import { mintClaim, outcomeRecord } from "./ledger.js"; import { appendEvidence, @@ -18,7 +19,10 @@ import { tombstone, } from "./ledger_store.js"; import { load as loadLessons } from "./lessons_store.js"; -import { list as listFacts, readFact } from "./recall.js"; +// listStored, NOT list: the bridge reconciles the FILE store against the ledger. The +// merged `list` (P2 read flip) includes ledger-only teammate facts, which have no file +// and would read here as "deleted from the store". +import { listStored as listFacts, readFact } from "./recall.js"; import { epochDay, gitAuthor } from "./util.js"; /** One best-effort policy for the whole bridge (never throws into a caller). */ @@ -175,7 +179,13 @@ export function reconcileFacts(store, ledgerDir, t = epochDay()) { } let removed = 0; for (const c of loadClaims(ledgerDir)) { - if (c.kind === "fact" && !c.tombstone && !current.has(c.id)) { + // Only reconcile claims THIS author minted: a locally-authored claim with no + // backing file means the file was deleted (consolidate rm'd a duplicate). A + // teammate's claim (arrived via `forge ledger merge`) never had a local file — + // with the P2 read flip it IS the readable fact, and tombstoning it here would + // silently delete team knowledge on every consolidate. + const mine = (c.provenance?.author ?? "") === gitAuthor(); + if (c.kind === "fact" && !c.tombstone && mine && !current.has(c.id)) { tombstone(ledgerDir, c.id, { author: gitAuthor(), reason: "removed-from-store", t }); removed++; } diff --git a/src/ledger_read.js b/src/ledger_read.js new file mode 100644 index 0000000..527afae --- /dev/null +++ b/src/ledger_read.js @@ -0,0 +1,152 @@ +// forge ledger read — the P2 read-path flip (docs/plans/substrate-v2/01-pcm-protocol.md §7). +// Since P1 the PCM ledger has been the convergent WRITE store while the legacy stores +// (lessons/*.md, recall/brain facts) served every read. This module turns reads into a +// MERGED VIEW (legacy ∪ ledger) so knowledge that arrives via `forge ledger merge` +// actually reaches injection and retrieval. The legacy file is still the canonical +// LOCAL state — the ledger only ADDS what teammates know — so the merge dedupes by the +// legacy id with the legacy record winning. Retiring the legacy formats entirely is the +// next step (ROADMAP.md). +// +// Everything here is READ-ONLY and best-effort by design: hooks call these on every +// session start / pre-edit, so a missing or corrupt ledger degrades to legacy-only — +// never an error, never a write. +import { DEFAULT_HALF_LIFE_DAYS, val, validOutcome } from "./ledger.js"; +import { loadClaims, repoLedger } from "./ledger_store.js"; +import { load } from "./lessons_store.js"; +import { slug } from "./util.js"; + +/** + * Map a ledger `lesson` claim onto the legacy lesson shape (lessons.js), so every + * existing consumer — matchScore, selectForInjection, confidenceOf, summary — can rank + * a teammate's claim without knowing the ledger exists. Inverse-ish of + * ledger_bridge.lessonClaim: body carries trigger/whatWentWrong/correctedBehavior, and + * the legacy lesson id rides in provenance.task (it is excluded from the content + * address so teammates converge on one claim id). + * + * The legacy lifecycle (candidate/active/quarantined/retired) is local mutable state + * the ledger deliberately does not store, so status is DERIVED from evidence: + * + * | claim state | derived status | rationale | + * |--------------------------------------|----------------|-----------| + * | tombstoned | "retired" | a retraction is the ledger's retirement | + * | val(claim, nowDay) ≥ 0.6 | "active" | one fresh confirm (bridge oracle w=0.5 → val 0.6) clears it — mirrors confirm()'s promote-on-recurrence | + * | val < 0.45 and ≥ 1 contradiction | "quarantined" | net-negative outcome evidence — mirrors contradict()'s demotion | + * | otherwise | "candidate" | a fresh claim sits at the 0.5 prior, exactly newLesson() | + * + * Count/date fields are rebuilt from the evidence log: evidenceCount = valid confirm + * outcomes, contradictionCount = valid contradict outcomes, lastConfirmedDay = latest + * confirm t (else the mint day), createdDay = the mint day (provenance.t). + * @param {any} claim a materialized lesson claim (ledger_store.loadClaims view) + * @param {number} [nowDay] epoch day used for the val() decay clock + * @returns {object} a legacy-shaped lesson + */ +export function claimToLesson(claim, nowDay = 0) { + const body = claim.body ?? {}; + const evidence = (claim.evidence ?? []).filter(validOutcome); + const confirms = evidence.filter((e) => e.result === "confirm"); + const contradictions = evidence.length - confirms.length; + const createdDay = claim.provenance?.t ?? 0; + const v = val(claim, nowDay); + const status = claim.tombstone + ? "retired" + : v >= 0.6 + ? "active" + : v < 0.45 && contradictions >= 1 + ? "quarantined" + : "candidate"; + return { + id: String(claim.provenance?.task || "") || `lsn_${claim.id.slice(0, 8)}`, + trigger: { + files: body.trigger?.files ?? [], + symbols: body.trigger?.symbols ?? [], + keywords: body.trigger?.keywords ?? [], + action: body.trigger?.action || undefined, + }, + scope: claim.scope?.level ?? "repo", + whatWentWrong: body.whatWentWrong ?? "", + correctedBehavior: body.correctedBehavior ?? "", + evidenceCount: confirms.length, + contradictionCount: contradictions, + quarantineReconfirms: 0, + status, + createdDay, + lastConfirmedDay: confirms.length ? Math.max(...confirms.map((e) => e.t ?? 0)) : createdDay, + halfLifeDays: DEFAULT_HALF_LIFE_DAYS, + // The claim id is the audit pointer back to the ledger (`forge ledger blame `). + provenance: { episodes: [], signals: [], claim: claim.id }, + }; +} + +/** Every lesson claim of the repo ledger, mapped to legacy shape. [] when there is no + * ledger or it is unreadable (best-effort — hooks call this on every event). */ +export function ledgerLessons(root, nowDay = 0) { + try { + return loadClaims(repoLedger(root)) + .filter((c) => c.kind === "lesson") + .map((c) => claimToLesson(c, nowDay)); + } catch { + return []; + } +} + +// Within the ledger, two claims can map to one legacy id (a distillation supersedes its +// template claim; both carry the same provenance.task). Prefer the live one, then the +// most recently confirmed; ties keep the first in claim-id order (loadClaims is sorted), +// so the pick is deterministic across replicas. +const preferred = (a, b) => { + if ((a.status === "retired") !== (b.status === "retired")) return a.status === "retired" ? b : a; + return b.lastConfirmedDay > a.lastConfirmedDay ? b : a; +}; + +/** + * The merged lesson read: legacy `load(root)` ∪ ledger lessons, deduped by the LEGACY + * id with the legacy file winning — the legacy store is still the canonical local + * state (recordMistake/confirm/contradict edit it), so a local lesson's own shadow + * claim (same provenance.task) is invisible here, while a teammate's claim (different + * task, no local file) surfaces. Order is deterministic: legacy lessons in file order, + * then ledger-only lessons sorted by lesson id. + * @param {string} root + * @param {number} [nowDay] + * @returns {object[]} + */ +export function mergedLessons(root, nowDay = 0) { + const legacy = load(root); + const local = new Set(legacy.map((l) => l.id)); + const byId = new Map(); + for (const l of ledgerLessons(root, nowDay)) { + if (local.has(l.id)) continue; // legacy wins — the local file is canonical + const prev = byId.get(l.id); + byId.set(l.id, prev ? preferred(prev, l) : l); + } + return [...legacy, ...[...byId.values()].sort((a, b) => (a.id < b.id ? -1 : 1))]; +} + +/** Live (non-tombstoned) `fact` claims of one ledger as {slug, name, text} — the fact + * counterpart of ledgerLessons. Best-effort: [] when the ledger is missing/corrupt. */ +export function ledgerFacts(dir) { + try { + return loadClaims(dir) + .filter((c) => c.kind === "fact" && !c.tombstone) + .map((c) => ({ + slug: slug(c.body?.name ?? "") || "fact", + name: String(c.body?.name ?? ""), + text: String(c.body?.text ?? ""), + })); + } catch { + return []; + } +} + +/** + * Merged fact slugs for a file store + its ledger: the file store's slugs win on + * collision (a stored file is the canonical local value; shadowFact tombstones stale + * same-name claims, so a surviving collision means the file is newer). Sorted, unique. + * @param {string[]} fileSlugs slugs of the file-backed facts (already the store's truth) + * @param {string} dir the ledger directory that shadows this store + * @returns {string[]} + */ +export function mergeFactSlugs(fileSlugs, dir) { + const seen = new Set(fileSlugs); + for (const f of ledgerFacts(dir)) seen.add(f.slug); + return [...seen].sort(); +} diff --git a/src/recall.js b/src/recall.js index 456f06c..816a11c 100644 --- a/src/recall.js +++ b/src/recall.js @@ -10,6 +10,10 @@ import { join } from "node:path"; // imported it from (lessons_store, guards, tests). See ledger.js for the precision // rationale (credential formats + key-assigned-to-value, never a bare English mention). import { SECRET_RE } from "./ledger.js"; +// The merged read helper (P2 read flip). Import cycle note: ledger_read → lessons_store +// → recall is function-level only (no module-eval use on either side), so ESM resolves +// it safely — same pattern as lessons_store's own SECRET_RE import from here. +import { mergeFactSlugs } from "./ledger_read.js"; export { SECRET_RE }; @@ -47,7 +51,10 @@ export function readFact(store, slug) { return m ? { name: m[1].trim(), text: m[2].trim() } : { name: slug, text: raw.trim() }; } -export function list(store) { +/** File-backed fact slugs ONLY — the store the write path (add/consolidate) manages. + * The ledger bridge reconciles against THIS list: a merged teammate fact has no file, + * and treating it as "deleted from the store" would tombstone it away. */ +export function listStored(store) { const dir = factsDir(store); if (!existsSync(dir)) return []; return readdirSync(dir) @@ -56,6 +63,13 @@ export function list(store) { .sort(); } +/** The read view (P2 read flip): file facts ∪ live facts from the personal ledger this + * store shadows into (`/ledger` — see `forge recall add`). A file wins on slug + * collision (the file is the canonical local value). */ +export function list(store) { + return mergeFactSlugs(listStored(store), join(store, "ledger")); +} + export function reindex(store) { const items = list(store); mkdirSync(store, { recursive: true }); diff --git a/src/reuse.js b/src/reuse.js index fabeb9a..63df5d4 100644 Binary files a/src/reuse.js and b/src/reuse.js differ diff --git a/src/route.js b/src/route.js index 047604f..5e2e62e 100644 --- a/src/route.js +++ b/src/route.js @@ -8,10 +8,10 @@ import { adjudicate, asText, buildRunner, llmEnabled } from "./adjudicate.js"; import { matchingLessons } from "./cortex.js"; import { gitChurn, grepFanout } from "./cortex_features.js"; import { recordRoute } from "./cost_report.js"; -import { load as loadLessons } from "./lessons_store.js"; +import { mergedLessons } from "./ledger_read.js"; import { MODELS } from "./model_tiers.js"; import { preflightRepo, referencedEntities } from "./preflight.js"; -import { clamp01, contentHash } from "./util.js"; +import { clamp01, contentHash, epochDay } from "./util.js"; // Weights sum to 1. Each raw signal is normalized by the point where it reads as "complex". @@ -181,7 +181,8 @@ export function routeTask( const { symbols, files } = referencedEntities(task); const fanout = symbols.reduce((m, sym) => Math.max(m, grepFanout(root, sym)), 0); const churn = files.reduce((m, f) => Math.max(m, gitChurn(root, f)), 0); - const pastMistakes = matchingLessons(loadLessons(root), { + // Merged view (P2 read flip): teammate lessons raise past-mistake density here too. + const pastMistakes = matchingLessons(mergedLessons(root, epochDay()), { files, symbols, }).length; diff --git a/src/substrate.js b/src/substrate.js index b988132..2eeb6d5 100644 --- a/src/substrate.js +++ b/src/substrate.js @@ -12,7 +12,7 @@ import { assemble as assembleContext } from "./context.js"; import { matchingLessons } from "./cortex.js"; import { recordGate } from "./cost_report.js"; import { leanRepo } from "./lean.js"; -import { load as loadLessons } from "./lessons_store.js"; +import { mergedLessons } from "./ledger_read.js"; import { clarifyBlock, preflightRepo, referencedEntities } from "./preflight.js"; import { reusePeek, reuseQuery } from "./reuse.js"; import { meterRoute, routeTask } from "./route.js"; @@ -264,7 +264,8 @@ export function substrateCheck( const scope = scopedFiles.length ? decompose(root, scopedFiles) : { clusters: [], independentGroups: 0 }; - const lessons = matchingLessons(loadLessons(root), { + // Merged view (P2 read flip): a teammate's merged lesson counts in the advisory too. + const lessons = matchingLessons(mergedLessons(root, epochDay()), { files: scopedFiles, symbols: entities.symbols, }); diff --git a/src/uivisual.js b/src/uivisual.js new file mode 100644 index 0000000..4565683 --- /dev/null +++ b/src/uivisual.js @@ -0,0 +1,334 @@ +// forge uicheck visual — the Playwright visual loop (P6 §5, +// docs/plans/substrate-v2/07-ui-quality-gate.md). The static fingerprint parses +// SOURCE CSS/classes; it cannot see rendered reality — cascade results, computed +// values, runtime theming. This module renders the page in a real browser (when one +// is available) and feeds the COMPUTED styles of every visible element through the +// SAME fingerprint pipeline as `uicheck design`, so the two gates share one geometry. +// +// ADR-0005 discipline: playwright is an OPTIONAL tier. package.json stays +// dependency-free; the module resolves playwright-core/playwright dynamically (or an +// explicit FORGE_PLAYWRIGHT module path) and degrades to a clear "skipped" note — +// absence is never a crash and never a gate failure. +// +// Security: a CLI that fetches arbitrary URLs by default is an exfiltration hazard +// (a hook could be steered into beaconing repo contents via a crafted URL). http(s) +// targets are therefore refused unless the host is loopback — `--remote` is the +// explicit human override. +import { existsSync, mkdirSync } from "node:fs"; +import { createRequire } from "node:module"; +import { isAbsolute, join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { BRAND } from "./brand.js"; +import { + activeTasteStyle, + fingerprintText, + loadProjectFingerprint, + loadTasteProfile, + profileChecks, + scaleChecks, + UI_GATE_DEFAULTS, + uiGate, +} from "./uifingerprint.js"; + +// --------------------------------------------------------------------------- +// Optional-tier resolution — dynamic, never a package.json dependency. +// --------------------------------------------------------------------------- + +/** + * Resolve a playwright runtime: FORGE_PLAYWRIGHT (an explicit module path or bare + * name — how tests and scratch installs point at an out-of-tree copy) wins and is + * authoritative when set; otherwise try `playwright-core` then `playwright`. Returns + * the module (something with `.chromium.launch`) or null — never throws. Browser + * binaries follow playwright's own PLAYWRIGHT_BROWSERS_PATH handling. + * @returns {Promise<{chromium:{launch:Function}}|null>} + */ +export async function resolvePlaywright() { + const explicit = process.env.FORGE_PLAYWRIGHT; + const candidates = explicit ? [explicit] : ["playwright-core", "playwright"]; + for (const spec of candidates) { + try { + let entry = spec; + if (spec.includes("/") || spec.includes("\\")) { + // A path (to the package dir or its entry file): resolve through require's + // algorithm so a directory's package.json "main" is honored, then import as + // a file URL (playwright ships CJS; the default export is module.exports). + entry = pathToFileURL( + createRequire(import.meta.url).resolve(isAbsolute(spec) ? spec : resolve(spec)), + ).href; + } + const mod = await import(entry); + const pw = mod?.default?.chromium ? mod.default : mod; + if (typeof pw?.chromium?.launch === "function") return pw; + } catch {} + } + return null; +} + +// --------------------------------------------------------------------------- +// Target resolution — the security guard lives here, BEFORE any browser exists. +// --------------------------------------------------------------------------- + +const LOOPBACK = new Set(["localhost", "127.0.0.1", "[::1]", "::1", "0.0.0.0"]); +const isLoopbackHost = (host) => + LOOPBACK.has(host) || + host.endsWith(".localhost") || + /^127\.\d{1,3}\.\d{1,3}\.\d{1,3}$/.test(host); + +/** + * Normalize a CLI target to a navigable URL. Local paths become file:// URLs; + * http(s) URLs are allowed only for loopback hosts unless `remote` is set — the + * refusal is the default so no hook or agent can quietly point the gate at the + * network. Never throws. + * @param {string} target + * @param {{remote?:boolean, cwd?:string}} [opts] + * @returns {{ok:true, url:string}|{ok:false, reason:string}} + */ +export function resolveTarget(target, { remote = false, cwd = process.cwd() } = {}) { + const t = String(target ?? "").trim(); + if (!t) return { ok: false, reason: "no target given" }; + if (/^https?:\/\//i.test(t)) { + let u; + try { + u = new URL(t); + } catch { + return { ok: false, reason: `unparseable URL: ${t}` }; + } + if (!remote && !isLoopbackHost(u.hostname)) + return { + ok: false, + reason: `refusing non-local URL ${u.origin} — fetching arbitrary URLs by default is an exfiltration hazard; pass --remote to override deliberately`, + }; + return { ok: true, url: u.href }; + } + if (/^file:\/\//i.test(t)) return { ok: true, url: t }; + if (/^[a-z][a-z0-9+.-]*:/i.test(t)) + return { ok: false, reason: `unsupported URL scheme: ${t} (use a file path or http(s))` }; + const p = isAbsolute(t) ? t : join(cwd, t); + if (!existsSync(p)) return { ok: false, reason: `no such file: ${t}` }; + return { ok: true, url: pathToFileURL(p).href }; +} + +// --------------------------------------------------------------------------- +// Computed styles → the shared fingerprint vector. +// --------------------------------------------------------------------------- + +/** + * @typedef {{color?:string, backgroundColor?:string, margin?:string, padding?:string, + * gap?:string, fontFamily?:string, borderRadius?:string, boxShadow?:string}} ComputedRecord + */ + +// Runs INSIDE the page (a string so the repo typechecks without DOM lib types): +// walk visible elements and lift exactly the computed properties the fingerprint +// measures. Values arrive fully resolved — the cascade, var() indirection, and +// runtime theming have already happened, which is the whole point of the loop. +// Two artifact classes must NOT leak into the vector, because they are layout +// residue rather than authored design decisions: +// - `margin: auto` — getComputedStyle returns the USED value (a centered 1180px +// container at 1280 reports "0px 50px"), so margins go through the Typed OM +// (computedStyleMap), which keeps `auto` symbolic; auto sides are dropped. +// - elements the page never painted (e.g.