diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index f9297b26..442c3875 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,18 @@ +## [Main thread / Opus, autoattended] D-SUBSTRATE-B-CONSUMER-DOC-FIX — codex P1 correction on PR #465 (audit retention caveat) + +**Branch:** doc/knowledge-old-stack-capability-parity-fix. Follow-up to merged PR #465; addresses codex P1 finding that §2.1 + §5.1 overclaimed Lance-versions-as-immutable-audit. + +**The overclaim corrected:** §2.1 said "versions never disappear"; §5.1 said "consumers should NOT introduce separate stores." Lance 7.0+ supports `Dataset::cleanup_old_versions` + `lance.auto_cleanup.*` — the version log is retention-policy-gated, not by-construction-immutable. Following the original guidance could make historical audit reads disappear after cleanup. + +**Corrections applied:** +- §2.1 audit bullet renamed from "Immutable audit" to "Audit (retention-policy-gated)"; explicit guidance: disable auto-cleanup OR tag versions OR route audit-class events to a separate append-only sink; regulatory-grade audit requires the external sink — Lance alone is NOT a substitute. +- §5.1 renamed from "Three OLD components collapse to one" to "Two-and-a-half OLD components collapse to one"; non-regulatory audit (with retention configured) shares Lance versions; regulatory audit remains a separate concern. +- The three-primitives codification (E-SUBSTRATE-B-CAPABILITY-ROADMAP) survives — the multi-purpose-Lance-versions claim is still load-bearing; only the audit guarantee + the consumer default change. + +**Outcome:** doc + EPIPHANIES + AGENT_LOG only, no code changes. Spot-check: the overclaim and the corrected text are both in §2.1/§5.1 of the diff. + +--- + ## [Main thread / Opus, autoattended] D-SUBSTRATE-B-CONSUMER-DOC — `.claude/knowledge/old-stack-capability-parity.md` SHIPPED (companion to lab-vs-canonical-surface + hollow-wire-failure-modes) **Branch:** doc/knowledge-old-stack-capability-parity (this PR). New `.claude/knowledge/` doc capturing the substrate-b consumer integration shape: the seven-capability composition (`lance-graph` storage + `surrealdb kv-lance` KV + Tantivy search + DataFusion OLAP + ractor actors + `LanceVersionWatcher` in-proc bus + external Zitadel IAM), the three load-bearing primitives (Lance versions as multi-purpose temporal; palette256+Hamming per-element auth; ractor-Actor + Lance-version-as-state-machine = Rubicon), and the capability roadmap (built / partial / not-yet) honest accounting. diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 10b2d991..bb0053fd 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,22 @@ +## 2026-06-04 — E-AUDIT-RETENTION-CAVEAT — substrate-b consumer doc Lance-versions-as-audit claim was overstated; corrected to retention-policy-gated (codex P1 on #465) + +**Status:** CORRECTION (codex P1 on PR #465, 2026-06-04; merged + immediate follow-up correction per the no-silent-edit discipline — the FIX appends; the original epiphany E-SUBSTRATE-B-CAPABILITY-ROADMAP stands as the corrected reference now reads). + +**The overclaim (now corrected in `.claude/knowledge/old-stack-capability-parity.md`):** §2.1 said *"Immutable audit = append-only by construction — versions never disappear; the log IS the audit trail."* §5.1 followed up with *"Three OLD components collapse to one ... consumers should NOT introduce separate stores."* + +**The reality codex caught:** Lance 7.0+ exposes `Dataset::cleanup_old_versions` and `lance.auto_cleanup.*` settings. Old versions CAN be removed for storage management — the version log is therefore **not guaranteed immutable without explicit retention policy**. Consumers following the doc's guidance to drop their separate audit store could see historical audit reads disappear after cleanup. + +**The corrected framing:** +1. **Audit is retention-policy-gated**, not by-construction-immutable. For audit-class workloads, retention must be configured (disable auto-cleanup, tag versions, OR route to a separate append-only sink). +2. **Regulatory-grade audit** ("cannot be deleted, cannot be manipulated") requires a separate signed write-once sink — substrate-b doesn't claim to replace it. +3. **The collapse is two-and-a-half, not three.** Historisation + TSDB collapse outright; audit is conditional on retention policy + workload class (non-regulatory: yes with retention; regulatory: no, external sink still required). + +**Why this matters for the substrate-b shape:** the three-primitives codification (E-SUBSTRATE-B-CAPABILITY-ROADMAP) survives — the multi-purpose-Lance-versions claim is still load-bearing. What changes is the audit guarantee + the consumer-guidance default ("introduce no separate store"): now reads "introduce no separate store *for non-regulatory audit, with retention configured*; regulatory audit remains a separate concern." + +**Cross-ref:** PR #465 (merged) + the follow-up correction PR; `.claude/knowledge/old-stack-capability-parity.md` §2.1 + §5.1 (corrected); codex P1 finding (audit retention outside prunable Lance versions). + +--- + ## 2026-06-04 — E-SUBSTRATE-B-CAPABILITY-ROADMAP — three load-bearing NEW-stack primitives codified; consumer integration shape documented **Status:** FINDING (substrate-b consumer integration pattern, codified after the OGAR / surrealdb / ractor / lance-graph correspondence work converged on three structural primitives, 2026-06-04). diff --git a/.claude/knowledge/old-stack-capability-parity.md b/.claude/knowledge/old-stack-capability-parity.md index bac5421d..cd7d007e 100644 --- a/.claude/knowledge/old-stack-capability-parity.md +++ b/.claude/knowledge/old-stack-capability-parity.md @@ -30,9 +30,9 @@ A single primitive — Lance versions — serves three distinct capabilities a s - **Point-in-time query** = `dataset.checkout_version(V_ref)` — pin an immutable snapshot at any version - **Time-series** = the version log itself — every commit is a versioned event with a timestamp -- **Immutable audit** = append-only by construction — versions never disappear; the log IS the audit trail +- **Audit (retention-policy-gated)** = append-only **at write time**, but Lance supports version cleanup (`Dataset::cleanup_old_versions` + the `lance.auto_cleanup.*` settings, Lance 7.0+). The version log is therefore **not guaranteed immutable without explicit retention policy**. For audit-class workloads, consumers MUST configure retention — either by disabling auto-cleanup on the dataset, tagging versions for retention, OR routing audit-class events to a separate append-only sink (signed write-once object store, regulator-grade audit ledger). For regulatory-grade *"cannot be deleted, cannot be manipulated"* guarantees, the external signed sink is **mandatory** — Lance versions alone are NOT a substitute. -This is the substrate-b efficiency claim: three capabilities, one primitive, no separate storage. The implications are captured in `STANDING_WAVE_ARCHITECTURE.md` §1 (in substrate-b consumer repos). +This is the substrate-b efficiency claim, with the audit caveat: three capabilities, one primitive — for non-regulatory audit, Lance versions + a retention policy serve. For regulatory audit, the external signed sink remains a separate concern (no claim made). The implications are captured in `STANDING_WAVE_ARCHITECTURE.md` §1 (in substrate-b consumer repos). ### 2.2 Per-element auth = palette256 + Hamming popcount on Binary16K @@ -108,9 +108,14 @@ Substrate-b consumers run dual-stack workload replay against this contract; the ## 5. Integration patterns that fall out of this shape -### 5.1 Three OLD components collapse to one when substrate-b is the target +### 5.1 Two-and-a-half OLD components collapse to one when substrate-b is the target -A consumer migrating from a separate Historisation layer + a separate time-series database + a separate signed-audit-log service replaces all three with Lance versions. This is the design-pattern claim; it follows from §2.1. Substrate-b consumers should NOT introduce separate stores for these three capabilities. +A consumer migrating from a separate Historisation layer + a separate time-series database replaces both with Lance versions outright — that part of the design-pattern claim follows from §2.1. **The audit case is conditional:** + +- For **non-regulatory** audit (operational logging, compliance-as-best-effort), Lance versions serve IF the retention policy is configured to preserve the audit window (auto-cleanup disabled, versions tagged for retention, or `cleanup_old_versions` not invoked on the audit dataset). Substrate-b consumers SHOULD make this policy explicit in their deployment config. +- For **regulatory-grade** audit ("cannot be deleted, cannot be manipulated" — the kind of guarantee required for compliance frameworks that mandate immutable audit trails), Lance versions alone are NOT a substitute. A separate signed write-once sink (object-storage with object-lock + signature, or a regulator-grade audit ledger) remains a separate concern; substrate-b doesn't claim to replace it. + +The honest framing: substrate-b collapses Historisation + TSDB into one primitive (Lance versions) and **shares storage with non-regulatory audit when retention is configured**, but does not displace a regulatory-grade audit sink. Treat regulatory audit as orthogonal. ### 5.2 ACL changes take effect immediately, by construction