From 17e2bc9fc27c9fd3a8231c1b33b547866e824d63 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Thu, 4 Jun 2026 22:34:27 +0200 Subject: [PATCH 1/3] =?UTF-8?q?fix(knowledge):=20audit=20retention=20cavea?= =?UTF-8?q?t=20=E2=80=94=20Lance=20versions=20are=20retention-policy-gated?= =?UTF-8?q?,=20not=20by-construction-immutable=20(codex=20P1=20on=20#465)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §2.1 audit bullet: renamed from 'Immutable audit' to 'Audit (retention-policy- gated)'; explicit retention guidance (disable auto-cleanup OR tag versions OR route to separate sink); regulatory-grade audit requires external signed write-once sink — Lance versions alone NOT a substitute. §5.1 collapse pattern: renamed from 'Three OLD components collapse to one' to 'Two-and-a-half OLD components collapse to one'; non-regulatory audit (with retention configured) shares Lance versions; regulatory audit remains a separate concern. The three-primitives codification (E-SUBSTRATE-B-CAPABILITY-ROADMAP) survives. Multi-purpose-Lance-versions claim still load-bearing — what changes is the audit guarantee + the consumer-default guidance. Codex P1 finding on #465: Lance 7.0+ exposes Dataset::cleanup_old_versions + lance.auto_cleanup.*; following the original 'introduce no separate store' guidance could make historical audit reads disappear after cleanup. --- .claude/knowledge/old-stack-capability-parity.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/.claude/knowledge/old-stack-capability-parity.md b/.claude/knowledge/old-stack-capability-parity.md index bac5421d..cd7d007e 100644 --- a/.claude/knowledge/old-stack-capability-parity.md +++ b/.claude/knowledge/old-stack-capability-parity.md @@ -30,9 +30,9 @@ A single primitive — Lance versions — serves three distinct capabilities a s - **Point-in-time query** = `dataset.checkout_version(V_ref)` — pin an immutable snapshot at any version - **Time-series** = the version log itself — every commit is a versioned event with a timestamp -- **Immutable audit** = append-only by construction — versions never disappear; the log IS the audit trail +- **Audit (retention-policy-gated)** = append-only **at write time**, but Lance supports version cleanup (`Dataset::cleanup_old_versions` + the `lance.auto_cleanup.*` settings, Lance 7.0+). The version log is therefore **not guaranteed immutable without explicit retention policy**. For audit-class workloads, consumers MUST configure retention — either by disabling auto-cleanup on the dataset, tagging versions for retention, OR routing audit-class events to a separate append-only sink (signed write-once object store, regulator-grade audit ledger). For regulatory-grade *"cannot be deleted, cannot be manipulated"* guarantees, the external signed sink is **mandatory** — Lance versions alone are NOT a substitute. -This is the substrate-b efficiency claim: three capabilities, one primitive, no separate storage. The implications are captured in `STANDING_WAVE_ARCHITECTURE.md` §1 (in substrate-b consumer repos). +This is the substrate-b efficiency claim, with the audit caveat: three capabilities, one primitive — for non-regulatory audit, Lance versions + a retention policy serve. For regulatory audit, the external signed sink remains a separate concern (no claim made). The implications are captured in `STANDING_WAVE_ARCHITECTURE.md` §1 (in substrate-b consumer repos). ### 2.2 Per-element auth = palette256 + Hamming popcount on Binary16K @@ -108,9 +108,14 @@ Substrate-b consumers run dual-stack workload replay against this contract; the ## 5. Integration patterns that fall out of this shape -### 5.1 Three OLD components collapse to one when substrate-b is the target +### 5.1 Two-and-a-half OLD components collapse to one when substrate-b is the target -A consumer migrating from a separate Historisation layer + a separate time-series database + a separate signed-audit-log service replaces all three with Lance versions. This is the design-pattern claim; it follows from §2.1. Substrate-b consumers should NOT introduce separate stores for these three capabilities. +A consumer migrating from a separate Historisation layer + a separate time-series database replaces both with Lance versions outright — that part of the design-pattern claim follows from §2.1. **The audit case is conditional:** + +- For **non-regulatory** audit (operational logging, compliance-as-best-effort), Lance versions serve IF the retention policy is configured to preserve the audit window (auto-cleanup disabled, versions tagged for retention, or `cleanup_old_versions` not invoked on the audit dataset). Substrate-b consumers SHOULD make this policy explicit in their deployment config. +- For **regulatory-grade** audit ("cannot be deleted, cannot be manipulated" — the kind of guarantee required for compliance frameworks that mandate immutable audit trails), Lance versions alone are NOT a substitute. A separate signed write-once sink (object-storage with object-lock + signature, or a regulator-grade audit ledger) remains a separate concern; substrate-b doesn't claim to replace it. + +The honest framing: substrate-b collapses Historisation + TSDB into one primitive (Lance versions) and **shares storage with non-regulatory audit when retention is configured**, but does not displace a regulatory-grade audit sink. Treat regulatory audit as orthogonal. ### 5.2 ACL changes take effect immediately, by construction From 000c57799a9ebbda2365d3ba5fefaf049eaafcdd Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Thu, 4 Jun 2026 22:34:28 +0200 Subject: [PATCH 2/3] epiphany(E-AUDIT-RETENTION-CAVEAT): substrate-b consumer doc Lance-versions-as-audit claim corrected to retention-policy-gated (codex P1 on #465) --- .claude/board/EPIPHANIES.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 10b2d991..bb0053fd 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,22 @@ +## 2026-06-04 — E-AUDIT-RETENTION-CAVEAT — substrate-b consumer doc Lance-versions-as-audit claim was overstated; corrected to retention-policy-gated (codex P1 on #465) + +**Status:** CORRECTION (codex P1 on PR #465, 2026-06-04; merged + immediate follow-up correction per the no-silent-edit discipline — the FIX appends; the original epiphany E-SUBSTRATE-B-CAPABILITY-ROADMAP stands as the corrected reference now reads). + +**The overclaim (now corrected in `.claude/knowledge/old-stack-capability-parity.md`):** §2.1 said *"Immutable audit = append-only by construction — versions never disappear; the log IS the audit trail."* §5.1 followed up with *"Three OLD components collapse to one ... consumers should NOT introduce separate stores."* + +**The reality codex caught:** Lance 7.0+ exposes `Dataset::cleanup_old_versions` and `lance.auto_cleanup.*` settings. Old versions CAN be removed for storage management — the version log is therefore **not guaranteed immutable without explicit retention policy**. Consumers following the doc's guidance to drop their separate audit store could see historical audit reads disappear after cleanup. + +**The corrected framing:** +1. **Audit is retention-policy-gated**, not by-construction-immutable. For audit-class workloads, retention must be configured (disable auto-cleanup, tag versions, OR route to a separate append-only sink). +2. **Regulatory-grade audit** ("cannot be deleted, cannot be manipulated") requires a separate signed write-once sink — substrate-b doesn't claim to replace it. +3. **The collapse is two-and-a-half, not three.** Historisation + TSDB collapse outright; audit is conditional on retention policy + workload class (non-regulatory: yes with retention; regulatory: no, external sink still required). + +**Why this matters for the substrate-b shape:** the three-primitives codification (E-SUBSTRATE-B-CAPABILITY-ROADMAP) survives — the multi-purpose-Lance-versions claim is still load-bearing. What changes is the audit guarantee + the consumer-guidance default ("introduce no separate store"): now reads "introduce no separate store *for non-regulatory audit, with retention configured*; regulatory audit remains a separate concern." + +**Cross-ref:** PR #465 (merged) + the follow-up correction PR; `.claude/knowledge/old-stack-capability-parity.md` §2.1 + §5.1 (corrected); codex P1 finding (audit retention outside prunable Lance versions). + +--- + ## 2026-06-04 — E-SUBSTRATE-B-CAPABILITY-ROADMAP — three load-bearing NEW-stack primitives codified; consumer integration shape documented **Status:** FINDING (substrate-b consumer integration pattern, codified after the OGAR / surrealdb / ractor / lance-graph correspondence work converged on three structural primitives, 2026-06-04). From cd850fb130289cb0eb051880d560ca41e38eb1e9 Mon Sep 17 00:00:00 2001 From: AdaWorldAPI Date: Thu, 4 Jun 2026 22:34:31 +0200 Subject: [PATCH 3/3] =?UTF-8?q?agent-log:=20D-SUBSTRATE-B-CONSUMER-DOC-FIX?= =?UTF-8?q?=20=E2=80=94=20codex=20P1=20correction=20(audit=20retention=20c?= =?UTF-8?q?aveat)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude/board/AGENT_LOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index f9297b26..442c3875 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,18 @@ +## [Main thread / Opus, autoattended] D-SUBSTRATE-B-CONSUMER-DOC-FIX — codex P1 correction on PR #465 (audit retention caveat) + +**Branch:** doc/knowledge-old-stack-capability-parity-fix. Follow-up to merged PR #465; addresses codex P1 finding that §2.1 + §5.1 overclaimed Lance-versions-as-immutable-audit. + +**The overclaim corrected:** §2.1 said "versions never disappear"; §5.1 said "consumers should NOT introduce separate stores." Lance 7.0+ supports `Dataset::cleanup_old_versions` + `lance.auto_cleanup.*` — the version log is retention-policy-gated, not by-construction-immutable. Following the original guidance could make historical audit reads disappear after cleanup. + +**Corrections applied:** +- §2.1 audit bullet renamed from "Immutable audit" to "Audit (retention-policy-gated)"; explicit guidance: disable auto-cleanup OR tag versions OR route audit-class events to a separate append-only sink; regulatory-grade audit requires the external sink — Lance alone is NOT a substitute. +- §5.1 renamed from "Three OLD components collapse to one" to "Two-and-a-half OLD components collapse to one"; non-regulatory audit (with retention configured) shares Lance versions; regulatory audit remains a separate concern. +- The three-primitives codification (E-SUBSTRATE-B-CAPABILITY-ROADMAP) survives — the multi-purpose-Lance-versions claim is still load-bearing; only the audit guarantee + the consumer default change. + +**Outcome:** doc + EPIPHANIES + AGENT_LOG only, no code changes. Spot-check: the overclaim and the corrected text are both in §2.1/§5.1 of the diff. + +--- + ## [Main thread / Opus, autoattended] D-SUBSTRATE-B-CONSUMER-DOC — `.claude/knowledge/old-stack-capability-parity.md` SHIPPED (companion to lab-vs-canonical-surface + hollow-wire-failure-modes) **Branch:** doc/knowledge-old-stack-capability-parity (this PR). New `.claude/knowledge/` doc capturing the substrate-b consumer integration shape: the seven-capability composition (`lance-graph` storage + `surrealdb kv-lance` KV + Tantivy search + DataFusion OLAP + ractor actors + `LanceVersionWatcher` in-proc bus + external Zitadel IAM), the three load-bearing primitives (Lance versions as multi-purpose temporal; palette256+Hamming per-element auth; ractor-Actor + Lance-version-as-state-machine = Rubicon), and the capability roadmap (built / partial / not-yet) honest accounting.