From aac812162fc9f5a33437b171646e5b74c31774c9 Mon Sep 17 00:00:00 2001 From: "jan (via bardioc)" Date: Wed, 3 Jun 2026 12:11:59 +0000 Subject: [PATCH 1/2] docs: dn_redis is Redis-protocol EMULATION, not a Redis service dep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarifies that the dn_redis module in lance-graph-cognitive is a PROTOCOL/SHAPE adapter, not a service dependency. Three valid backends: external Redis (a); in-binary OGIT class view + DataFusion query over Lance (b, the zero-boundary target); anything that produces the documented ada:dn key shape (c). Same pattern as FalkorDB (formerly RedisGraph) and KuzuDB — talk Redis without BEING Redis. Adopters reading the dn_redis module + the lab-vs-canonical discipline docs sometimes conclude that an external Redis server is required; this doc clarifies that 'protocol-only adapter' means the SHAPE is the contract, not a service binding. Doc-only; new file at docs/DN_REDIS_EMULATION.md. Pairs naturally with the cluster-asymmetry + append-only-Raft docs (PR #452/#453/ #454) — all three address common mis-reads of lance-graph idioms through a legacy lens. Provenance: - Surfaced by: bardioc PR #15 conversation thread - Draft outline: bardioc/UPSTREAM_LANCE_GRAPH_PR_CANDIDATES.md T1.5 - Sister doc: APPEND_ONLY_RAFT_DOVETAIL.md (PR #452/#454) - Sister doc: CLUSTER_ASYMMETRY.md (PR #453/#454) --- docs/DN_REDIS_EMULATION.md | 61 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 docs/DN_REDIS_EMULATION.md diff --git a/docs/DN_REDIS_EMULATION.md b/docs/DN_REDIS_EMULATION.md new file mode 100644 index 00000000..2f24fbe1 --- /dev/null +++ b/docs/DN_REDIS_EMULATION.md @@ -0,0 +1,61 @@ +# dn_redis is Redis-protocol EMULATION, not a Redis dependency + +## TL;DR + +`lance-graph-cognitive::container_bs::dn_redis` is a **protocol/shape adapter** for HHTL-keyed hot lookups. It is NOT a dependency on an external Redis server. Adopters reading the module and the lab-vs-canonical-surface discipline doc correctly identify that `dn_redis` is "protocol-only"; they sometimes incorrectly conclude that an external Redis server is therefore required. + +This doc clarifies that **"protocol-only adapter" means the SHAPE is the contract — the backend is open**. Three valid backends: + +- (a) an external Redis cluster (traditional hybrid deployment) +- (b) **in-binary**: an OGIT class view + DataFusion query over the local Lance dataset, projected through the `dn_redis` adapter shape — Redis-protocol responses emitted from your own data, **no Redis service required** +- (c) anything else that produces the `ada:dn:{hex}` / `ada:spine:{hex}` key shape with the documented operations + +## Audience + +Anyone reading `lance-graph-cognitive::container_bs::dn_redis` (the 253-line protocol-only adapter) and wondering whether they need to deploy a Redis server. Also anyone reading SPRINT plan docs that reference `dn_redis` as "protocol-only" without further qualification. + +## The pattern (FalkorDB / KuzuDB precedent) + +[FalkorDB](https://www.falkordb.com/) (formerly RedisGraph) and [KùzuDB](https://kuzudb.com/) both "talk Redis" — they expose the Redis wire protocol — while the storage and execution underneath are completely different. They are NOT Redis; they emulate the wire protocol so existing Redis clients work. + +`dn_redis` follows the same pattern. It defines: + +- Key conventions: `ada:dn:{hex}` for individual nodes, `ada:spine:{hex}` for spine entries +- Walk-to-root operation shape (`MGET ada:dn:{ancestor1} ada:dn:{ancestor2} ...`) +- Children/subtree patterns (`KEYS ada:dn:{prefix}??` for one-level; `KEYS ada:dn:{prefix}*` for subtree) +- Pipeline operations (`RedisPipeline` + `RedisCommand` enum) + +These are **shapes**, not service dependencies. The shape can be served by any backend that produces the right responses. + +## Why this matters + +Consumers reading the `dn_redis` module + the surrounding lance-graph discipline docs (lab-vs-canonical-surface, the "protocol-only adapter" annotation that appears across the brutal-fix planning material in downstream consumers like `AdaWorldAPI/bardioc`) correctly identify that it is protocol-only. They incorrectly conclude that an external Redis server is therefore required. This doc clarifies that "protocol-only adapter" means the SHAPE is the contract — the backend is open. + +For a single-binary deployment (e.g. `bardioc/substrate-b`) the natural backend is option (b): the in-binary OGIT class view + DataFusion query over Lance. This produces Redis-protocol-shaped responses without running a Redis server. The single-binary "zero application-level boundaries" deployment shape (see `docs/APPEND_ONLY_RAFT_DOVETAIL.md` for the broader argument) survives the addition of HHTL-keyed hot lookups precisely because `dn_redis` is emulation, not a service. + +For consumers that already operate a Redis cluster, option (a) is fine. For consumers that prefer purpose-built graph engines exposing Redis wire protocol, FalkorDB and KùzuDB demonstrate that the pattern is well-established. + +## Recommended consumer pattern + +For the in-binary backend (option b): + +1. Implement a `dn_redis::Backend` impl (or whatever the trait name resolves to in the adapter's surface) that intercepts the documented operations +2. Project them through DataFusion queries over the local Lance dataset +3. Wrap the responses in the Redis-protocol shape the adapter expects +4. Mount the backend at the same call sites that a Redis client would be mounted + +See the bardioc B1 substrate-b reference for a worked example once that consumer's reference-implementation doc lands (separate proposal under upstream-contributions T2.1). + +## What this doc does NOT claim + +- **It does not claim FalkorDB or KùzuDB use `dn_redis`.** Those are independent products demonstrating the wire-protocol-emulation pattern; `dn_redis` borrows the pattern but is its own thing. +- **It does not claim option (a) is wrong.** External Redis is a perfectly valid backend for `dn_redis`. The point is that adopters are not LOCKED IN to it. +- **It does not specify the in-binary backend implementation.** That's consumer code; this doc explains why such consumer code is well-formed, not how to write it. + +## Cross-references + +- `lance-graph-cognitive::container_bs::dn_redis` — the protocol adapter +- `lab-vs-canonical-surface.md` — the canonical-vs-lab discipline that frames adapters +- FalkorDB project (Redis wire protocol emulation precedent) +- KùzuDB project (similar pattern) +- `AdaWorldAPI/bardioc` PR #15 conversation thread — where this clarification surfaced From 6ca9f86834e33ecdbb2808c328a43ddb8a1518a0 Mon Sep 17 00:00:00 2001 From: "jan (via bardioc)" Date: Wed, 3 Jun 2026 12:46:35 +0000 Subject: [PATCH 2/2] fix(codex+coderabbit): dn_redis is a key-shape protocol, not wire-protocol emulation Four review findings on PR #455 walked back, all real: Codex P2 1 - Redis Cluster is not plug-and-play. The walk_to_root / mget_dns operations use MGET, but the current key layout (ada:dn:{hex}) has no shared hash tag. Redis Cluster requires multi-key ops to belong to the same hash slot. So external Redis CLUSTER is NOT a valid backend without re-keying with hash tags. Added explicit caveat. Codex P2 2 - dn_redis is NOT wire-protocol emulation. The module exports Rust String keys + a RedisCommand enum + RedisPipeline. There is no RESP encoder, no parser, no listener, no command executor. Existing Redis clients cannot communicate with it. The FalkorDB / KuzuDB precedent does NOT apply (those projects implement the actual RESP wire protocol; dn_redis does not). Reframed the doc to "key-shape protocol + Rust command type model" - which is what the module actually is. Codex P2 3 - There is no dn_redis::Backend trait to implement. The "Recommended consumer pattern" first step cited a nonexistent API. Rewrote the section to show what consumer code actually has to provide: a struct + execute function that takes RedisCommand and runs it against the chosen backend. The example is illustrative (not a shipped trait). CodeRabbit minor - Same critique class. The speculative API wording is gone. Net effect: the doc now accurately describes what dn_redis IS (key-shape + command type model) and what adopters have to provide (an executor). Two valid backend categories enumerated honestly: (A) standalone Redis with optional hash-tag re-keying for clustered deployments; (B) in-binary executor over Lance via DataFusion. File renamed: DN_REDIS_EMULATION.md -> DN_REDIS_KEY_SHAPE_PROTOCOL.md to match the corrected framing. Provenance: codex P2 + coderabbit minor on PR #455 commit aac81216. --- docs/DN_REDIS_EMULATION.md | 61 ------------------- docs/DN_REDIS_KEY_SHAPE_PROTOCOL.md | 92 +++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 61 deletions(-) delete mode 100644 docs/DN_REDIS_EMULATION.md create mode 100644 docs/DN_REDIS_KEY_SHAPE_PROTOCOL.md diff --git a/docs/DN_REDIS_EMULATION.md b/docs/DN_REDIS_EMULATION.md deleted file mode 100644 index 2f24fbe1..00000000 --- a/docs/DN_REDIS_EMULATION.md +++ /dev/null @@ -1,61 +0,0 @@ -# dn_redis is Redis-protocol EMULATION, not a Redis dependency - -## TL;DR - -`lance-graph-cognitive::container_bs::dn_redis` is a **protocol/shape adapter** for HHTL-keyed hot lookups. It is NOT a dependency on an external Redis server. Adopters reading the module and the lab-vs-canonical-surface discipline doc correctly identify that `dn_redis` is "protocol-only"; they sometimes incorrectly conclude that an external Redis server is therefore required. - -This doc clarifies that **"protocol-only adapter" means the SHAPE is the contract — the backend is open**. Three valid backends: - -- (a) an external Redis cluster (traditional hybrid deployment) -- (b) **in-binary**: an OGIT class view + DataFusion query over the local Lance dataset, projected through the `dn_redis` adapter shape — Redis-protocol responses emitted from your own data, **no Redis service required** -- (c) anything else that produces the `ada:dn:{hex}` / `ada:spine:{hex}` key shape with the documented operations - -## Audience - -Anyone reading `lance-graph-cognitive::container_bs::dn_redis` (the 253-line protocol-only adapter) and wondering whether they need to deploy a Redis server. Also anyone reading SPRINT plan docs that reference `dn_redis` as "protocol-only" without further qualification. - -## The pattern (FalkorDB / KuzuDB precedent) - -[FalkorDB](https://www.falkordb.com/) (formerly RedisGraph) and [KùzuDB](https://kuzudb.com/) both "talk Redis" — they expose the Redis wire protocol — while the storage and execution underneath are completely different. They are NOT Redis; they emulate the wire protocol so existing Redis clients work. - -`dn_redis` follows the same pattern. It defines: - -- Key conventions: `ada:dn:{hex}` for individual nodes, `ada:spine:{hex}` for spine entries -- Walk-to-root operation shape (`MGET ada:dn:{ancestor1} ada:dn:{ancestor2} ...`) -- Children/subtree patterns (`KEYS ada:dn:{prefix}??` for one-level; `KEYS ada:dn:{prefix}*` for subtree) -- Pipeline operations (`RedisPipeline` + `RedisCommand` enum) - -These are **shapes**, not service dependencies. The shape can be served by any backend that produces the right responses. - -## Why this matters - -Consumers reading the `dn_redis` module + the surrounding lance-graph discipline docs (lab-vs-canonical-surface, the "protocol-only adapter" annotation that appears across the brutal-fix planning material in downstream consumers like `AdaWorldAPI/bardioc`) correctly identify that it is protocol-only. They incorrectly conclude that an external Redis server is therefore required. This doc clarifies that "protocol-only adapter" means the SHAPE is the contract — the backend is open. - -For a single-binary deployment (e.g. `bardioc/substrate-b`) the natural backend is option (b): the in-binary OGIT class view + DataFusion query over Lance. This produces Redis-protocol-shaped responses without running a Redis server. The single-binary "zero application-level boundaries" deployment shape (see `docs/APPEND_ONLY_RAFT_DOVETAIL.md` for the broader argument) survives the addition of HHTL-keyed hot lookups precisely because `dn_redis` is emulation, not a service. - -For consumers that already operate a Redis cluster, option (a) is fine. For consumers that prefer purpose-built graph engines exposing Redis wire protocol, FalkorDB and KùzuDB demonstrate that the pattern is well-established. - -## Recommended consumer pattern - -For the in-binary backend (option b): - -1. Implement a `dn_redis::Backend` impl (or whatever the trait name resolves to in the adapter's surface) that intercepts the documented operations -2. Project them through DataFusion queries over the local Lance dataset -3. Wrap the responses in the Redis-protocol shape the adapter expects -4. Mount the backend at the same call sites that a Redis client would be mounted - -See the bardioc B1 substrate-b reference for a worked example once that consumer's reference-implementation doc lands (separate proposal under upstream-contributions T2.1). - -## What this doc does NOT claim - -- **It does not claim FalkorDB or KùzuDB use `dn_redis`.** Those are independent products demonstrating the wire-protocol-emulation pattern; `dn_redis` borrows the pattern but is its own thing. -- **It does not claim option (a) is wrong.** External Redis is a perfectly valid backend for `dn_redis`. The point is that adopters are not LOCKED IN to it. -- **It does not specify the in-binary backend implementation.** That's consumer code; this doc explains why such consumer code is well-formed, not how to write it. - -## Cross-references - -- `lance-graph-cognitive::container_bs::dn_redis` — the protocol adapter -- `lab-vs-canonical-surface.md` — the canonical-vs-lab discipline that frames adapters -- FalkorDB project (Redis wire protocol emulation precedent) -- KùzuDB project (similar pattern) -- `AdaWorldAPI/bardioc` PR #15 conversation thread — where this clarification surfaced diff --git a/docs/DN_REDIS_KEY_SHAPE_PROTOCOL.md b/docs/DN_REDIS_KEY_SHAPE_PROTOCOL.md new file mode 100644 index 00000000..32c37e6e --- /dev/null +++ b/docs/DN_REDIS_KEY_SHAPE_PROTOCOL.md @@ -0,0 +1,92 @@ +# dn_redis is a key-shape protocol + Rust command type model — not a Redis service dep + +## TL;DR + +`lance-graph-cognitive::container_bs::dn_redis` is a **key-shape protocol definition + a Rust-side command type model** for HHTL-keyed hot lookups. It is NOT: + +- A network-protocol implementation (no RESP encoder/parser, no listener, no command executor) +- A drop-in replacement that lets existing Redis clients talk to lance-graph +- A wire-protocol emulator (the FalkorDB / KùzuDB precedent does NOT apply here — those projects implement the Redis RESP protocol; dn_redis does not) + +What dn_redis IS: + +- A **Rust API** that exports key-construction helpers (`dn_key`, `spine_key`, `walk_to_root_keys`, `children_pattern`, `subtree_pattern`) producing `String` keys like `ada:dn:{hex}` and `ada:spine:{hex}` +- A **command type model** (`RedisCommand` enum + `RedisPipeline` struct) that adopters can populate to describe the operations they need to execute +- A **serde layer** for `CogRecord` payloads (`cog_record_to_bytes`, `cog_record_from_bytes`) + +## What is missing — and what adopters have to provide + +The module exports the SHAPE of the operations; adopters provide the EXECUTOR. There is no `Backend` trait, no listener, no `connect()` function, no `execute()` method on `RedisPipeline` — those are what consumer code must implement. + +To use dn_redis, a consumer must: + +1. Decide on a backend (see below) +2. Write an executor that takes `RedisPipeline` (or individual `RedisCommand` values) and runs them against the chosen backend +3. Mount the executor at the call sites where the consumer's cognitive substrate needs hot-key lookups + +This doc proposes formalizing two valid backend categories so adopters do not have to discover them by reverse-engineering the call sites. + +## Two valid backend categories + +### Backend A: standalone Redis (or Redis with hash-tag re-keying) + +A consumer can run a Redis server and pipe `RedisCommand`s to it via any Redis client (`redis-rs`, `fred`, etc.). The consumer writes the executor: receive `RedisCommand`, translate to the client's API, return results. + +**Important caveat (per codex P2 review on PR #455):** the documented `walk_to_root` operation uses `MGET ada:dn:{ancestor1} ada:dn:{ancestor2} ...`. In a **Redis Cluster** deployment, `MGET` requires all keys to belong to the same hash slot — which means they must share a `{hash_tag}` substring per the [Redis Cluster specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/). The current key layout `ada:dn:{hex}` does NOT include a hash tag, so cluster `MGET` will return `CROSSSLOT Keys in request don't hash to the same slot` errors. + +For Redis Cluster to be a valid backend, EITHER: +- (i) The key layout must be re-shaped to include a shared hash tag for keys that are co-queried (e.g. `ada:dn:{root_basin}:0102...` so all descendants of a basin hash to one slot) +- (ii) The consumer must split multi-key operations into per-key calls (defeats the pipeline) +- (iii) The consumer runs standalone Redis (not Cluster) + +This caveat is the practical reason most consumers should treat dn_redis's key shape as "designed for standalone Redis OR for an in-binary executor", not for a clustered Redis deployment without re-shaping. + +### Backend B: in-binary executor over Lance via DataFusion + +A consumer can implement an executor that takes the same `RedisCommand` shape and runs it against the local Lance dataset via DataFusion queries. The result is Redis-protocol-shaped responses (via the Rust types — not wire-protocol bytes) emitted from the consumer's own data, with **no Redis service required**. + +What the consumer writes (per codex P2 #3 — there is no shipped trait to implement; this IS new consumer code): + +```rust +// Consumer code, NOT lance-graph code +struct LanceBackend { /* ... */ } + +impl LanceBackend { + fn execute(&self, cmd: RedisCommand) -> Result { + match cmd { + RedisCommand::Get(key) => { + let dn = self.parse_dn_from_key(&key)?; + let row = self.lance.read_by_dn(dn).await?; + Ok(RedisValue::Bulk(cog_record_to_bytes(&row))) + } + RedisCommand::Mget(keys) => { /* batch lookup over Lance */ } + RedisCommand::Keys(pattern) => { /* DataFusion scan filtered by prefix */ } + // ... etc per the enum's variants + } + } +} +``` + +This is "Redis-shape over Lance" — the consumer projects Lance results through the command-result type. There is no protocol parsing because no wire protocol is involved; everything is in-process Rust types. + +## What this doc does NOT claim + +- **It does not claim FalkorDB or KùzuDB use dn_redis.** Those are independent products that implement the actual Redis RESP wire protocol; dn_redis does not implement RESP. The earlier version of this doc cited them as a precedent for "talk Redis without being Redis" — that framing was wrong. The honest framing is more constrained: dn_redis provides the key-shape and command-type contract that a consumer can EITHER pipe to a real Redis (standalone) OR execute in-binary against their own data. +- **It does not document a network protocol.** There is no listener, parser, or executor shipped by lance-graph for dn_redis. Calling it "wire-protocol emulation" (as a prior version of this doc did) misleads consumers about what is implemented vs what they must implement themselves. +- **It does not claim Redis Cluster is plug-and-play.** Per the caveat above, the current key shape is incompatible with cluster `MGET` slot routing. + +## Why this matters + +For single-binary deployments (e.g. `AdaWorldAPI/bardioc/substrate-b`) the natural backend is option (B): an in-binary executor over Lance. This means "zero application-level boundaries within substrate-b" (per the PR #452 / #454 append-only-Raft doc) survives the addition of HHTL-keyed hot lookups precisely because dn_redis is a TYPE MODEL, not a service dependency. + +For consumers who already operate a standalone Redis (with no clustering or with cluster + re-keyed hash tags), option (A) is straightforward: write the executor that translates `RedisCommand` to a Redis client's API. + +The earlier version of this doc framed dn_redis as wire-protocol emulation; codex review on PR #455 caught the inaccuracy. The corrected framing is more constrained but more honest: dn_redis is the SHAPE, the consumer provides the EXECUTION. + +## Cross-references + +- `lance-graph-cognitive::container_bs::dn_redis` — the key-shape protocol + command type model +- `lab-vs-canonical-surface.md` — the canonical-vs-lab discipline that frames adapters +- Companion docs `APPEND_ONLY_RAFT_DOVETAIL.md` + `CLUSTER_ASYMMETRY.md` (PR #452 / #453 / #454 — merged) +- `AdaWorldAPI/bardioc` PR #15 conversation thread (where this doc + the corrections originated) +- [Redis Cluster specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/) for the hash-slot constraint