diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cdd269a5..7e9efa55 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -88,7 +88,7 @@ jobs: .github/workflows/ci.yml|Cargo.toml|Cargo.lock|crates/*/Cargo.toml) run_rustfs_ci=true ;; crates/omnigraph/src/storage.rs) run_rustfs_ci=true ;; crates/omnigraph/src/db/manifest.rs|crates/omnigraph/src/db/manifest/*) run_rustfs_ci=true ;; - crates/omnigraph/tests/s3_storage.rs|crates/omnigraph/tests/write_cost_s3.rs|crates/omnigraph/tests/lance_surface_guards.rs|crates/omnigraph/tests/helpers/*) run_rustfs_ci=true ;; + crates/omnigraph/tests/s3_storage.rs|crates/omnigraph/tests/write_cost_s3.rs|crates/omnigraph/tests/lance_surface_guards.rs|crates/omnigraph/tests/memwal_enrollment_gate.rs|crates/omnigraph/tests/helpers/*) run_rustfs_ci=true ;; crates/omnigraph/src/table_store.rs|crates/omnigraph/src/instrumentation.rs) run_rustfs_ci=true ;; crates/omnigraph/src/runtime_cache.rs|crates/omnigraph/src/graph_index/*) run_rustfs_ci=true ;; crates/omnigraph-cluster/src/store.rs|crates/omnigraph-cluster/src/serve.rs) run_rustfs_ci=true ;; @@ -411,9 +411,29 @@ jobs: set -e echo "$output" [ "$status" -eq 0 ] || exit "$status" - echo "$output" | grep -Eq "test result: ok\. 1 passed" \ + echo "$output" | grep -Fq \ + "SKIP public_physical_ref_token_rejects_s3_same_version_aba" \ + && { echo "::error::S3 physical-ref guard skipped — vacuous pass"; exit 1; } + echo "$output" | grep -Eq "test result: ok\. 1 passed; 0 failed" \ || { echo "::error::S3 physical-ref guard matched no test — vacuous pass"; exit 1; } + - name: Run RustFS MemWAL enrollment gate + if: matrix.shard == 'engine' + run: | + set +e + output=$(cargo test --locked -p omnigraph-engine --test memwal_enrollment_gate \ + s3_memwal_enrollment_gate_positive_and_listing_negatives \ + -- --exact --nocapture 2>&1) + status=$? + set -e + echo "$output" + [ "$status" -eq 0 ] || exit "$status" + echo "$output" | grep -Fq \ + "SKIP s3_memwal_enrollment_gate_positive_and_listing_negatives" \ + && { echo "::error::S3 MemWAL enrollment gate skipped — vacuous pass"; exit 1; } + echo "$output" | grep -Eq "test result: ok\. 1 passed; 0 failed" \ + || { echo "::error::S3 MemWAL enrollment gate matched no test — vacuous pass"; exit 1; } + # NOTE: the RFC-013 step-3a data-table opener COST gate (write_cost_s3) used # to run here. It is a deterministic IO-count gate, not a correctness test — # performance/cost contracts belong in a dedicated perf harness on a stable diff --git a/WAL-thinking.md b/WAL-thinking.md new file mode 100644 index 00000000..edf475e1 --- /dev/null +++ b/WAL-thinking.md @@ -0,0 +1,475 @@ +# WAL Thinking + +Working notes, 2026-07-17. Plain-language grounding for the WAL/streaming +discussion ([RFC-018](docs/rfcs/0018-ingest-wal.md) → +[RFC-026](docs/rfcs/0026-memwal-streaming-ingest.md)). Three parts: the +contract difference between an interactive graph commit and durable stream +admission, the expected performance shape and evidence still required, and the +build inventory. + +--- + +## 1. Interactive graph commit vs WAL admission + later fold + +The deepest way to see it is at the acknowledgement boundary: + +- an **interactive write** acknowledges only after validation, durable recovery + ownership, all affected Lance dataset commits, and one `__manifest` + publication; +- a **stream write** acknowledges earlier, after checks that need no graph read + and after its WAL data is durable; +- state-dependent checks and graph visibility happen later, in a **fold**. + +These steps do not physically happen in one instant. The contract is that an +interactive caller receives success only after all of them complete, while a +stream caller explicitly chooses the earlier durability-only acknowledgement. + +### Interactive write — graph-visible before acknowledgement + +One `mutate` or `load` request may already contain many rows, statements, and +affected tables: + +```text +one interactive request: + + 1. resolve recovery and capture one coherent graph view + 2. validate and stage every affected table + 3. revalidate authority and arm one recovery sidecar + 4. commit one exact Lance transaction per affected dataset + 5. confirm the physical effects + 6. publish every achieved dataset version in one __manifest CAS + 7. finalize required local views and acknowledge the committed success; + remove satisfied recovery residue best-effort + + after acknowledgement: + ✓ physical effects are durable + ✓ subsequent committed reads can select the new graph state + ✓ validation and conflict checks completed + ✓ one graph time-travel point exists +``` + +A query that captured the previous snapshot before publication continues to see +that old snapshot. Atomic publication means new committed reads never see only +some of the affected tables; it does not mutate an already-running query's +snapshot. A sidecar-cleanup failure after publication does not reverse success +or become a user error; a later recovery barrier can prove and finalize it. + +The cost concern is real but currently qualitative: every successful +interactive request pays the recovery, per-dataset commit, and graph +publication protocol. Graph commits also pass through the serial +`graph_head`/`__manifest` authority point. Current throughput and critical-path +round trips must be measured on the current Lance pin before attaching a number +to that floor. + +### Stream admission — durable before graph-visible + +One logical stream row follows a different contract: + +```text +one stream row: + + 1. validate shape, types, required/default fields, enum/range/check rules, + reserved fields, and stream mode + 2. route it to the active owner of the table's MemWAL shard + 3. include it in a submitted Lance WAL batch + 4. wait for the durability result and acknowledge the caller + + after acknowledgement: + ✓ the submitted WAL data is durable and replayable + ✓ checks that require no graph read have completed + ✗ default graph reads do not see it yet + ✗ graph-dependent checks have not completed + ✗ no graph commit/time-travel point exists yet +``` + +The warm, uncontended WAL persistence path is normally one conditional object +store PUT per submitted Lance batch. It is not an unconditional one-call +guarantee: first open, WAL-position discovery, replay, fencing, retries, and +multipart behavior can add operations. + +MemWAL has exactly one active writer epoch per shard. Many producers may share +one routed, warm shard owner, but independent processes do not append +uncontended to the same shard: a new claimant fences its predecessor. Initial +RFC-026 delivery therefore uses one unsharded owner per `(table, main)`; +horizontal writer scaling comes later from deterministic key-based sharding. + +### Fold — deferred graph publication, paid once per batch + +There is no fixed 30-second or 10,000-row contract. A folder may be triggered by +rows, bytes, maximum lag, resource pressure, or an operator request. It consumes +an ordered, durable generation cut: + +```text +a fold: + + capture one exact stream binding and generation cut + → run graph-dependent checks + (referential integrity, cardinality, cross-version uniqueness, + embeddings, and other base-dependent work) + → strict mode: stop and mark the shard blocked on a permanent failure + dead_letter mode: stage a typed reject in _ingest_rejects + → stage accepted rows with Lance merge-insert and merged_generations + → stage reject/audit participants required by the disposition + → arm one RFC-022 recovery sidecar + → commit every affected Lance dataset + → classify every achieved participant and durably record EffectsConfirmed + → publish all achieved versions in one __manifest CAS + → accepted rows and configured reject/audit state become visible together + → remove the satisfied sidecar best-effort +``` + +The fold is a new stream-specific RFC-022 effect adapter on the existing +publication and recovery chassis. It reuses that chassis, but it adds stream +binding and generation authority, fold-time validation, merge progress, +reject/audit atomicity, and later GC eligibility. + +Folded data is not deleted from MemWAL immediately. WAL entries and flushed +generations become eligible for later GC only after the exact fold is +graph-visible, its recovery sidecar is resolved, maintained indexes have caught +up, retained versions no longer need the data, no fresh-read guard pins it, and +writer fencing remains safe. + +### Side by side + +| | Interactive request | Stream row + later fold | +|---|---|---| +| Acknowledgement means | The graph commit is durable and published | The stream row is durably accepted for ordered processing | +| Graph visibility | Before success is returned | After a successful fold; lag depends on triggers, validation, bounds, and fold health | +| Pre-ack checks | Complete request validation | Checks that need no graph/base-table read | +| Deferred outcome | None after success | Strict fold may block; configured dead-letter may record a reject | +| Same-key behavior | Serialize, retry, or fail loudly | Last-write-wins only when every occurrence of a key maps to the same shard | +| Versions | One graph commit per successful request, which may contain many rows | One graph commit per successful fold, plus one Lance version per affected participant | +| Crash story | RFC-022 recovery resolves any table-effect/publication gap | WAL replay preserves durable input; RFC-022 recovery still resolves fold effects/publication | +| Best fit | Edits requiring immediate graph visibility and synchronous rejection | High-rate feeds that accept delayed visibility and deferred state-dependent disposition | + +### The trade-offs we deliberately accept + +1. **Visibility lag.** “Durable” no longer means “present in default graph + reads.” Visibility follows a successful fold and is not promised within an + unmeasured number of seconds. +2. **Deferred graph-dependent disposition.** An acknowledged row may later + block a strict fold or be atomically dead-lettered when that mode is + configured. The acknowledgement cannot be withdrawn. +3. **More lifecycle machinery.** Enrollment, owner routing, backpressure, + drain/seal/resume, replay, fold health, and cleanup become operator-visible + responsibilities. +4. **Write and storage amplification.** Data may exist in a WAL entry, a + flushed Lance generation, the base dataset, and maintained indexes before + obsolete copies can be reclaimed. + +One line for the team: + +> A graph commit means “the graph changed.” A WAL acknowledgement means “the +> system durably accepted this row for ordered processing.” A successful fold +> gives that row its contract-defined graph-visible or reject disposition. + +--- + +## 2. Expected performance shape — and what is not measured yet + +The expected win is **amortization**, not a proven request-count multiplier: + +- the interactive path pays the graph publication protocol per successful + request; +- the streaming acknowledgement path pays WAL durability per submitted Lance + batch; +- a fold pays graph publication once for a generation batch. + +That can materially improve acknowledgement latency and sustained admission +throughput when many logical rows share WAL and fold work. The magnitude is not +yet measured. + +### RC.1 does not provide automatic 100 ms group commit + +Lance RC.1 defaults to durable writes. In durable mode, each `ShardWriter::put` +triggers an immediate WAL flush and waits for it. The default 100 ms interval +is part of the same writer machinery and bounds buffered/non-durable +accumulation; it does not delay a durable put to form a guaranteed group-commit +window. + +Therefore, predictable group commit requires an explicit OmniGraph admission +design that combines multiple logical rows into one submitted Lance `put` (or +another measured coalescing strategy) while preserving: + +- ordered per-row acknowledgements; +- per-actor admission accounting and bounds; +- cancellation and ambiguous-response semantics; +- bounded latency for sparse traffic; +- typed propagation of flush and fencing failures. + +Without that adapter, 100 sequential durable Lance puts normally target 100 +separate WAL entries, not ten shared 100 ms flushes; retries may add PUT +attempts. + +### What request volume actually depends on + +The useful comparison is: + +```text +interactive work = interactive requests × graph-publication work per request + +stream admission = submitted WAL batches × WAL durability work per batch + +fold work = productive folds × work for their rows, bytes, generations, + indexes, validation, and dataset participants +``` + +The number of submitted WAL batches depends on the batching policy, row bytes, +latency bound, shard count, backpressure, retries, and multipart behavior. +Generation flushes and folds also scale with data volume and index work. +MemWAL changes the unit of amortization; it does not make storage operations +independent of write volume. + +### What current evidence says + +The checked-in current-path instruments establish bounded components, not a +complete end-to-end call count: + +- a focused shallow single-insert run on 2026-07-17 observed 5 tracked + data-table reads and 18 tracked `__manifest` reads; +- the schema/recovery adapter reports its text reads, existence checks, + sidecar writes, and deletion separately; +- the bounded data-write test observed one data-table write; +- those counters overlap and do not identify sequential critical-path hops, so + they cannot be added into a truthful “12 calls per write” result. + +See [`write_cost.rs`](crates/omnigraph/tests/write_cost.rs). There is currently +no matched RC.1 RustFS/S3 benchmark of one-row interactive latency, sustained +branch throughput, or the not-yet-built streaming path. RFC-018's historical +single-digit language and upstream prototype results are not measurements of +the current OmniGraph path. + +Consequently, the following are hypotheses rather than results: + +- a specific current calls-per-write number; +- a one-PUT-per-100-ms streaming rate; +- a single-digit current branch-throughput ceiling; +- 8×, 80×, or 500× request-count reductions; +- a fixed two-times byte multiplier; +- any claim that WAL request volume stops scaling with input. + +### Required benchmark before quoting a multiplier + +Run a matched RC.1 RustFS/S3 instrument after the admission adapter exists: + +1. fix schema, row shape, row bytes, table count, and index configuration; +2. compare interactive one-row requests, existing `load` batching, and stream + admission at 1, 100, and 1,000 logical rows/sec, plus a sparse one-row/minute + case; +3. declare batch size and maximum batch delay rather than relying on defaults; +4. record achieved throughput, backpressure, p50/p95/p99 acknowledgement + latency, and fold visibility lag; +5. count GET/HEAD/LIST/PUT/DELETE and multipart operations separately for the + acknowledgement and background paths; +6. record serial critical-path hops, uploaded bytes, retained bytes before and + after GC, fold rows/bytes/generations, and every affected dataset; +7. run multiple fresh trials and preserve the raw records. + +The comparison must state the semantic difference: interactive success is +graph-visible, whereas stream success is durability-only. + +The honest summary today is: + +> MemWAL can move durable acknowledgement off the graph-publication path and +> amortize graph-visible publication over folds. RC.1 flushes durable data per +> submitted put, so predictable group commit is an OmniGraph integration +> responsibility. The improvement factor remains to be measured. + +--- + +## 3. Inventory: Lance substrate vs OmniGraph integration + +### Lance primitives we plan to consume + +These are real substrate capabilities, not a turnkey graph-streaming product: + +| Lance primitive | What it gives us — and the boundary | +|---|---| +| **WAL entries on object storage** | Sequenced Arrow IPC entries with bit-reversed names. A warm uncontended append is normally one conditional PUT; open, recovery, retries, and fencing add work. | +| **Durability results/watchers** | The acknowledgement primitive. Predictable multi-request group commit still requires an OmniGraph batching policy. | +| **Epoch-fenced shard writers and fence sentinels** | One active owner per shard and stale-writer detection. This is per-shard fencing, not OmniGraph's graph-wide distributed recovery fence. | +| **MemTables and flushed generations** | Recent rows live in WAL-backed memory and later in small Lance datasets. Maintained FTS/vector/scalar indexes exist only when explicitly configured and supported. | +| **`merged_generations`** | Merge progress updated atomically with a base-table merge. It supplies the marker needed for idempotent per-table folding; RFC-022 still owns graph publication and partial-effect recovery. | +| **WAL replay** | Durable entries are replayed to reconstruct the MemTable under the next valid claimant. The guarantee depends on respecting fencing, lifecycle, compatibility, and safe GC. | +| **LSM merging reads** | A scanner over the base table, selected flushed generations not yet safely replaceable by the base read plan, and optionally same-process active/frozen MemTables. OmniGraph must build and retain a coherent graph-level fresh-read cut. It does not query raw WAL files as a normal read source. | +| **GC eligibility rules** | Upstream specifies when generations may be obsolete and warns that deleting WAL files can weaken fencing. RC.1 does not provide OmniGraph's graph-aware MemWAL garbage collector. | +| **Staged merge-insert** | `execute_uncommitted` plus atomic `merged_generations` fits the RFC-022 staged shape. There is no one-call MemWAL fold API. | +| **Shared sessions/store parameters in RC.1** | Writer-created generations and base-backed scanner paths reuse the base dataset's access context; fresh-only construction still receives that context from its caller. OmniGraph owns long-lived writer/session lifecycle and scanner integration. | +| **Key-based sharding** *(later)* | Horizontal scale when every occurrence of one key deterministically maps to one shard. | + +OmniGraph already has reusable chassis: the manifest publisher, generic +recovery-sidecar framework, graph lineage, stable table identity, keyed write +adapter, and validation components. “Reusable” does not mean “unchanged”: +RFC-026 still needs stream lifecycle authority, fold read sets, reject/audit +participants, actor-range provenance, fresh cuts, and cleanup coordination. + +### What OmniGraph must build + +| Responsibility | Required contract | +|---|---| +| **Format capability and refusal** | Streaming-capable graph stamp, strict old/new binary refusal, and export/init/load rebuild behavior. | +| **`@stream` intent and enrollment** | Bind stable table/incarnation identity, location/main ref, never-reused enrollment ID, shard namespace, and configuration under recovery; separately persist the mutable current-HEAD witness and advance it with every allowed base commit. | +| **Public surface** | `POST /graphs/{graph_id}/streams/{type_name}/ingest`, status/fold/quiesce/resume endpoints, `omnigraph stream …` commands, and OpenAPI parity. Existing `/ingest` remains the deprecated load alias. | +| **Writer registry and routing** | Warm writers keyed by exact table identity, enrollment, and shard; one routed owner per initial `(table, main)`; bounded memory, inflight bytes, idle eviction, and backpressure. | +| **Durability batching** | Explicitly combine logical rows into submitted Lance puts while bounding latency and preserving ordered acknowledgements, actor accounting, cancellation, and typed failures. | +| **Pre-ack validation** | Run every rule that needs no graph/base-table read before WAL persistence. | +| **Fold adapter** | Capture stream binding and generation authority, run deferred checks, stage merge-insert with `merged_generations`, and publish through RFC-022. | +| **Strict and dead-letter disposition** | Strict blocking by default; configured rejects, accepted rows, audit, and merge progress committed atomically. | +| **Policy, lineage, and audit** | Dedicated Cedar actions plus fold actor and contributor-range provenance. | +| **Quiescence and rebind** | Durable `OPEN → DRAINING → SEALED`, stop admission, fence/drain owners, flush/fold, verify empty progress, resume or bind a fresh physical enrollment. | +| **Fresh reads** *(later)* | Explicit committed/fresh IR mode, exact base-plus-MemWAL cut, merged-generation exclusion, retention guards, and documented lack of cross-table atomicity. | +| **Cleanup** | Graph-aware generation/WAL GC integrated with visibility, recovery, time travel, indexes, fresh-read guards, and fencing safety. | +| **Evidence and operations** | Surface guards, enrollment/fold failpoints, race/crash matrix, cost budgets, metrics, status, API tests, and CLI parity. | + +The table intentionally omits “small/medium” estimates. Atomic rejection, +quiescence, fresh-read retention, and GC are correctness protocols; size them +only after their implementation spikes expose the real work. + +### The RC.1 enrollment gap, stated precisely + +What RC.1 exposes: + +```text +initialize MemWAL index + → commits CreateIndex internally + → mutates the Dataset handle + → returns Result<()> + +open the selected ShardWriter + → separately creates/updates the shard manifest + → claims an epoch with atomic, epoch-fenced writes + → returns a writer that exposes the claimed epoch +``` + +Shard claiming is protected; it is not an “unprotected step that returns +nothing.” The missing upstream shape is one externally recoverable contract +covering both: + +- ownership/classification of the MemWAL index-enrollment effect; and +- provisioning, admission sealing, reopening, and reclaiming the intended shard + objects. + +An upstream exact receipt and public admission lifecycle would simplify this +substantially. We should propose that change, but its review and release timing +is not a prerequisite for the bounded-profile decision. Gate E0 passed; the +upstream shape still gates overlapping-process topology, while bounded +activation now depends on OmniGraph's Phase A and later production proofs. + +### Gate E0: a no-wait decision, not activation + +RFC-026 now has a production-neutral Gate E0. The question is narrower than +“can we ship streaming on RC.1?” It asks: “can public RC.1 state distinguish the +only effects our first enrollment is allowed to create after success, failure, +or a lost result without a history scan or ambiguous listing?” The answer is +yes for the bounded profile below. + +The terminology matters. One value cannot serve two incompatible jobs: + +- the **stable enrollment binding** is the logical table/incarnation, + location/main ref, never-reused enrollment ID, pre-minted shard ID, and + configuration hash. The enrollment ID/config version is also embedded in + Lance's persisted namespaced writer defaults so lost-result classification + does not depend on the replaceable MemWAL index UUID; +- the **current-HEAD witness** is the current `BranchIdentifier`, Lance version, + transaction UUID, and manifest e_tag. + +The witness changes on every ordinary Lance commit. It is not a stable +“physical-ref incarnation.” The bounded design makes that manageable by giving +an `OPEN` stream exclusive authority to advance its base-table HEAD. Only fold +or recovery may move it; Mutation/Load, BranchMerge, SchemaApply, Optimize, +EnsureIndices, repair, cleanup, and branch operations touching that table must +refuse before effect or drain first. Every allowed commit publishes the next +witness atomically with the table pointer and stream lifecycle row. + +We do not use a long-lived Lance tag as the default stable anchor. It would pin +the enrollment-time table snapshot—potentially an old full-table file set after +rewrite/compaction—and add another auxiliary effect to the enrollment crash +matrix. It remains a measured fallback only if a future profile truly needs +interactive base writers while a stream stays open. + +Gate E0's evidence-backed support boundary is **main-only, one unsharded keyed +shard, and one live OmniGraph writer process for the graph**. A crash successor +is allowed only after external exclusivity. An overlapping second process and +raw Lance writers are unsupported, not silently “best effort.” + +The E0 evidence suite proved: + +1. exact baseline HEAD `N` with no MemWAL index; +2. initializer success yields exactly `N + 1`, whose transaction reads `N` and + contains only the intended singleton MemWAL `CreateIndex`, exact unsharded + configuration, and namespaced enrollment/config-version marker; +3. discarding the initializer result and reopening yields the same exact + allowed-successor classification; +4. the public writer path creates/claims only the pre-minted shard, with the + expected unsharded spec, observable epoch, no flushed generation, and no + data-bearing WAL; any deterministic data-less fence artifact is enumerated; +5. the truth table accepts only no effect, the exact index-only successor, and + that successor plus the exact empty shard. Wrong configuration, intervening + HEAD, foreign shard, unexpected data, or ambiguity is `RecoveryRequired`; +6. once an allowed effect exists, the candidate is roll-forward-only. It never + deletes, reclaims, or restores MemWAL state from inferred ownership; and +7. the current-HEAD witness is stable across unchanged reopen, changes after an + ordinary commit, catches same-path/same-version recreation locally and on + S3/RustFS, and can be classified with measured history-bounded work. + +The first cost attempt was rejected: local `checkout_latest` may use filesystem +`read_dir` outside `IOTracker`. The accepted classifier instead opens exact `N` +through the manifest-pinned physical URI, probes only `N + 1` with the public +but guide-hidden `Dataset::has_successor_version`, then asks exact `N + 1` +whether a buried `N + 2` exists. `AttemptTracker` records attempts before +forwarding, including `NotFound` and errors. At baseline versions 8 and 80 it +records the same six-attempt shape—four successful manifest HEADs, one +`NotFound` manifest HEAD, one successful manifest GET—and zero list calls. A +Unix execute-only `_versions` tripwire proves the exact path works when latest +enumeration fails; an unreadable exact HEAD errors instead of becoming absence. + +The local run has 14 substantive cells plus one explicit unconfigured-S3 skip. +The configured RustFS cell passes non-vacuously with the same six-attempt, +zero-list shape and covers the positive no-effect → lost-result index → +pre-minted empty-shard → unchanged-reopen sequence plus foreign shard, +malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest +refusals. Separate surface guards own object-store ABA and pin the doc-hidden +successor, flush/drain, and merged-generation surfaces; CI rejects skipped E0 +and ABA cells. + +E0 added no `@stream`, lifecycle rows, a sidecar schema, public APIs, WAL +acknowledgements, or a format stamp. Internal schema v6 remains production +truth. Its green result authorizes only Phase A's enrollment recovery, central +writer exclusion, lifecycle/current-witness state, and admission-lease races— +not stream release. The exact upstream receipt/seal remains the preferred +simplification and the broader-topology gate. + +That later activation work still is not the Optimize proof copied to a new +writer kind. Optimize is content-preserving maintenance and grants no future +durability authority. Enrollment authorizes acknowledgements, so its production +slice must cover initialization, shard claim, fixed binding publication, +admission lease, first put, lost acknowledgement, drain, restart, and attempted +foreign ownership before a single row can be acknowledged. + +### Bottom line + +Lance supplies the WAL/LSM substrate: durable log entries, shard fencing, +replay, generations, LSM reads, staged merge primitives, merge progress, and +sharding. OmniGraph still owns the graph-hard parts: enrollment identity, +publication authority, graph-atomic folding and rejection, recovery, +lifecycle/quiescence, fresh snapshot cuts, format gating, policy/audit, and +cleanup integration. + +The plan is: + +1. retain Gate E0's green exact-version classifier as the RC.1 substrate gate; + schema v6 remains unchanged; +2. implement exact enrollment recovery, central + `OPEN`-table writer exclusion, lifecycle state, and the admission-lease crash + matrix while keeping public streaming inactive; +3. close format/refusal/rebuild and fold/ack durability gates before activating + durable admission; +4. design and measure explicit durability batching before claiming group + commit or a performance multiplier; +5. pursue the exact upstream receipt/seal API in parallel, then use it to + simplify the adapter and broaden topology when it lands. + +We tested the narrower support contract instead of waiting on the upstream +calendar, and Gate E0 supports it. Until every later production gate passes, +RFC-026 remains draft and no stream format, lifecycle, or acknowledgement path +is active. diff --git a/crates/omnigraph/tests/checkpoint_retention_cost.rs b/crates/omnigraph/tests/checkpoint_retention_cost.rs new file mode 100644 index 00000000..438da5f8 --- /dev/null +++ b/crates/omnigraph/tests/checkpoint_retention_cost.rs @@ -0,0 +1,1404 @@ +//! RFC-025 pre-activation decision instrument for checkpoint-registry access. +//! +//! This is deliberately a production-neutral Lance fixture. It models the +//! proposed checkpoint name, header, table, and GC-boundary rows in an isolated +//! dataset without changing OmniGraph's accepted manifest schema or format +//! stamp. Fixed live-checkpoint count and catalog width are held constant while +//! unrelated manifest journal history grows. +//! +//! The three measured operations are the complete authority reads required by +//! the proposed API: checkpoint list, exact name lookup (name reservation plus +//! header/table projection), and cleanup-root planning. Every operation is +//! measured from a tracked cold open and then repeated on the same Dataset and +//! Session. Object-store reads/bytes include the cold open; Lance execution +//! summaries separately report scan I/O and the pinned RC.1 debug counters. +//! +//! The fixture is a decision gate, not evidence that the format has shipped. +//! Its assertions preserve observed blockers rather than treating flat +//! `rows_scanned` as proof of flat physical I/O. + +mod helpers; + +use std::collections::{BTreeSet, HashMap}; +use std::future::Future; +use std::panic::{AssertUnwindSafe, resume_unwind}; +use std::sync::{Arc, Mutex}; + +use arrow_array::{Array, RecordBatch, RecordBatchIterator, StringArray, UInt64Array}; +use arrow_schema::{DataType, Field, Schema, SchemaRef}; +use datafusion::logical_expr::Expr; +use datafusion::prelude::{col, lit}; +use futures::{FutureExt, TryStreamExt}; +use lance::Dataset; +use lance::dataset::optimize::{CompactionOptions, compact_files}; +use lance::dataset::scanner::ExecutionSummaryCounts; +use lance::dataset::{MergeInsertBuilder, WhenMatched, WhenNotMatched, WriteMode, WriteParams}; +use lance::datatypes::LANCE_UNENFORCED_PRIMARY_KEY; +use lance::index::DatasetIndexExt; +use lance::session::Session; +use lance_file::version::LanceFileVersion; +use lance_index::IndexType; +use lance_index::scalar::ScalarIndexParams; +use lance_io::utils::tracking_store::IOTracker; + +use helpers::cost::open_tracked_lance_dataset; + +const CATALOG_WIDTH: usize = 10; +const LIVE_CHECKPOINTS: usize = 3; +const DEFAULT_DEPTHS: [u64; 2] = [20, 80]; +const UNCOVERED_TAIL: u64 = 8; +const ROWS_PER_HISTORY_PUBLISH: u64 = 3; + +const OBJECT_ID_INDEX: &str = "rfc025_object_id_btree"; +const OBJECT_TYPE_INDEX: &str = "rfc025_object_type_btree"; +const CHECKPOINT_ID_INDEX: &str = "rfc025_checkpoint_id_btree"; + +const PROJECTION: [&str; 13] = [ + "object_id", + "object_type", + "checkpoint_id", + "checkpoint_name", + "state", + "generation", + "stable_table_id", + "table_incarnation_id", + "table_path", + "table_branch", + "table_version", + "physical_ref_incarnation", + "gc_cutoff", +]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +enum HistoryLayout { + Compacted, + Uncompacted, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +enum IndexState { + Absent, + Reconciled, + BoundedTail, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +enum AccessMode { + ColdOpen, + WarmRepeat, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +enum AuthorityOperation { + List, + Show, + CleanupPlan, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum StorageBackend { + Local, + S3, +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +struct AuthorityCost { + object_store_reads: u64, + object_store_read_bytes: u64, + scan_iops: u64, + scan_requests: u64, + scan_read_bytes: u64, + indices_loaded: u64, + index_pages_loaded: u64, + fragments_scanned: u64, + ranges_scanned: u64, + rows_scanned: u64, + output_rows: u64, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct CurvePoint { + base_depth: u64, + journal_depth: u64, + layout: HistoryLayout, + index_state: IndexState, + operation: AuthorityOperation, + access: AccessMode, + total_fragments: u64, + uncovered_fragments: u64, + cost: AuthorityCost, +} + +type CostCurve = Vec; + +#[derive(Debug, Clone)] +struct AuthorityRow { + object_id: String, + object_type: String, + checkpoint_id: Option, + checkpoint_name: Option, + state: Option, + generation: Option, + stable_table_id: Option, + table_incarnation_id: Option, + table_path: Option, + table_branch: Option, + table_version: Option, + physical_ref_incarnation: Option, + gc_cutoff: Option, +} + +#[derive(Debug, Clone, Copy)] +struct Coverage { + total_fragments: u64, + uncovered_fragments: u64, +} + +struct RetentionFixture { + uri: String, + dataset: Dataset, + journal_depth: u64, +} + +impl RetentionFixture { + async fn create(uri: String) -> Self { + let mut rows = checkpoint_authority_rows(); + rows.push(graph_head_row(0)); + + let schema = authority_schema(); + let batch = rows_to_batch(&rows, &schema); + let reader = RecordBatchIterator::new([Ok(batch)], schema); + let params = WriteParams { + mode: WriteMode::Create, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + auto_cleanup: None, + skip_auto_cleanup: true, + ..Default::default() + }; + let dataset = Dataset::write(reader, &uri, Some(params)) + .await + .unwrap_or_else(|error| panic!("failed to create RFC-025 fixture at {uri}: {error}")); + + Self { + uri, + dataset, + journal_depth: 0, + } + } + + async fn publish_until(&mut self, target_depth: u64) { + assert!(target_depth >= self.journal_depth); + while self.journal_depth < target_depth { + self.publish_one().await; + } + } + + async fn publish_one(&mut self) { + let next_depth = self.journal_depth + 1; + let stable_id = (self.journal_depth as usize % CATALOG_WIDTH) as u64 + 1; + let rows = vec![ + table_version_history_row(stable_id, next_depth), + graph_commit_row(next_depth), + graph_head_row(next_depth), + ]; + let schema = authority_schema(); + let batch = rows_to_batch(&rows, &schema); + let reader = RecordBatchIterator::new([Ok(batch)], schema); + + let mut builder = MergeInsertBuilder::try_new( + Arc::new(self.dataset.clone()), + vec!["object_id".to_string()], + ) + .unwrap(); + builder + .when_matched(WhenMatched::UpdateAll) + .when_not_matched(WhenNotMatched::InsertAll) + .conflict_retries(0) + .use_index(false) + .skip_auto_cleanup(true); + let (dataset, stats) = builder + .try_build() + .unwrap() + .execute_reader(Box::new(reader)) + .await + .unwrap_or_else(|error| { + panic!("RFC-025 fixture publish at depth {next_depth} failed: {error}") + }); + assert_eq!(stats.num_inserted_rows, 2); + assert_eq!(stats.num_updated_rows, 1); + self.dataset = Arc::try_unwrap(dataset).unwrap_or_else(|arc| (*arc).clone()); + self.journal_depth = next_depth; + } + + async fn compact(&mut self) { + compact_files(&mut self.dataset, CompactionOptions::default(), None) + .await + .unwrap_or_else(|error| panic!("RFC-025 fixture compaction failed: {error}")); + } + + async fn create_authority_indices(&mut self) { + for (column, name) in [ + ("object_id", OBJECT_ID_INDEX), + ("object_type", OBJECT_TYPE_INDEX), + ("checkpoint_id", CHECKPOINT_ID_INDEX), + ] { + self.dataset + .create_index_builder(&[column], IndexType::BTree, &ScalarIndexParams::default()) + .name(name.to_string()) + .replace(true) + .await + .unwrap_or_else(|error| panic!("RFC-025 {column} BTREE creation failed: {error}")); + } + } + + async fn coverage(&self) -> Option { + let indices = self.dataset.load_indices().await.unwrap(); + let total_fragments = self.dataset.fragments().len() as u64; + let mut maximum_uncovered = 0; + + for (column, name) in [ + ("object_id", OBJECT_ID_INDEX), + ("object_type", OBJECT_TYPE_INDEX), + ("checkpoint_id", CHECKPOINT_ID_INDEX), + ] { + let field_id = self.dataset.schema().field(column).unwrap().id; + let segments: Vec<_> = indices + .iter() + .filter(|index| index.name == name && index.fields == vec![field_id]) + .collect(); + if segments.is_empty() { + return None; + } + let uncovered = self + .dataset + .fragments() + .iter() + .filter(|fragment| { + let id = u32::try_from(fragment.id).expect("test fragment id fits u32"); + !segments.iter().any(|segment| { + segment + .fragment_bitmap + .as_ref() + .is_some_and(|bitmap| bitmap.contains(id)) + }) + }) + .count() as u64; + maximum_uncovered = maximum_uncovered.max(uncovered); + } + + Some(Coverage { + total_fragments, + uncovered_fragments: maximum_uncovered, + }) + } + + async fn operation_pair( + &self, + operation: AuthorityOperation, + ) -> [(AccessMode, AuthorityCost); 2] { + let tracker = IOTracker::default(); + let session = Arc::new(Session::default()); + let dataset = open_tracked_lance_dataset(&self.uri, session, &tracker) + .await + .unwrap_or_else(|error| panic!("tracked RFC-025 open failed: {error}")); + + let cold = execute_operation(&dataset, operation).await; + let cold_io = tracker.incremental_stats(); + let cold_cost = authority_cost(&cold.summaries, &cold_io, cold.output_rows); + + let warm = execute_operation(&dataset, operation).await; + let warm_io = tracker.incremental_stats(); + let warm_cost = authority_cost(&warm.summaries, &warm_io, warm.output_rows); + + [ + (AccessMode::ColdOpen, cold_cost), + (AccessMode::WarmRepeat, warm_cost), + ] + } +} + +fn authority_schema() -> SchemaRef { + let object_id_metadata: HashMap = + [(LANCE_UNENFORCED_PRIMARY_KEY.to_string(), "true".to_string())] + .into_iter() + .collect(); + Arc::new(Schema::new(vec![ + Field::new("object_id", DataType::Utf8, false).with_metadata(object_id_metadata), + Field::new("object_type", DataType::Utf8, false), + Field::new("checkpoint_id", DataType::Utf8, true), + Field::new("checkpoint_name", DataType::Utf8, true), + Field::new("state", DataType::Utf8, true), + Field::new("generation", DataType::UInt64, true), + Field::new("stable_table_id", DataType::UInt64, true), + Field::new("table_incarnation_id", DataType::UInt64, true), + Field::new("table_path", DataType::Utf8, true), + Field::new("table_branch", DataType::Utf8, true), + Field::new("table_version", DataType::UInt64, true), + Field::new("physical_ref_incarnation", DataType::Utf8, true), + Field::new("gc_cutoff", DataType::UInt64, true), + ])) +} + +fn checkpoint_id(index: usize) -> String { + format!("fixture-checkpoint-{index:04}") +} + +fn checkpoint_name(index: usize) -> String { + format!("checkpoint-{index:02}") +} + +/// Test-only reference contract for RFC-025's eventual checkpoint-name-v1 +/// normalizer. Identity is deliberately ASCII and case-sensitive: successful +/// normalization returns the input unchanged, so no case folding or Unicode +/// equivalence can collapse two authority keys. +fn normalize_checkpoint_name_v1(name: &str) -> Result<&str, &'static str> { + if name.is_empty() || name.len() > 128 { + return Err("checkpoint name must be 1..=128 bytes"); + } + if !name.is_ascii() { + return Err("checkpoint name must be ASCII"); + } + if name.starts_with("__") || name.starts_with("ogcp_") { + return Err("checkpoint name uses a reserved prefix"); + } + if name.starts_with('/') || name.ends_with('/') { + return Err("checkpoint name cannot start or end with '/'"); + } + if name.contains("//") { + return Err("checkpoint name cannot contain an empty segment"); + } + if name.contains("..") { + return Err("checkpoint name cannot contain '..'"); + } + if name.contains('\\') { + return Err("checkpoint name cannot contain a backslash"); + } + if name.split('/').any(|segment| segment.ends_with(".lock")) { + return Err("checkpoint name segments cannot end with '.lock'"); + } + if !name.split('/').all(|segment| { + segment + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"._-".contains(&byte)) + }) { + return Err("checkpoint segments allow only [A-Za-z0-9._-]"); + } + Ok(name) +} + +fn table_incarnation(stable_id: u64) -> u64 { + 10_000 + stable_id +} + +fn table_path(stable_id: u64) -> String { + format!( + "nodes/{stable_id:016x}-{:016x}", + table_incarnation(stable_id) + ) +} + +fn checkpoint_authority_rows() -> Vec { + let mut rows = Vec::new(); + for checkpoint_index in 1..=LIVE_CHECKPOINTS { + let id = checkpoint_id(checkpoint_index); + let generated_name = checkpoint_name(checkpoint_index); + let name = normalize_checkpoint_name_v1(&generated_name) + .expect("fixture checkpoint name must satisfy V1") + .to_string(); + rows.push(AuthorityRow { + object_id: format!("checkpoint_name:v1:{name}"), + object_type: "checkpoint_name".to_string(), + checkpoint_id: Some(id.clone()), + checkpoint_name: Some(name.clone()), + state: Some("live".to_string()), + generation: Some(1), + stable_table_id: None, + table_incarnation_id: None, + table_path: None, + table_branch: None, + table_version: None, + physical_ref_incarnation: None, + gc_cutoff: None, + }); + rows.push(AuthorityRow { + object_id: format!("checkpoint:{id}"), + object_type: "checkpoint".to_string(), + checkpoint_id: Some(id.clone()), + checkpoint_name: Some(name), + state: Some("live".to_string()), + generation: Some(1), + stable_table_id: None, + table_incarnation_id: None, + table_path: Some("__manifest".to_string()), + table_branch: Some("main".to_string()), + table_version: Some(checkpoint_index as u64 + 1), + physical_ref_incarnation: Some(format!("manifest-ref-{checkpoint_index}")), + gc_cutoff: None, + }); + for offset in 0..CATALOG_WIDTH { + let stable_id = offset as u64 + 1; + rows.push(AuthorityRow { + object_id: format!( + "checkpoint_table:{id}:{stable_id:016x}:{:016x}", + table_incarnation(stable_id) + ), + object_type: "checkpoint_table".to_string(), + checkpoint_id: Some(id.clone()), + checkpoint_name: None, + state: Some("live".to_string()), + generation: Some(1), + stable_table_id: Some(stable_id), + table_incarnation_id: Some(table_incarnation(stable_id)), + table_path: Some(table_path(stable_id)), + table_branch: Some("main".to_string()), + table_version: Some(checkpoint_index as u64 + stable_id), + physical_ref_incarnation: Some(format!("table-ref-{checkpoint_index}-{stable_id}")), + gc_cutoff: None, + }); + } + } + + for lineage in 0..=CATALOG_WIDTH { + rows.push(AuthorityRow { + object_id: format!("gc_boundary:fixture-lineage-{lineage:04}"), + object_type: "gc_boundary".to_string(), + checkpoint_id: None, + checkpoint_name: None, + state: Some("live".to_string()), + generation: None, + stable_table_id: (lineage > 0).then_some(lineage as u64), + table_incarnation_id: (lineage > 0).then_some(table_incarnation(lineage as u64)), + table_path: Some(if lineage == 0 { + "__manifest".to_string() + } else { + table_path(lineage as u64) + }), + table_branch: Some("main".to_string()), + table_version: None, + physical_ref_incarnation: Some(format!("lineage-ref-{lineage}")), + gc_cutoff: Some(0), + }); + } + rows +} + +fn empty_history_row(object_id: String, object_type: &str) -> AuthorityRow { + AuthorityRow { + object_id, + object_type: object_type.to_string(), + checkpoint_id: None, + checkpoint_name: None, + state: None, + generation: None, + stable_table_id: None, + table_incarnation_id: None, + table_path: None, + table_branch: None, + table_version: None, + physical_ref_incarnation: None, + gc_cutoff: None, + } +} + +fn table_version_history_row(stable_id: u64, depth: u64) -> AuthorityRow { + let mut row = empty_history_row( + format!( + "table_version:{stable_id:016x}:{:016x}:{depth:020}", + table_incarnation(stable_id) + ), + "table_version", + ); + row.stable_table_id = Some(stable_id); + row.table_incarnation_id = Some(table_incarnation(stable_id)); + row.table_path = Some(table_path(stable_id)); + row.table_branch = Some("main".to_string()); + row.table_version = Some(depth + 1); + row +} + +fn graph_commit_row(depth: u64) -> AuthorityRow { + let mut row = empty_history_row(format!("fixture-graph-commit-{depth:020}"), "graph_commit"); + row.generation = Some(depth); + row +} + +fn graph_head_row(depth: u64) -> AuthorityRow { + let mut row = empty_history_row("graph_head:main".to_string(), "graph_head"); + row.generation = Some(depth); + row +} + +fn rows_to_batch(rows: &[AuthorityRow], schema: &SchemaRef) -> RecordBatch { + RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.object_id.as_str()) + .collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.object_type.as_str()) + .collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.checkpoint_id.as_deref()) + .collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.checkpoint_name.as_deref()) + .collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.state.as_deref()) + .collect::>(), + )), + Arc::new(UInt64Array::from( + rows.iter().map(|row| row.generation).collect::>(), + )), + Arc::new(UInt64Array::from( + rows.iter() + .map(|row| row.stable_table_id) + .collect::>(), + )), + Arc::new(UInt64Array::from( + rows.iter() + .map(|row| row.table_incarnation_id) + .collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.table_path.as_deref()) + .collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.table_branch.as_deref()) + .collect::>(), + )), + Arc::new(UInt64Array::from( + rows.iter().map(|row| row.table_version).collect::>(), + )), + Arc::new(StringArray::from( + rows.iter() + .map(|row| row.physical_ref_incarnation.as_deref()) + .collect::>(), + )), + Arc::new(UInt64Array::from( + rows.iter().map(|row| row.gc_cutoff).collect::>(), + )), + ], + ) + .unwrap() +} + +#[derive(Debug)] +struct OperationResult { + summaries: Vec, + output_rows: u64, +} + +async fn execute_scan( + dataset: &Dataset, + filter: Expr, +) -> (ExecutionSummaryCounts, Vec) { + let summaries = Arc::new(Mutex::new(Vec::::new())); + let callback_summaries = Arc::clone(&summaries); + let mut scanner = dataset.scan(); + scanner + .project(&PROJECTION) + .unwrap() + .filter_expr(filter) + .use_scalar_index(true) + .scan_stats_callback(Arc::new(move |summary| { + callback_summaries.lock().unwrap().push(summary.clone()); + })); + let batches: Vec = scanner + .try_into_stream() + .await + .unwrap() + .try_collect() + .await + .unwrap(); + let summaries = summaries.lock().unwrap(); + assert_eq!(summaries.len(), 1); + (summaries[0].clone(), batches) +} + +async fn execute_operation(dataset: &Dataset, operation: AuthorityOperation) -> OperationResult { + match operation { + AuthorityOperation::List => { + let (summary, batches) = + execute_scan(dataset, col("object_type").eq(lit("checkpoint"))).await; + assert_eq!(object_ids(&batches), expected_checkpoint_headers()); + OperationResult { + summaries: vec![summary], + output_rows: row_count(&batches), + } + } + AuthorityOperation::Show => { + let expected_name = checkpoint_name(1); + let (name_summary, name_batches) = execute_scan( + dataset, + col("object_id").eq(lit(format!("checkpoint_name:v1:{expected_name}"))), + ) + .await; + assert_eq!( + object_ids(&name_batches), + BTreeSet::from([format!("checkpoint_name:v1:{expected_name}")]) + ); + let resolved_id = only_checkpoint_id(&name_batches); + assert_eq!(resolved_id, checkpoint_id(1)); + + let authority_types = + col("object_type").in_list(vec![lit("checkpoint"), lit("checkpoint_table")], false); + let (projection_summary, projection_batches) = execute_scan( + dataset, + col("checkpoint_id") + .eq(lit(resolved_id)) + .and(authority_types), + ) + .await; + assert_eq!( + object_ids(&projection_batches), + expected_checkpoint_projection(1) + ); + OperationResult { + summaries: vec![name_summary, projection_summary], + output_rows: row_count(&name_batches) + row_count(&projection_batches), + } + } + AuthorityOperation::CleanupPlan => { + let (summary, batches) = execute_scan( + dataset, + col("object_type").in_list( + vec![ + lit("checkpoint"), + lit("checkpoint_table"), + lit("gc_boundary"), + ], + false, + ), + ) + .await; + assert_eq!(object_ids(&batches), expected_cleanup_roots()); + OperationResult { + summaries: vec![summary], + output_rows: row_count(&batches), + } + } + } +} + +fn object_ids(batches: &[RecordBatch]) -> BTreeSet { + let mut ids = BTreeSet::new(); + for batch in batches { + let values = batch + .column_by_name("object_id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + for row in 0..batch.num_rows() { + assert!( + ids.insert(values.value(row).to_string()), + "duplicate authority row" + ); + } + } + ids +} + +fn only_checkpoint_id(batches: &[RecordBatch]) -> String { + assert_eq!(row_count(batches), 1); + let values = batches[0] + .column_by_name("checkpoint_id") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap(); + assert!(!values.is_null(0)); + values.value(0).to_string() +} + +fn row_count(batches: &[RecordBatch]) -> u64 { + batches.iter().map(|batch| batch.num_rows() as u64).sum() +} + +fn expected_checkpoint_headers() -> BTreeSet { + (1..=LIVE_CHECKPOINTS) + .map(|index| format!("checkpoint:{}", checkpoint_id(index))) + .collect() +} + +fn expected_checkpoint_projection(index: usize) -> BTreeSet { + let id = checkpoint_id(index); + std::iter::once(format!("checkpoint:{id}")) + .chain((1..=CATALOG_WIDTH).map(|stable_id| { + format!( + "checkpoint_table:{id}:{stable_id:016x}:{:016x}", + table_incarnation(stable_id as u64) + ) + })) + .collect() +} + +fn expected_cleanup_roots() -> BTreeSet { + (1..=LIVE_CHECKPOINTS) + .flat_map(|index| expected_checkpoint_projection(index).into_iter()) + .chain( + (0..=CATALOG_WIDTH).map(|lineage| format!("gc_boundary:fixture-lineage-{lineage:04}")), + ) + .collect() +} + +fn required_debug_metric(summary: &ExecutionSummaryCounts, name: &str) -> u64 { + u64::try_from( + *summary + .all_counts + .get(name) + .unwrap_or_else(|| panic!("pinned RC.1 scan summary omitted `{name}`: {summary:?}")), + ) + .unwrap() +} + +fn authority_cost( + summaries: &[ExecutionSummaryCounts], + object_io: &lance_io::utils::tracking_store::IoStats, + output_rows: u64, +) -> AuthorityCost { + AuthorityCost { + object_store_reads: object_io.read_iops, + object_store_read_bytes: object_io.read_bytes, + scan_iops: summaries.iter().map(|summary| summary.iops as u64).sum(), + scan_requests: summaries + .iter() + .map(|summary| summary.requests as u64) + .sum(), + scan_read_bytes: summaries + .iter() + .map(|summary| summary.bytes_read as u64) + .sum(), + indices_loaded: summaries + .iter() + .map(|summary| summary.indices_loaded as u64) + .sum(), + index_pages_loaded: summaries + .iter() + .map(|summary| summary.parts_loaded as u64) + .sum(), + fragments_scanned: summaries + .iter() + .map(|summary| required_debug_metric(summary, "fragments_scanned")) + .sum(), + ranges_scanned: summaries + .iter() + .map(|summary| required_debug_metric(summary, "ranges_scanned")) + .sum(), + rows_scanned: summaries + .iter() + .map(|summary| required_debug_metric(summary, "rows_scanned")) + .sum(), + output_rows, + } +} + +async fn record_state( + curve: &mut CostCurve, + fixture: &RetentionFixture, + base_depth: u64, + layout: HistoryLayout, + index_state: IndexState, + backend: StorageBackend, +) { + let coverage = fixture.coverage().await.unwrap_or(Coverage { + total_fragments: fixture.dataset.fragments().len() as u64, + uncovered_fragments: fixture.dataset.fragments().len() as u64, + }); + for operation in [ + AuthorityOperation::List, + AuthorityOperation::Show, + AuthorityOperation::CleanupPlan, + ] { + for (access, cost) in fixture.operation_pair(operation).await { + let point = CurvePoint { + base_depth, + journal_depth: fixture.journal_depth, + layout, + index_state, + operation, + access, + total_fragments: coverage.total_fragments, + uncovered_fragments: coverage.uncovered_fragments, + cost, + }; + eprintln!( + "RFC025 backend={:?} depth={} actual={} layout={:?} index={:?} op={:?} access={:?} fragments={}/{} object_io={}/{} scan_io={}/{}/{} pages={} scanned=f{}/r{}/rows{} output={}", + backend, + point.base_depth, + point.journal_depth, + point.layout, + point.index_state, + point.operation, + point.access, + point.total_fragments - point.uncovered_fragments, + point.total_fragments, + point.cost.object_store_reads, + point.cost.object_store_read_bytes, + point.cost.scan_iops, + point.cost.scan_requests, + point.cost.scan_read_bytes, + point.cost.index_pages_loaded, + point.cost.fragments_scanned, + point.cost.ranges_scanned, + point.cost.rows_scanned, + point.cost.output_rows, + ); + curve.push(point); + } + } +} + +async fn run_matrix(base_uri: &str, depths: &[u64], backend: StorageBackend) -> CostCurve { + let mut curve = Vec::new(); + for layout in [HistoryLayout::Uncompacted, HistoryLayout::Compacted] { + for &depth in depths { + let uri = format!( + "{}/{layout:?}-{depth}-{}", + base_uri.trim_end_matches('/'), + ulid::Ulid::new() + ); + let mut fixture = RetentionFixture::create(uri).await; + fixture.publish_until(depth).await; + if layout == HistoryLayout::Compacted { + fixture.compact().await; + } + + assert!(fixture.coverage().await.is_none()); + record_state( + &mut curve, + &fixture, + depth, + layout, + IndexState::Absent, + backend, + ) + .await; + + fixture.create_authority_indices().await; + assert_eq!(fixture.coverage().await.unwrap().uncovered_fragments, 0); + record_state( + &mut curve, + &fixture, + depth, + layout, + IndexState::Reconciled, + backend, + ) + .await; + + for _ in 0..UNCOVERED_TAIL { + fixture.publish_one().await; + } + assert_eq!( + fixture.coverage().await.unwrap().uncovered_fragments, + UNCOVERED_TAIL + ); + record_state( + &mut curve, + &fixture, + depth, + layout, + IndexState::BoundedTail, + backend, + ) + .await; + } + } + curve +} + +fn point( + curve: &[CurvePoint], + depth: u64, + layout: HistoryLayout, + state: IndexState, + operation: AuthorityOperation, + access: AccessMode, +) -> CurvePoint { + *curve + .iter() + .find(|point| { + point.base_depth == depth + && point.layout == layout + && point.index_state == state + && point.operation == operation + && point.access == access + }) + .unwrap_or_else(|| { + panic!("missing point {depth}/{layout:?}/{state:?}/{operation:?}/{access:?}") + }) +} + +fn expected_rows(operation: AuthorityOperation) -> u64 { + match operation { + AuthorityOperation::List => LIVE_CHECKPOINTS as u64, + AuthorityOperation::Show => (CATALOG_WIDTH + 2) as u64, + AuthorityOperation::CleanupPlan => { + (LIVE_CHECKPOINTS * (CATALOG_WIDTH + 1) + CATALOG_WIDTH + 1) as u64 + } + } +} + +fn scans_per_operation(operation: AuthorityOperation) -> u64 { + match operation { + AuthorityOperation::Show => 2, + AuthorityOperation::List | AuthorityOperation::CleanupPlan => 1, + } +} + +fn assert_flat(label: &str, shallow: u64, deep: u64) { + assert_eq!(deep, shallow, "{label} must be flat as history grows"); +} + +fn assert_grows(label: &str, shallow: u64, deep: u64) { + assert!( + deep > shallow, + "{label} must remain an explicit no-go until substrate behavior changes: shallow={shallow}, deep={deep}" + ); +} + +fn assert_changes(label: &str, shallow: u64, deep: u64) { + assert_ne!( + deep, shallow, + "{label} must remain an explicit history-sensitive no-go until substrate behavior changes" + ); +} + +fn assert_non_growing(label: &str, shallow: u64, deep: u64) { + assert!( + deep <= shallow, + "{label} grew with history: shallow={shallow}, deep={deep}" + ); +} + +fn assert_bounded_growth(label: &str, shallow: u64, deep: u64, slack: u64) { + assert!( + deep <= shallow + slack, + "{label} exceeded its fixed bound: shallow={shallow}, deep={deep}, slack={slack}" + ); +} + +fn assert_reconciled_physical_slopes( + curve: &[CurvePoint], + depths: &[u64], + backend: StorageBackend, +) { + let shallow_depth = depths[0]; + let deep_depth = *depths.last().unwrap(); + assert!(deep_depth > shallow_depth); + + for layout in [HistoryLayout::Uncompacted, HistoryLayout::Compacted] { + for operation in [ + AuthorityOperation::List, + AuthorityOperation::Show, + AuthorityOperation::CleanupPlan, + ] { + for access in [AccessMode::ColdOpen, AccessMode::WarmRepeat] { + let shallow = point( + curve, + shallow_depth, + layout, + IndexState::Reconciled, + operation, + access, + ); + let deep = point( + curve, + deep_depth, + layout, + IndexState::Reconciled, + operation, + access, + ); + + assert_flat( + "reconciled rows_scanned proxy", + shallow.cost.rows_scanned, + deep.cost.rows_scanned, + ); + assert_flat( + "reconciled scanned ranges", + shallow.cost.ranges_scanned, + deep.cost.ranges_scanned, + ); + assert_flat( + "reconciled result fragments", + shallow.cost.fragments_scanned, + deep.cost.fragments_scanned, + ); + assert_flat( + "reconciled index pages", + shallow.cost.index_pages_loaded, + deep.cost.index_pages_loaded, + ); + match layout { + HistoryLayout::Uncompacted => { + assert_flat( + "reconciled uncompacted scan I/O operations", + shallow.cost.scan_iops, + deep.cost.scan_iops, + ); + assert_flat( + "reconciled uncompacted scan requests", + shallow.cost.scan_requests, + deep.cost.scan_requests, + ); + } + HistoryLayout::Compacted => { + // At the decision-scale endpoint Lance crosses one + // additional range-read boundary in the compacted data + // file. Keep that physical transition visible and + // bounded instead of calling the operation count flat. + assert_bounded_growth( + "reconciled compacted scan I/O operations", + shallow.cost.scan_iops, + deep.cost.scan_iops, + 1, + ); + assert_bounded_growth( + "reconciled compacted scan requests", + shallow.cost.scan_requests, + deep.cost.scan_requests, + 1, + ); + } + } + + match (layout, access) { + (HistoryLayout::Uncompacted, _) => assert_flat( + "reconciled uncompacted scan bytes", + shallow.cost.scan_read_bytes, + deep.cost.scan_read_bytes, + ), + (HistoryLayout::Compacted, AccessMode::ColdOpen) => assert_changes( + "BLOCKER: history-sensitive compacted cold scan bytes", + shallow.cost.scan_read_bytes, + deep.cost.scan_read_bytes, + ), + (HistoryLayout::Compacted, AccessMode::WarmRepeat) => assert_changes( + "BLOCKER: history-sensitive compacted warm scan bytes", + shallow.cost.scan_read_bytes, + deep.cost.scan_read_bytes, + ), + } + + match (layout, access, backend) { + (HistoryLayout::Uncompacted, AccessMode::ColdOpen, StorageBackend::Local) => { + assert_non_growing( + "local uncompacted cold object reads", + shallow.cost.object_store_reads, + deep.cost.object_store_reads, + ); + assert_non_growing( + "local uncompacted cold object bytes", + shallow.cost.object_store_read_bytes, + deep.cost.object_store_read_bytes, + ); + } + (HistoryLayout::Uncompacted, AccessMode::ColdOpen, StorageBackend::S3) => { + assert_grows( + "BLOCKER: S3 uncompacted cold-open object reads", + shallow.cost.object_store_reads, + deep.cost.object_store_reads, + ); + assert_grows( + "BLOCKER: S3 uncompacted cold-open object bytes", + shallow.cost.object_store_read_bytes, + deep.cost.object_store_read_bytes, + ); + } + (HistoryLayout::Uncompacted, AccessMode::WarmRepeat, _) => { + assert_flat( + "uncompacted warm object reads", + shallow.cost.object_store_reads, + deep.cost.object_store_reads, + ); + assert_flat( + "uncompacted warm object bytes", + shallow.cost.object_store_read_bytes, + deep.cost.object_store_read_bytes, + ); + } + (HistoryLayout::Compacted, AccessMode::ColdOpen, _) => { + assert_bounded_growth( + "compacted cold object reads", + shallow.cost.object_store_reads, + deep.cost.object_store_reads, + 1, + ); + assert_changes( + "BLOCKER: history-sensitive compacted cold object bytes", + shallow.cost.object_store_read_bytes, + deep.cost.object_store_read_bytes, + ); + } + (HistoryLayout::Compacted, AccessMode::WarmRepeat, StorageBackend::Local) => { + assert_flat( + "local compacted warm object reads", + shallow.cost.object_store_reads, + deep.cost.object_store_reads, + ); + assert_flat( + "local compacted warm object bytes", + shallow.cost.object_store_read_bytes, + deep.cost.object_store_read_bytes, + ); + } + (HistoryLayout::Compacted, AccessMode::WarmRepeat, StorageBackend::S3) => { + assert_flat( + "S3 compacted warm object reads", + shallow.cost.object_store_reads, + deep.cost.object_store_reads, + ); + assert_changes( + "BLOCKER: history-sensitive S3 compacted warm object bytes", + shallow.cost.object_store_read_bytes, + deep.cost.object_store_read_bytes, + ); + } + } + } + } + } +} + +fn assert_matrix_contract(curve: &[CurvePoint], depths: &[u64], backend: StorageBackend) { + assert_eq!(curve.len(), depths.len() * 2 * 3 * 3 * 2); + for &depth in depths { + for layout in [HistoryLayout::Uncompacted, HistoryLayout::Compacted] { + for operation in [ + AuthorityOperation::List, + AuthorityOperation::Show, + AuthorityOperation::CleanupPlan, + ] { + for access in [AccessMode::ColdOpen, AccessMode::WarmRepeat] { + let absent = point(curve, depth, layout, IndexState::Absent, operation, access); + let reconciled = point( + curve, + depth, + layout, + IndexState::Reconciled, + operation, + access, + ); + let tail = point( + curve, + depth, + layout, + IndexState::BoundedTail, + operation, + access, + ); + + for sample in [absent, reconciled, tail] { + assert_eq!(sample.cost.output_rows, expected_rows(operation)); + } + assert_eq!(absent.uncovered_fragments, absent.total_fragments); + assert_eq!(absent.cost.indices_loaded, 0); + assert_eq!(absent.cost.index_pages_loaded, 0); + assert!(absent.cost.rows_scanned >= expected_rows(operation)); + + assert_eq!(reconciled.uncovered_fragments, 0); + assert_eq!(reconciled.cost.rows_scanned, expected_rows(operation)); + assert_eq!(tail.uncovered_fragments, UNCOVERED_TAIL); + assert!(tail.cost.rows_scanned >= reconciled.cost.rows_scanned); + assert!( + tail.cost.rows_scanned + <= reconciled.cost.rows_scanned + + UNCOVERED_TAIL + * ROWS_PER_HISTORY_PUBLISH + * scans_per_operation(operation) + ); + assert!( + tail.cost.fragments_scanned + <= reconciled.cost.fragments_scanned + + UNCOVERED_TAIL * scans_per_operation(operation) + ); + assert!( + tail.cost.ranges_scanned + <= reconciled.cost.ranges_scanned + + UNCOVERED_TAIL * scans_per_operation(operation) + ); + if access == AccessMode::ColdOpen { + assert!(reconciled.cost.indices_loaded > 0); + assert!(reconciled.cost.index_pages_loaded > 0); + } + } + } + } + } + + for operation in [ + AuthorityOperation::List, + AuthorityOperation::Show, + AuthorityOperation::CleanupPlan, + ] { + let shallow = point( + curve, + depths[0], + HistoryLayout::Uncompacted, + IndexState::Absent, + operation, + AccessMode::ColdOpen, + ); + let deep = point( + curve, + *depths.last().unwrap(), + HistoryLayout::Uncompacted, + IndexState::Absent, + operation, + AccessMode::ColdOpen, + ); + assert!( + deep.cost.rows_scanned > shallow.cost.rows_scanned, + "absent-index {operation:?} must expose history growth" + ); + } + + assert_reconciled_physical_slopes(curve, depths, backend); +} + +async fn run_with_cleanup( + run: RunFuture, + cleanup: CleanupFuture, + cleanup_context: &str, +) where + RunFuture: Future, + CleanupFuture: Future>, +{ + let run_result = AssertUnwindSafe(run).catch_unwind().await; + let cleanup_result = cleanup.await; + match run_result { + Ok(()) => cleanup_result.unwrap_or_else(|error| panic!("{cleanup_context}: {error}")), + Err(payload) => { + if let Err(error) = cleanup_result { + eprintln!("{cleanup_context} after matrix failure also failed: {error}"); + } + resume_unwind(payload); + } + } +} + +#[test] +fn checkpoint_name_v1_is_ascii_case_sensitive_and_collision_free() { + let accepted = [ + "a", + "Prod", + "prod", + "release/2026-07-17", + "team_a/model.v1", + "Ogcp_user", + &"x".repeat(128), + ]; + for name in accepted { + assert_eq!(normalize_checkpoint_name_v1(name), Ok(name)); + } + assert_ne!( + normalize_checkpoint_name_v1("Prod").unwrap(), + normalize_checkpoint_name_v1("prod").unwrap(), + "case folding would collide distinct checkpoint authority keys" + ); + + let too_long = "x".repeat(129); + let rejected = [ + "", + "café", + "/leading", + "trailing/", + "double//slash", + "dot..dot", + "back\\slash", + "writer.lock", + "parent/child.lock/leaf", + "__internal", + "ogcp_owned", + "space name", + "question?", + too_long.as_str(), + ]; + for name in rejected { + assert!( + normalize_checkpoint_name_v1(name).is_err(), + "reference normalizer unexpectedly accepted {name:?}" + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn local_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go() { + let dir = tempfile::tempdir().unwrap(); + let base_uri = dir.path().join("rfc025-retention-cost"); + let curve = run_matrix( + base_uri.to_str().unwrap(), + &DEFAULT_DEPTHS, + StorageBackend::Local, + ) + .await; + assert_matrix_contract(&curve, &DEFAULT_DEPTHS, StorageBackend::Local); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn s3_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go() { + let Ok(bucket) = std::env::var("OMNIGRAPH_S3_TEST_BUCKET") else { + eprintln!( + "SKIP s3_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go: \ + OMNIGRAPH_S3_TEST_BUCKET unset" + ); + return; + }; + let prefix = std::env::var("OMNIGRAPH_S3_TEST_PREFIX") + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| "omnigraph-itests".to_string()); + let base_uri = format!( + "s3://{bucket}/{prefix}/rfc025-retention-cost/{}-{}", + std::process::id(), + ulid::Ulid::new() + ); + let (store, path) = lance_io::object_store::ObjectStore::from_uri(&base_uri) + .await + .expect("configured RFC-025 S3/RustFS fixture prefix must resolve"); + run_with_cleanup( + async { + let curve = run_matrix(&base_uri, &DEFAULT_DEPTHS, StorageBackend::S3).await; + assert_matrix_contract(&curve, &DEFAULT_DEPTHS, StorageBackend::S3); + }, + async move { + store + .remove_dir_all(path) + .await + .map_err(|error| error.to_string()) + }, + "configured RFC-025 S3/RustFS fixture cleanup must succeed", + ) + .await; +} + +/// On-demand decision-scale run. It performs three complete authority reads +/// across both physical layouts and all index states at 10/100/1,000 real +/// Lance commits, so it remains outside the every-PR test path. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[ignore = "RFC-025 decision instrument: thousands of real Lance commits"] +async fn local_checkpoint_retention_matrix_at_one_thousand_commits() { + let dir = tempfile::tempdir().unwrap(); + let base_uri = dir.path().join("rfc025-retention-cost-decision"); + let depths = [10, 100, 1_000]; + let curve = run_matrix(base_uri.to_str().unwrap(), &depths, StorageBackend::Local).await; + assert_matrix_contract(&curve, &depths, StorageBackend::Local); +} diff --git a/crates/omnigraph/tests/helpers/cost.rs b/crates/omnigraph/tests/helpers/cost.rs index 5f4e2f3e..128bff54 100644 --- a/crates/omnigraph/tests/helpers/cost.rs +++ b/crates/omnigraph/tests/helpers/cost.rs @@ -59,6 +59,167 @@ pub async fn open_tracked_lance_dataset( .await } +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum AttemptOutcome { + Pending, + Success, + NotFound, + Error, + StreamStarted, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ObjectStoreAttempt { + pub method: &'static str, + pub path: Path, + pub outcome: AttemptOutcome, +} + +/// Records object-store read attempts, including failed/NotFound HEADs. +/// +/// Lance's `IOTracker` intentionally records only successful `get_opts` +/// results. Recovery classifiers need the attempted exact-key shape too, +/// because an expected NotFound is part of their proof. This wrapper records +/// before forwarding and then annotates the outcome. +#[derive(Debug, Default, Clone)] +pub struct AttemptTracker(Arc>>); + +impl AttemptTracker { + fn begin(&self, method: &'static str, path: Path) -> usize { + let mut attempts = self.0.lock().unwrap(); + let index = attempts.len(); + attempts.push(ObjectStoreAttempt { + method, + path, + outcome: AttemptOutcome::Pending, + }); + index + } + + fn finish(&self, index: usize, result: &OSResult) { + let outcome = match result { + Ok(_) => AttemptOutcome::Success, + Err(object_store::Error::NotFound { .. }) => AttemptOutcome::NotFound, + Err(_) => AttemptOutcome::Error, + }; + self.0.lock().unwrap()[index].outcome = outcome; + } + + fn record_stream(&self, method: &'static str, path: Path) { + self.0.lock().unwrap().push(ObjectStoreAttempt { + method, + path, + outcome: AttemptOutcome::StreamStarted, + }); + } + + pub fn incremental_attempts(&self) -> Vec { + std::mem::take(&mut *self.0.lock().unwrap()) + } +} + +impl WrappingObjectStore for AttemptTracker { + fn wrap(&self, _store_prefix: &str, target: Arc) -> Arc { + Arc::new(AttemptTrackingStore { + target, + tracker: self.clone(), + }) + } +} + +#[derive(Debug)] +struct AttemptTrackingStore { + target: Arc, + tracker: AttemptTracker, +} + +impl fmt::Display for AttemptTrackingStore { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "AttemptTrackingStore({})", self.target) + } +} + +#[async_trait] +impl ObjectStore for AttemptTrackingStore { + async fn put_opts( + &self, + location: &Path, + payload: PutPayload, + opts: PutOptions, + ) -> OSResult { + self.target.put_opts(location, payload, opts).await + } + + async fn put_multipart_opts( + &self, + location: &Path, + opts: PutMultipartOptions, + ) -> OSResult> { + self.target.put_multipart_opts(location, opts).await + } + + async fn get_opts(&self, location: &Path, options: GetOptions) -> OSResult { + let method = if options.head { "head" } else { "get_opts" }; + let index = self.tracker.begin(method, location.clone()); + let result = self.target.get_opts(location, options).await; + self.tracker.finish(index, &result); + result + } + + fn delete_stream( + &self, + locations: BoxStream<'static, OSResult>, + ) -> BoxStream<'static, OSResult> { + self.target.delete_stream(locations) + } + + fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, OSResult> { + self.tracker + .record_stream("list", prefix.cloned().unwrap_or_default()); + self.target.list(prefix) + } + + fn list_with_offset( + &self, + prefix: Option<&Path>, + offset: &Path, + ) -> BoxStream<'static, OSResult> { + self.tracker + .record_stream("list_with_offset", prefix.cloned().unwrap_or_default()); + self.target.list_with_offset(prefix, offset) + } + + async fn list_with_delimiter(&self, prefix: Option<&Path>) -> OSResult { + let index = self + .tracker + .begin("list_with_delimiter", prefix.cloned().unwrap_or_default()); + let result = self.target.list_with_delimiter(prefix).await; + self.tracker.finish(index, &result); + result + } + + async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> OSResult<()> { + self.target.copy_opts(from, to, options).await + } +} + +pub async fn open_attempt_tracked_lance_dataset_at_version( + uri: &str, + version: u64, + session: Arc, + tracker: &AttemptTracker, +) -> lance::Result { + DatasetBuilder::from_uri(uri) + .with_version(version) + .with_session(session) + .with_store_params(ObjectStoreParams { + object_store_wrapper: Some(Arc::new(tracker.clone())), + ..Default::default() + }) + .load() + .await +} + /// Object-store op counts for one measured operation, by table class — the /// vocabulary cost tests assert in (vs raw `IOTracker::stats().read_iops`). #[derive(Debug, Clone, Copy, Default)] diff --git a/crates/omnigraph/tests/lance_surface_guards.rs b/crates/omnigraph/tests/lance_surface_guards.rs index a974882e..abfac3fc 100644 --- a/crates/omnigraph/tests/lance_surface_guards.rs +++ b/crates/omnigraph/tests/lance_surface_guards.rs @@ -23,12 +23,18 @@ //! Functions decorated `#[tokio::test]` actually run; they construct real //! values and assert field shapes / types. +use std::collections::HashMap; use std::sync::Arc; use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, StringArray}; use arrow_schema::{DataType, Field, Schema}; use lance::Dataset; use lance::dataset::builder::DatasetBuilder; +use lance::dataset::cleanup::{CleanupPolicy, cleanup_old_versions}; +use lance::dataset::mem_wal::{ + BatchDurableWatcher, DatasetMemWalExt, InitializeMemWalBuilder, ShardManifestStore, + ShardWriter, ShardWriterConfig, WriteResult, +}; use lance::dataset::optimize::{CompactionOptions, compact_files}; use lance::dataset::refs::BranchIdentifier; use lance::dataset::transaction::{Operation, Transaction}; @@ -53,6 +59,10 @@ use lance_index::scalar::ScalarIndexParams; use lance_io::object_store::ObjectStoreRegistry; use lance_namespace::LanceNamespace; use lance_table::io::commit::{ManifestLocation, ManifestNamingScheme}; +use lance_table::system_index::mem_wal::{ + IndexCatchupProgress, MemWalIndexDetails, MergedGeneration, ShardId, ShardManifest, + ShardStatus, ShardingField, ShardingSpec, +}; use omnigraph_compiler::schema::parser::parse_schema; #[test] @@ -99,6 +109,34 @@ async fn fresh_dataset(uri: &str) -> Dataset { Dataset::write(reader, uri, Some(params)).await.unwrap() } +/// Append one uniquely keyed row while preserving the V2_2/stable-row-id shape +/// used by the production tables. Tag/cleanup guards use this to create exact, +/// distinguishable versions without introducing a graph-level writer. +async fn append_guard_row(dataset: &mut Dataset, id: &str, value: i32) { + let schema = Arc::new(Schema::from(dataset.schema())); + let batch = RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(StringArray::from(vec![id])), + Arc::new(Int32Array::from(vec![value])), + ], + ) + .unwrap(); + let reader = RecordBatchIterator::new(vec![Ok(batch)], schema); + dataset + .append( + reader, + Some(WriteParams { + mode: WriteMode::Append, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }), + ) + .await + .unwrap(); +} + /// RFC-024 Gate A candidate built exclusively from public Lance surfaces. /// /// `BranchIdentifier` distinguishes named-ref lifetimes, the current @@ -587,11 +625,14 @@ async fn public_physical_ref_token_rejects_local_same_version_aba() { recreate_named_branch_and_assert_token_changes(&mut second, true).await; } -/// Exercise the production-shaped shared-Session case. Once the -/// canonical first incarnation is cached, deleting and recreating main at the -/// same URI/version must still resolve the second incarnation rather than reuse -/// the cached transaction UUID. A fresh public DatasetBuilder open is the -/// cache-bypass fallback and must agree with the shared-Session result. +/// Exercise the production-shaped shared-Session case. "Stable" here means +/// stable across an unchanged reopen: an ordinary commit must rotate the +/// current transaction/e_tag witness while preserving the branch identifier. +/// Once the canonical first incarnation is cached, deleting and recreating +/// main at the same URI/version must still resolve the second incarnation +/// rather than reuse the cached transaction UUID. A fresh public DatasetBuilder +/// open is the cache-bypass fallback and must agree with the shared-Session +/// result. #[tokio::test] async fn local_physical_ref_token_is_stable_and_survives_shared_session_aba() { let dir = tempfile::tempdir().unwrap(); @@ -608,7 +649,7 @@ async fn local_physical_ref_token_is_stable_and_survives_shared_session_aba() { drop(first_committed); let shared_session = Arc::new(Session::default()); - let first = DatasetBuilder::from_uri(uri) + let mut first = DatasetBuilder::from_uri(uri) .with_session(shared_session.clone()) .load() .await @@ -630,6 +671,25 @@ async fn local_physical_ref_token_is_stable_and_survives_shared_session_aba() { "a canonical token must be stable across unchanged shared-Session reopens" ); drop(first_again); + + append_guard_row(&mut first, "ordinary-head-advance", 3).await; + let advanced_token = public_physical_ref_incarnation(&first).await; + assert_eq!( + advanced_token.branch_identifier, first_token.branch_identifier, + "an ordinary main commit must preserve the native branch identifier" + ); + assert_ne!( + advanced_token.transaction_uuid, first_token.transaction_uuid, + "the public composite is a current-HEAD witness: an ordinary commit must rotate its transaction UUID" + ); + assert_ne!( + advanced_token.manifest_e_tag, first_token.manifest_e_tag, + "the public composite is a current-HEAD witness: an ordinary commit must rotate its manifest e_tag" + ); + assert_ne!( + advanced_token, first_token, + "the current-HEAD witness must not be mistaken for an immutable dataset-incarnation token" + ); drop(first); std::fs::remove_dir_all(&path).expect("the first local dataset must be deleted completely"); @@ -683,8 +743,8 @@ async fn local_physical_ref_token_is_stable_and_survives_shared_session_aba() { async fn public_physical_ref_token_rejects_s3_same_version_aba() { let Ok(bucket) = std::env::var("OMNIGRAPH_S3_TEST_BUCKET") else { eprintln!( - "skipping RFC-024 S3 physical-ref token guard: \ - OMNIGRAPH_S3_TEST_BUCKET is not set" + "SKIP public_physical_ref_token_rejects_s3_same_version_aba: \ + OMNIGRAPH_S3_TEST_BUCKET unset" ); return; }; @@ -765,6 +825,149 @@ async fn public_physical_ref_token_rejects_s3_same_version_aba() { .expect("configured S3/RustFS test prefix cleanup must succeed"); } +// --- Guard 2c: RFC-026 public MemWAL enrollment/admission surfaces -------- +// +// Gate E0 consumes only public RC.1 shapes. The initializer still commits +// internally and returns no receipt; shard claim is a separate writer-open +// effect. Pin that exact boundary plus the public read-back, fencing, and +// durability-watcher surfaces. Runtime classification belongs to the isolated +// enrollment probe rather than being duplicated in this compatibility file. +#[allow( + dead_code, + unreachable_code, + unused_variables, + unused_mut, + clippy::diverging_sub_expression +)] +async fn _compile_mem_wal_enrollment_and_admission_surfaces() -> lance::Result<()> { + let dataset: &mut Dataset = unimplemented!(); + + // RFC-026 Gate E0 deliberately consumes this doc-hidden, immediate + // attached-successor probe instead of resolving latest/listing history. + let _has_exact_successor: bool = dataset.has_successor_version().await?; + + let initializer: InitializeMemWalBuilder<'_> = dataset.initialize_mem_wal(); + let _: () = initializer + .unsharded() + .writer_config_defaults(ShardWriterConfig::default()) + .add_writer_config_default("omnigraph.enrollment_id", "compile-guard") + .execute() + .await?; + + let details: Option = dataset.mem_wal_index_details().await?; + if let Some(details) = details { + let _snapshot_ts_millis: i64 = details.snapshot_ts_millis; + let _num_shards: u32 = details.num_shards; + let _inline_snapshots: Option> = details.inline_snapshots; + let sharding_specs: Vec = details.sharding_specs; + for spec in sharding_specs { + let _spec_id: u32 = spec.spec_id; + let fields: Vec = spec.fields; + for field in fields { + let _field_id: String = field.field_id; + let _source_ids: Vec = field.source_ids; + let _transform: Option = field.transform; + let _expression: Option = field.expression; + let _result_type: String = field.result_type; + let _parameters: HashMap = field.parameters; + } + } + let _maintained_indexes: Vec = details.maintained_indexes; + let _merged_generations: Vec = details.merged_generations; + let _index_catchup: Vec = details.index_catchup; + let _writer_config_defaults: HashMap = details.writer_config_defaults; + } + let _shard_ids: Vec = dataset.list_mem_wal_latest_shard_ids().await?; + + let shard_id = ShardWriterConfig::default().shard_id; + let config = ShardWriterConfig::new(shard_id) + .with_shard_spec_id(1) + .with_durable_write(true) + .with_max_wal_buffer_size(1024 * 1024); + let _config_shard_id: ShardId = config.shard_id; + let _config_shard_spec_id: u32 = config.shard_spec_id; + let _config_durable_write: bool = config.durable_write; + let _config_max_wal_buffer_size: usize = config.max_wal_buffer_size; + let writer: ShardWriter = dataset.mem_wal_writer(shard_id, config).await?; + let _writer_shard_id = writer.shard_id(); + let _writer_epoch: u64 = writer.epoch(); + let writer_manifest: Option = writer.manifest().await?; + if let Some(manifest) = writer_manifest { + let _version: u64 = manifest.version; + let _epoch: u64 = manifest.writer_epoch; + let _status: ShardStatus = manifest.status; + let _last_seen: u64 = manifest.wal_entry_position_last_seen; + } + let _: () = writer.check_fenced().await?; + + let (write, watcher): (WriteResult, Option) = + writer.put_no_wait(Vec::new()).await?; + let _batch_positions: std::ops::Range = write.batch_positions; + if let Some(mut watcher) = watcher { + let _already_durable: bool = watcher.is_durable(); + let _: () = watcher.wait().await?; + } + + let memtable_stats = writer.memtable_stats().await?; + let _row_count: usize = memtable_stats.row_count; + let _batch_count: usize = memtable_stats.batch_count; + let _estimated_size: usize = memtable_stats.estimated_size; + let _generation: u64 = memtable_stats.generation; + let _max_buffered_batch_position: Option = memtable_stats.max_buffered_batch_position; + let _max_flushed_batch_position: Option = memtable_stats.max_flushed_batch_position; + let _pending_wal_start_batch_position: Option = + memtable_stats.pending_wal_start_batch_position; + let _pending_wal_end_batch_position: Option = + memtable_stats.pending_wal_end_batch_position; + let _pending_wal_batch_count: usize = memtable_stats.pending_wal_batch_count; + let _pending_wal_row_count: usize = memtable_stats.pending_wal_row_count; + let _pending_wal_estimated_bytes: usize = memtable_stats.pending_wal_estimated_bytes; + + let wal_stats = writer.wal_stats(); + let _next_wal_entry_position: u64 = wal_stats.next_wal_entry_position; + + let _: () = writer.force_seal_active().await?; + let _: () = writer.wait_for_flush_drain().await?; + + let write_stats = writer.stats(); + let _put_count: u64 = write_stats.put_count; + let _put_time: std::time::Duration = write_stats.put_time; + let _wal_flush_count: u64 = write_stats.wal_flush_count; + let _wal_flush_time: std::time::Duration = write_stats.wal_flush_time; + let _wal_flush_bytes: u64 = write_stats.wal_flush_bytes; + let _wal_io_time: std::time::Duration = write_stats.wal_io_time; + let _wal_io_count: u64 = write_stats.wal_io_count; + let _index_update_time: std::time::Duration = write_stats.index_update_time; + let _index_update_count: u64 = write_stats.index_update_count; + let _index_update_rows: u64 = write_stats.index_update_rows; + let _memtable_flush_count: u64 = write_stats.memtable_flush_count; + let _memtable_flush_time: std::time::Duration = write_stats.memtable_flush_time; + let _memtable_flush_rows: u64 = write_stats.memtable_flush_rows; + + let manifest_store: &ShardManifestStore = unimplemented!(); + let initialized: ShardManifest = manifest_store.initialize_shard(1, HashMap::new()).await?; + let latest: Option = manifest_store.read_latest().await?; + if let Some(latest) = latest { + let reread: ShardManifest = manifest_store.read_version(latest.version).await?; + let _claim: (u64, ShardManifest) = manifest_store.claim_epoch(reread.shard_spec_id).await?; + let _: () = manifest_store.check_fenced(reread.writer_epoch).await?; + } + let _updated: ShardManifest = manifest_store + .commit_update(initialized.writer_epoch, |current| ShardManifest { + version: current.version + 1, + ..current.clone() + }) + .await?; + + let mut merge = MergeInsertBuilder::try_new(Arc::new(dataset.clone()), vec!["id".to_string()])?; + let _: &mut MergeInsertBuilder = + merge.mark_generations_as_merged(vec![MergedGeneration::new(shard_id, 1)]); + + let _: () = writer.close().await?; + + Ok(()) +} + // --- Guard 3: checkout_version + restore async chain ----------------------- // // `db/manifest/recovery.rs:505-522` chains `Dataset::open(...).await? @@ -1004,7 +1207,7 @@ async fn _compile_uncommitted_merge_insert_field_shape() -> lance::Result<()> { // The branch-delete reconciler (`db/omnigraph/optimize.rs::reconcile_orphaned_branches`) // and the eager best-effort reclaim in `cleanup_deleted_branch_tables` call // `force_delete_branch` to drop orphaned branch refs. The single-authority -// design relies on five facts pinned here: +// design relies on six facts pinned here: // 1. plain `delete_branch` errors on a missing ref (so the design uses the // force variant instead); // 2. `force_delete_branch` removes an existing (forked) branch — the orphan @@ -1019,6 +1222,9 @@ async fn _compile_uncommitted_merge_insert_field_shape() -> lance::Result<()> { // 5. a live slash-name path-child makes force delete remove an ancestor's // BranchContents but intentionally retain its dataset files. OmniGraph's // prefix-disjoint live-name invariant prevents this false-success shape. +// 6. a tag targeting a named branch does not retain `tree/{branch}`. RFC-025 +// must therefore refuse graph-branch deletion while checkpoint authority +// names that branch; the Lance tag alone is not a deletion fence. #[tokio::test] async fn force_delete_branch_semantics() { @@ -1036,12 +1242,39 @@ async fn force_delete_branch_semantics() { // (2) force_delete_branch removes an existing (forked) branch. let base = ds.version().version; - ds.create_branch("feature", base, None).await.unwrap(); + let feature = ds.create_branch("feature", base, None).await.unwrap(); + let feature_version = feature.version().version; + let branch_delete_tag = concat!( + "ogcp_v1_01J00000000000000000000000_t_", + "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" + ); + ds.tags() + .create(branch_delete_tag, ("feature", feature_version)) + .await + .expect("the RFC-025 deterministic internal spelling must be a valid Lance tag"); ds.force_delete_branch("feature").await.unwrap(); assert!( !ds.list_branches().await.unwrap().contains_key("feature"), "force_delete_branch should remove an existing branch ref" ); + assert!( + !std::path::Path::new(uri) + .join("tree") + .join("feature") + .exists(), + "a tag targeting a named branch must not retain its physical branch tree" + ); + assert_eq!( + ds.tags().get(branch_delete_tag).await.unwrap().version, + feature_version, + "branch deletion must not be mistaken for tag deletion" + ); + assert!( + ds.checkout_version(branch_delete_tag).await.is_err(), + "the surviving tag must not make a deleted branch version readable; \ + OmniGraph's checkpoint-aware branch-delete guard is load-bearing" + ); + ds.tags().delete(branch_delete_tag).await.unwrap(); // (3) Force delete is idempotent even when both the ref and tree are absent. ds.force_delete_branch("never").await.unwrap(); @@ -1097,6 +1330,129 @@ async fn force_delete_branch_semantics() { ); } +// --- Guard 9b: RFC-025 tag targets and sparse cleanup protection ----------- +// +// This is deliberately a substrate-only activation gate: it writes no +// OmniGraph checkpoint rows and changes no graph format. It pins the Lance +// facts RFC-025 would consume: +// * the proposed deterministic `ogcp_v1_...` spellings are valid tag names; +// * exact main and named-branch targets remain distinct even at overlapping +// numeric versions; +// * a sparse tagged old version survives cleanup while adjacent eligible +// versions are reclaimed; and +// * deleting the tag makes that last old version reclaimable. +#[tokio::test] +async fn native_tags_pin_exact_main_and_named_branch_versions_through_cleanup() { + const MAIN_TAG: &str = concat!( + "ogcp_v1_01J00000000000000000000000_m_", + "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" + ); + const TABLE_TAG: &str = concat!( + "ogcp_v1_01J00000000000000000000000_t_", + "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff" + ); + + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("guard9b.lance"); + let uri = uri.to_str().unwrap(); + let mut main = fresh_dataset(uri).await; + + let main_v1 = main.version().version; + append_guard_row(&mut main, "main-only", 10).await; + let main_v2 = main.version().version; + assert_eq!( + main_v2, + main_v1 + 1, + "the main fixture must have two versions" + ); + + // Fork from main v1 after main has advanced. The branch deliberately + // reuses numeric v1 so the tag's branch component is load-bearing. + let mut feature = main + .create_branch("checkpoint-feature", main_v1, None) + .await + .unwrap(); + let feature_v1 = feature.version().version; + assert_eq!(feature_v1, main_v1); + append_guard_row(&mut feature, "feature-v2", 20).await; + let feature_v2 = feature.version().version; + append_guard_row(&mut feature, "feature-v3", 30).await; + let feature_v3 = feature.version().version; + append_guard_row(&mut feature, "feature-v4", 40).await; + let feature_v4 = feature.version().version; + assert_eq!((feature_v2, feature_v3, feature_v4), (2, 3, 4)); + + let main_head_before_tags = main.version().version; + let feature_head_before_tags = feature.version().version; + main.tags() + .create(MAIN_TAG, (None::<&str>, Some(main_v1))) + .await + .expect("RFC-025's deterministic manifest-tag spelling must be accepted"); + main.tags() + .create(TABLE_TAG, ("checkpoint-feature", feature_v2)) + .await + .expect("RFC-025's deterministic table-tag spelling must be accepted"); + + let main_contents = main.tags().get(MAIN_TAG).await.unwrap(); + assert_eq!(main_contents.branch, None); + assert_eq!(main_contents.version, main_v1); + let table_contents = main.tags().get(TABLE_TAG).await.unwrap(); + assert_eq!(table_contents.branch.as_deref(), Some("checkpoint-feature")); + assert_eq!(table_contents.version, feature_v2); + assert_eq!( + main.version().version, + main_head_before_tags, + "tag creation is auxiliary metadata and must not advance main" + ); + assert_eq!( + feature.version().version, + feature_head_before_tags, + "tag creation is auxiliary metadata and must not advance the named branch" + ); + + let tagged_main = main.checkout_version(MAIN_TAG).await.unwrap(); + assert_eq!(tagged_main.version().version, main_v1); + assert_eq!(tagged_main.count_rows(None).await.unwrap(), 2); + let tagged_feature = main.checkout_version(TABLE_TAG).await.unwrap(); + assert_eq!(tagged_feature.version().version, feature_v2); + assert_eq!(tagged_feature.count_rows(None).await.unwrap(), 3); + + let cleanup_policy = CleanupPolicy { + before_version: Some(feature_v4), + error_if_tagged_old_versions: false, + ..Default::default() + }; + let removed = cleanup_old_versions(&feature, cleanup_policy.clone()) + .await + .unwrap(); + assert_eq!( + removed.old_versions, 2, + "branch v1 and v3 are eligible and untagged; sparse tagged v2 must survive" + ); + assert!(feature.checkout_version(feature_v1).await.is_err()); + assert!(feature.checkout_version(feature_v3).await.is_err()); + assert!(feature.checkout_version(feature_v2).await.is_ok()); + assert!(feature.checkout_version(feature_v4).await.is_ok()); + assert!( + main.checkout_version(MAIN_TAG).await.is_ok(), + "branch cleanup must not confuse an overlapping main-version tag with a branch tag" + ); + + main.tags().delete(TABLE_TAG).await.unwrap(); + let removed = cleanup_old_versions(&feature, cleanup_policy) + .await + .unwrap(); + assert_eq!( + removed.old_versions, 1, + "deleting the tag must make the formerly pinned branch v2 reclaimable" + ); + assert!(feature.checkout_version(feature_v2).await.is_err()); + assert!( + main.checkout_version(MAIN_TAG).await.is_ok(), + "deleting one deterministic tag must not disturb a different checkpoint target" + ); +} + // --- Guard 10: blob-column compaction works in this Lance ------------------ // // Historical: through Lance 7.0.0, `compact_files` forced @@ -1791,7 +2147,8 @@ async fn unenforced_primary_key_is_immutable_once_set() { // // RFC-023 can rely on Lance's key-conflict fencing only when every keyed // insert produces an `Operation::Update.inserted_rows_filter`. On the pinned -// Lance revision that is a route-dependent contract: the v2 plan emits a filter when the ordered +// Lance revision that +// is a route-dependent contract: the v2 plan emits a filter when the ordered // ON field ids exactly match the unenforced PK, while the scalar-index v1 path // and non-PK ON shapes emit `None`. A matched-only v2 update still emits // `Some(empty Bloom)`, which is important because `Some` selects the strict diff --git a/crates/omnigraph/tests/memwal_enrollment_gate.rs b/crates/omnigraph/tests/memwal_enrollment_gate.rs new file mode 100644 index 00000000..0bb003ce --- /dev/null +++ b/crates/omnigraph/tests/memwal_enrollment_gate.rs @@ -0,0 +1,2079 @@ +//! RFC-026 Gate E0: production-neutral evidence for recoverable MemWAL enrollment. +//! +//! This file deliberately uses only public Lance RC.1 APIs. It does not add a +//! production writer, graph format, sidecar, or recovery path. Instead, it asks +//! whether an opaque `initialize_mem_wal().execute()` effect can be classified +//! after a lost acknowledgement as exactly the one allowed successor of a +//! captured dataset HEAD. +//! +//! The evidence is intentionally bounded: +//! - the observed receipt is reconstructed after the initializer commits; Lance +//! still does not return a caller-owned enrollment receipt; +//! - the durable enrollment marker lives in the MemWAL index's persisted writer +//! defaults and is checked independently of the replaceable index UUID; +//! - shard sealing/reopening is a second public effect, not atomic with index +//! enrollment. The lifecycle probe below proves the primitive, not a combined +//! enrollment transaction. + +mod helpers; + +use std::collections::HashMap; +use std::sync::Arc; + +use arrow_array::{Int32Array, RecordBatch, RecordBatchIterator, StringArray}; +use arrow_schema::{DataType, Field, Schema}; +use futures::{TryStream, TryStreamExt}; +use lance::Dataset; +use lance::dataset::builder::DatasetBuilder; +use lance::dataset::mem_wal::{ + DatasetMemWalExt, ShardManifestStore, ShardWriter, ShardWriterConfig, +}; +use lance::dataset::refs::BranchIdentifier; +use lance::dataset::transaction::{Operation, Transaction, TransactionBuilder, UpdateMap}; +use lance::dataset::{ + CommitBuilder, MergeInsertBuilder, WhenMatched, WhenNotMatched, WriteMode, WriteParams, +}; +use lance::datatypes::LANCE_UNENFORCED_PRIMARY_KEY; +use lance::index::DatasetIndexExt; +use lance::session::Session; +use lance_file::version::LanceFileVersion; +use lance_index::IndexType; +use lance_index::mem_wal::{ + FlushedGeneration, MEM_WAL_INDEX_NAME, MergedGeneration, ShardId, ShardStatus, +}; +use lance_index::scalar::ScalarIndexParams; +use object_store::{ObjectStoreExt, path::Path}; + +use helpers::cost::{ + AttemptOutcome, AttemptTracker, open_attempt_tracked_lance_dataset_at_version, +}; + +const ENROLLMENT_ID_KEY: &str = "omnigraph.enrollment_id"; +const CONFIG_VERSION_KEY: &str = "omnigraph.stream_config_version"; +const CONFIG_VERSION: &str = "1"; +const WAL_BUFFER_BYTES: usize = 1_048_576; + +#[derive(Debug, Clone, PartialEq, Eq)] +struct PreEnrollmentHead { + version: u64, + branch_identifier: BranchIdentifier, + transaction_uuid: String, + manifest_e_tag: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct ObservedEnrollmentReceipt { + version: u64, + transaction_uuid: String, + mem_wal_index_uuid: ShardId, + manifest_e_tag: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum CombinedEnrollmentState { + ExactNoEffect, + ExactIndexOnly(ObservedEnrollmentReceipt), + ExactIndexAndExpectedEmptyShard(ObservedEnrollmentReceipt), +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct RawMemWalRootInventory { + shard_ids: Vec, + malformed_prefixes: Vec, + loose_objects: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct StrictShardObjectInventory { + manifest_versions: Vec, + unexpected_objects: Vec, +} + +impl RawMemWalRootInventory { + fn is_empty(&self) -> bool { + self.shard_ids.is_empty() + && self.malformed_prefixes.is_empty() + && self.loose_objects.is_empty() + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct CurrentHeadWitness { + version: u64, + branch_identifier: BranchIdentifier, + transaction_uuid: String, + manifest_e_tag: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] +struct ProbeAttemptShape { + method: &'static str, + object_kind: &'static str, + outcome: AttemptOutcome, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct ExactEnrollmentProbeCost { + attempts: Vec, +} + +fn reject_if(condition: bool, message: impl Into) -> Result<(), String> { + if condition { + Err(message.into()) + } else { + Ok(()) + } +} + +fn relative_object_path(root: &Path, location: &Path) -> Result { + let root = root.as_ref().trim_end_matches('/'); + let location = location.as_ref(); + if root.is_empty() { + return Ok(location.trim_start_matches('/').to_string()); + } + let prefix = format!("{root}/"); + location + .strip_prefix(&prefix) + .map(str::to_string) + .ok_or_else(|| format!("object {location:?} is outside inventory root {root:?}")) +} + +fn documented_manifest_version(relative: &str) -> Option> { + if relative == "manifest/version_hint.json" { + return Some(None); + } + let Some(filename) = relative.strip_prefix("manifest/") else { + return None; + }; + let Some(bits) = filename.strip_suffix(".binpb") else { + return None; + }; + if bits.len() != 64 || !bits.bytes().all(|byte| matches!(byte, b'0' | b'1')) { + return None; + } + let reversed = u64::from_str_radix(bits, 2).ok()?; + Some(Some(reversed.reverse_bits())) +} + +fn shard_manifest_object_relative(shard_id: ShardId, version: u64) -> String { + format!("{shard_id}/manifest/{:064b}.binpb", version.reverse_bits()) +} + +async fn raw_mem_wal_root_inventory(dataset: &Dataset) -> Result { + let object_store = dataset + .object_store(None) + .await + .map_err(|error| error.to_string())?; + let root = dataset.branch_location().path.join("_mem_wal"); + let listed = object_store + .inner + .list_with_delimiter(Some(&root)) + .await + .map_err(|error| error.to_string())?; + let mut shard_ids = Vec::new(); + let mut malformed_prefixes = Vec::new(); + for prefix in listed.common_prefixes { + let relative = relative_object_path(&root, &prefix)?; + let component = relative.trim_end_matches('/'); + match ShardId::parse_str(component) { + Ok(shard_id) if !component.contains('/') => shard_ids.push(shard_id), + _ => malformed_prefixes.push(relative), + } + } + let mut loose_objects = listed + .objects + .iter() + .map(|metadata| relative_object_path(&root, &metadata.location)) + .collect::, _>>()?; + shard_ids.sort(); + malformed_prefixes.sort(); + loose_objects.sort(); + Ok(RawMemWalRootInventory { + shard_ids, + malformed_prefixes, + loose_objects, + }) +} + +async fn put_raw_mem_wal_object(dataset: &Dataset, relative: &str) { + let object_store = dataset.object_store(None).await.unwrap(); + let root = dataset.branch_location().path.join("_mem_wal"); + let path = Path::parse(format!("{}/{relative}", root.as_ref())).unwrap(); + object_store + .inner + .put(&path, "gate-e0".into()) + .await + .unwrap(); +} + +async fn strict_try_collect(stream: S) -> Result, E> +where + S: TryStream + Unpin, +{ + stream.try_collect().await +} + +async fn strict_shard_object_inventory( + dataset: &Dataset, + shard_id: ShardId, +) -> Result { + let object_store = dataset + .object_store(None) + .await + .map_err(|error| error.to_string())?; + let shard_component = shard_id.as_hyphenated().to_string(); + let shard_path = dataset + .branch_location() + .path + .join("_mem_wal") + .join(shard_component.as_str()); + // `try_collect` is intentional. Lance's public `list_versions` logs and + // swallows per-item listing errors; enrollment evidence must instead fail + // closed if even one recursive listing item cannot be observed. + let listed = strict_try_collect(object_store.inner.list(Some(&shard_path))) + .await + .map_err(|error| error.to_string())?; + let mut manifest_versions = Vec::new(); + let mut unexpected_objects = Vec::new(); + for metadata in listed { + let relative = relative_object_path(&shard_path, &metadata.location)?; + match documented_manifest_version(&relative) { + Some(Some(version)) => manifest_versions.push(version), + Some(None) => {} + None => unexpected_objects.push(relative), + } + } + manifest_versions.sort_unstable(); + unexpected_objects.sort(); + Ok(StrictShardObjectInventory { + manifest_versions, + unexpected_objects, + }) +} + +async fn unexpected_shard_objects( + dataset: &Dataset, + shard_id: ShardId, +) -> Result, String> { + Ok(strict_shard_object_inventory(dataset, shard_id) + .await? + .unexpected_objects) +} + +async fn exact_empty_shard_manifest( + dataset: &Dataset, + shard_id: ShardId, +) -> Result { + let object_store = dataset + .object_store(None) + .await + .map_err(|error| error.to_string())?; + let branch_path = dataset.branch_location().path; + let store = ShardManifestStore::new(object_store, &branch_path, shard_id, 2); + let inventory = strict_shard_object_inventory(dataset, shard_id).await?; + reject_if( + inventory.manifest_versions != vec![1], + format!( + "expected empty shard has non-initial manifest history: {:?}", + inventory.manifest_versions + ), + )?; + let manifest = store + .read_latest() + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "expected shard has no manifest".to_string())?; + validate_exact_fresh_shard_manifest(&manifest, shard_id)?; + reject_if( + !inventory.unexpected_objects.is_empty(), + format!( + "expected empty shard has WAL/generation/unknown manifest objects: {:?}", + inventory.unexpected_objects + ), + )?; + Ok(manifest) +} + +fn validate_exact_fresh_shard_manifest( + manifest: &lance_index::mem_wal::ShardManifest, + shard_id: ShardId, +) -> Result<(), String> { + reject_if( + manifest.shard_id != shard_id + || manifest.version != 1 + || manifest.shard_spec_id != 1 + || !manifest.shard_field_values.is_empty() + || manifest.writer_epoch != 1 + || manifest.replay_after_wal_entry_position != 0 + || manifest.wal_entry_position_last_seen != 0 + || manifest.current_generation != 1 + || !manifest.flushed_generations.is_empty() + || manifest.status != ShardStatus::Active, + format!("expected shard is not the exact fresh epoch-1 state: {manifest:?}"), + ) +} + +async fn assert_high_level_writer_is_exact_empty( + dataset: &Dataset, + writer: &ShardWriter, + expected_shard_id: ShardId, +) { + assert_eq!(writer.shard_id(), expected_shard_id); + assert_eq!(writer.epoch(), 1); + writer.check_fenced().await.unwrap(); + assert_eq!( + dataset.list_mem_wal_latest_shard_ids().await.unwrap(), + vec![expected_shard_id], + "high-level writer open must create exactly the pre-minted shard" + ); + assert_eq!( + raw_mem_wal_root_inventory(dataset).await.unwrap(), + RawMemWalRootInventory { + shard_ids: vec![expected_shard_id], + malformed_prefixes: Vec::new(), + loose_objects: Vec::new(), + }, + "raw MemWAL root inventory must agree with the high-level singleton shard" + ); + + let memtable = writer.memtable_stats().await.unwrap(); + assert_eq!(memtable.row_count, 0); + assert_eq!(memtable.batch_count, 0); + assert_eq!(memtable.generation, 1); + assert_eq!(memtable.max_buffered_batch_position, None); + assert_eq!(memtable.max_flushed_batch_position, None); + assert_eq!(memtable.pending_wal_start_batch_position, None); + assert_eq!(memtable.pending_wal_end_batch_position, None); + assert_eq!(memtable.pending_wal_batch_count, 0); + assert_eq!(memtable.pending_wal_row_count, 0); + assert_eq!(memtable.pending_wal_estimated_bytes, 0); + assert_eq!( + writer.wal_stats().next_wal_entry_position, + 1, + "a fresh writer's next cursor is one because no 1-based WAL entry exists" + ); + + let stats = writer.stats(); + assert_eq!(stats.put_count, 0); + assert_eq!(stats.wal_flush_count, 0); + assert_eq!(stats.wal_flush_bytes, 0); + assert_eq!(stats.wal_io_count, 0); + assert_eq!(stats.index_update_count, 0); + assert_eq!(stats.index_update_rows, 0); + assert_eq!(stats.memtable_flush_count, 0); + assert_eq!(stats.memtable_flush_rows, 0); + + let writer_manifest = writer.manifest().await.unwrap().unwrap(); + assert_eq!( + writer_manifest, + exact_empty_shard_manifest(dataset, expected_shard_id) + .await + .unwrap() + ); +} + +fn expected_writer_defaults(enrollment_id: &str) -> HashMap { + HashMap::from([ + (ENROLLMENT_ID_KEY.to_string(), enrollment_id.to_string()), + (CONFIG_VERSION_KEY.to_string(), CONFIG_VERSION.to_string()), + ("durable_write".to_string(), "true".to_string()), + ( + "max_wal_buffer_size".to_string(), + WAL_BUFFER_BYTES.to_string(), + ), + ]) +} + +fn intended_shard_writer_config(shard_id: ShardId) -> ShardWriterConfig { + // RC.1 persists defaults in the MemWAL index but does not apply them to a + // caller-provided writer config. Keep the admission effect exact by + // reconstructing the intended persisted values explicitly. + ShardWriterConfig::new(shard_id) + .with_shard_spec_id(1) + .with_durable_write(true) + .with_max_wal_buffer_size(WAL_BUFFER_BYTES) +} + +async fn shard_manifest_store(dataset: &Dataset, shard_id: ShardId) -> ShardManifestStore { + let object_store = dataset.object_store(None).await.unwrap(); + ShardManifestStore::new(object_store, &dataset.branch_location().path, shard_id, 2) +} + +async fn capture_pre_enrollment_head(dataset: &Dataset) -> Result { + reject_if( + dataset + .mem_wal_index_details() + .await + .map_err(|error| error.to_string())? + .is_some(), + "pre-enrollment HEAD already contains a MemWAL index", + )?; + + let transaction = dataset + .read_transaction() + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "pre-enrollment HEAD has no transaction".to_string())?; + reject_if( + transaction.uuid.is_empty(), + "pre-enrollment transaction UUID is empty", + )?; + + Ok(PreEnrollmentHead { + version: dataset.version().version, + branch_identifier: dataset + .branch_identifier() + .await + .map_err(|error| error.to_string())?, + transaction_uuid: transaction.uuid, + manifest_e_tag: dataset.manifest_location().e_tag.clone(), + }) +} + +/// Observe the only version chain admitted by the bounded profile. +/// +/// The caller holds the exclusive base-HEAD gate and unresolved recovery keeps +/// cleanup out, so attached versions cannot be skipped. Under that boundary, +/// opening deterministic `N` and probing its immediate successor is both +/// history-flat and stricter than resolving "latest". If `N + 1` exists, the +/// same immediate-successor probe on it rejects a buried `N + 2`; otherwise it +/// is the only possible effect. Every probe/open error fails closed. +async fn exact_enrollment_successor_dataset( + before: &PreEnrollmentHead, + dataset: &Dataset, +) -> Result, String> { + let successor_version = before + .version + .checked_add(1) + .ok_or_else(|| "pre-enrollment version cannot have a successor".to_string())?; + reject_if( + lance_table::format::is_detached_version(successor_version), + "pre-enrollment version crosses the detached-version boundary", + )?; + let predecessor = dataset + .checkout_version(before.version) + .await + .map_err(|error| format!("cannot read captured enrollment predecessor: {error}"))?; + let predecessor_transaction = predecessor + .read_transaction() + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "enrollment predecessor has no transaction".to_string())?; + reject_if( + predecessor + .branch_identifier() + .await + .map_err(|error| error.to_string())? + != before.branch_identifier + || predecessor_transaction.uuid != before.transaction_uuid + || predecessor.manifest_location().e_tag != before.manifest_e_tag + || predecessor + .mem_wal_index_details() + .await + .map_err(|error| error.to_string())? + .is_some(), + "enrollment predecessor is not the captured physical no-MemWAL HEAD", + )?; + + let has_successor = predecessor + .has_successor_version() + .await + .map_err(|error| format!("cannot probe enrollment successor: {error}"))?; + if !has_successor { + return Ok(None); + } + + let buried_version = successor_version + .checked_add(1) + .ok_or_else(|| "enrollment successor cannot have a successor".to_string())?; + reject_if( + lance_table::format::is_detached_version(buried_version), + "enrollment successor crosses the detached-version boundary", + )?; + let successor = predecessor + .checkout_version(successor_version) + .await + .map_err(|error| format!("cannot read observed enrollment successor: {error}"))?; + reject_if( + successor + .has_successor_version() + .await + .map_err(|error| format!("cannot probe buried enrollment effect: {error}"))?, + format!("enrollment effect is buried by attached version {buried_version}"), + )?; + Ok(Some(successor)) +} + +/// Classify only the exact singleton MemWAL-index successor of `before`. +/// +/// This is deliberately stricter than "MemWAL now exists": a wrong operation, +/// an intervening version, a ref-incarnation change, or any configuration drift +/// fails closed. It never resolves latest; all authority comes from exact +/// known-version probes under the bounded profile's exclusive HEAD gate. +async fn classify_exact_enrollment_successor( + before: &PreEnrollmentHead, + dataset: &Dataset, + expected_defaults: &HashMap, +) -> Result { + let successor = exact_enrollment_successor_dataset(before, dataset) + .await? + .ok_or_else(|| "exact enrollment successor is absent".to_string())?; + classify_exact_enrollment_successor_on_exact_dataset(before, &successor, expected_defaults) + .await +} + +async fn classify_exact_enrollment_successor_on_exact_dataset( + before: &PreEnrollmentHead, + dataset: &Dataset, + expected_defaults: &HashMap, +) -> Result { + let version = dataset.version().version; + let expected_version = before + .version + .checked_add(1) + .ok_or_else(|| "pre-enrollment version cannot have a successor".to_string())?; + reject_if( + version != expected_version, + format!( + "enrollment is not the exact one-version successor: before={}, after={version}", + before.version + ), + )?; + + let branch_identifier = dataset + .branch_identifier() + .await + .map_err(|error| error.to_string())?; + reject_if( + branch_identifier != before.branch_identifier, + "branch/ref incarnation changed during enrollment", + )?; + + let transaction = dataset + .read_transaction() + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "enrollment successor has no transaction".to_string())?; + reject_if( + transaction.read_version != before.version, + format!( + "enrollment transaction read version {} instead of {}", + transaction.read_version, before.version + ), + )?; + reject_if( + transaction.uuid.is_empty() || transaction.uuid == before.transaction_uuid, + "enrollment successor did not mint a distinct transaction UUID", + )?; + reject_if( + transaction.tag.is_some() || transaction.transaction_properties.is_some(), + "enrollment successor carries a tag or transaction_properties that the initializer never emits", + )?; + + let (new_indices, removed_indices) = match &transaction.operation { + Operation::CreateIndex { + new_indices, + removed_indices, + } => (new_indices, removed_indices), + _ => return Err("exact enrollment successor is not Operation::CreateIndex".to_string()), + }; + reject_if( + new_indices.len() != 1 || !removed_indices.is_empty(), + "enrollment CreateIndex is not one add with zero removals", + )?; + let index = &new_indices[0]; + reject_if( + index.name != MEM_WAL_INDEX_NAME, + format!( + "enrollment created index {:?}, expected {MEM_WAL_INDEX_NAME:?}", + index.name + ), + )?; + reject_if( + index.dataset_version != before.version + || !index.fields.is_empty() + || index.fragment_bitmap.is_some() + || index.index_details.is_none() + || index.index_version != 0 + || index.created_at.is_none() + || index.base_id.is_some() + || index.files.is_some(), + "MemWAL index metadata does not have the exact inline system-index shape", + )?; + + let details = dataset + .mem_wal_index_details() + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "CreateIndex successor has no readable MemWAL details".to_string())?; + reject_if( + details.snapshot_ts_millis != 0 + || details.num_shards != 1 + || details.inline_snapshots.is_some() + || details.sharding_specs.len() != 1 + || !details.maintained_indexes.is_empty() + || !details.merged_generations.is_empty() + || !details.index_catchup.is_empty(), + "MemWAL details do not match a fresh unsharded enrollment", + )?; + let sharding_spec = &details.sharding_specs[0]; + reject_if( + sharding_spec.spec_id != 1 || sharding_spec.fields.len() != 1, + "fresh unsharded enrollment has an unexpected sharding spec", + )?; + let sharding_field = &sharding_spec.fields[0]; + reject_if( + sharding_field.field_id != "bucket" + || !sharding_field.source_ids.is_empty() + || sharding_field.transform.as_deref() != Some("unsharded") + || sharding_field.expression.is_some() + || sharding_field.result_type != "int32" + || !sharding_field.parameters.is_empty(), + "fresh unsharded enrollment has an unexpected sharding field", + )?; + reject_if( + &details.writer_config_defaults != expected_defaults, + format!( + "persisted enrollment marker/config mismatch: expected={expected_defaults:?}, actual={:?}", + details.writer_config_defaults + ), + )?; + + let manifest_e_tag = dataset.manifest_location().e_tag.clone(); + if let Some(before_e_tag) = before.manifest_e_tag.as_deref() { + let after_e_tag = manifest_e_tag + .as_deref() + .ok_or_else(|| "enrollment successor lost the manifest e_tag".to_string())?; + reject_if( + after_e_tag == before_e_tag, + "enrollment successor reused the pre-enrollment manifest e_tag", + )?; + } + + Ok(ObservedEnrollmentReceipt { + version, + transaction_uuid: transaction.uuid.clone(), + mem_wal_index_uuid: index.uuid, + manifest_e_tag, + }) +} + +async fn classify_combined_enrollment_state( + before: &PreEnrollmentHead, + dataset: &Dataset, + expected_defaults: &HashMap, + expected_shard_id: ShardId, +) -> Result { + let raw_inventory = raw_mem_wal_root_inventory(dataset).await?; + let successor = exact_enrollment_successor_dataset(before, dataset).await?; + + if successor.is_none() { + reject_if( + !raw_inventory.is_empty(), + "exact captured no-effect state has MemWAL artifacts", + )?; + return Ok(CombinedEnrollmentState::ExactNoEffect); + } + + let receipt = classify_exact_enrollment_successor_on_exact_dataset( + before, + &successor.expect("checked above"), + expected_defaults, + ) + .await?; + if raw_inventory.is_empty() { + return Ok(CombinedEnrollmentState::ExactIndexOnly(receipt)); + } + let expected_inventory = RawMemWalRootInventory { + shard_ids: vec![expected_shard_id], + malformed_prefixes: Vec::new(), + loose_objects: Vec::new(), + }; + reject_if( + raw_inventory != expected_inventory, + format!( + "enrollment has foreign, malformed, extra, or loose artifacts: expected={expected_inventory:?}, actual={raw_inventory:?}" + ), + )?; + exact_empty_shard_manifest(dataset, expected_shard_id).await?; + Ok(CombinedEnrollmentState::ExactIndexAndExpectedEmptyShard( + receipt, + )) +} + +async fn current_mem_wal_index_uuid(dataset: &Dataset) -> ShardId { + let matches = dataset + .load_indices() + .await + .unwrap() + .iter() + .filter(|index| index.name == MEM_WAL_INDEX_NAME) + .map(|index| index.uuid) + .collect::>(); + assert_eq!( + matches.len(), + 1, + "a valid enrollment must have one singleton MemWAL system index" + ); + matches[0] +} + +async fn initialize_with_defaults( + dataset: &mut Dataset, + defaults: &HashMap, +) -> lance::Result<()> { + let mut builder = dataset.initialize_mem_wal().unsharded(); + for (key, value) in defaults { + builder = builder.add_writer_config_default(key.clone(), value.clone()); + } + builder.execute().await +} + +async fn initialize_bucket_with_defaults( + dataset: &mut Dataset, + defaults: &HashMap, +) -> lance::Result<()> { + let mut builder = dataset.initialize_mem_wal().bucket_sharding("id", 1); + for (key, value) in defaults { + builder = builder.add_writer_config_default(key.clone(), value.clone()); + } + builder.execute().await +} + +async fn initialize_and_close_empty_shard( + dataset: &mut Dataset, + defaults: &HashMap, + shard_id: ShardId, +) { + initialize_with_defaults(dataset, defaults).await.unwrap(); + let writer = dataset + .mem_wal_writer(shard_id, intended_shard_writer_config(shard_id)) + .await + .unwrap(); + writer.close().await.unwrap(); +} + +async fn fresh_dataset(uri: &str) -> Dataset { + let schema = Arc::new(Schema::new(vec![ + Field::new("id", DataType::Utf8, false).with_metadata(HashMap::from([( + LANCE_UNENFORCED_PRIMARY_KEY.to_string(), + "true".to_string(), + )])), + Field::new("value", DataType::Int32, false), + ])); + let batch = RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(StringArray::from(vec!["alice"])), + Arc::new(Int32Array::from(vec![1])), + ], + ) + .unwrap(); + let reader = RecordBatchIterator::new([Ok(batch)], schema); + Dataset::write( + reader, + uri, + Some(WriteParams { + mode: WriteMode::Create, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }), + ) + .await + .unwrap() +} + +fn one_row(dataset: &Dataset, id: &str, value: i32) -> RecordBatch { + let schema = Arc::new(Schema::from(dataset.schema())); + RecordBatch::try_new( + schema, + vec![ + Arc::new(StringArray::from(vec![id])), + Arc::new(Int32Array::from(vec![value])), + ], + ) + .unwrap() +} + +async fn append_row(dataset: &mut Dataset, id: &str, value: i32) { + let batch = one_row(dataset, id, value); + let reader = RecordBatchIterator::new([Ok(batch.clone())], batch.schema()); + dataset + .append( + reader, + Some(WriteParams { + mode: WriteMode::Append, + enable_stable_row_ids: true, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }), + ) + .await + .unwrap(); +} + +async fn merge_row_and_mark_generation( + dataset: Dataset, + id: &str, + value: i32, + merged_generation: MergedGeneration, +) -> Dataset { + let batch = one_row(&dataset, id, value); + let reader = RecordBatchIterator::new([Ok(batch.clone())], batch.schema()); + let mut builder = + MergeInsertBuilder::try_new(Arc::new(dataset), vec!["id".to_string()]).unwrap(); + builder + .when_matched(WhenMatched::UpdateAll) + .when_not_matched(WhenNotMatched::InsertAll) + .conflict_retries(0) + .mark_generations_as_merged(vec![merged_generation]); + let (dataset, stats) = builder + .try_build() + .unwrap() + .execute_reader(Box::new(reader)) + .await + .unwrap(); + assert_eq!(stats.num_inserted_rows, 1); + assert_eq!(stats.num_updated_rows, 0); + (*dataset).clone() +} + +fn witness_object_kind(path: &str) -> &'static str { + if path.contains("/_versions/") || path.ends_with(".manifest") { + "manifest" + } else if path.contains("/_transactions/") { + "transaction" + } else if path.contains("/_indices/") { + "index" + } else { + "other" + } +} + +async fn prepare_enrollment_at_baseline_version( + uri: &str, + baseline_version: u64, + enrollment_id: &str, +) -> (PreEnrollmentHead, HashMap) { + let mut dataset = fresh_dataset(uri).await; + while dataset.version().version < baseline_version { + let version = dataset.version().version + 1; + let transaction = Transaction::new( + dataset.version().version, + Operation::UpdateConfig { + config_updates: Some(UpdateMap { + update_entries: vec![ + ( + "omnigraph.gate_e0_history".to_string(), + Some(format!("{version:020}")), + ) + .into(), + ], + replace: false, + }), + table_metadata_updates: None, + schema_metadata_updates: None, + field_metadata_updates: HashMap::new(), + }, + None, + ); + dataset = CommitBuilder::new(Arc::new(dataset)) + .execute(transaction) + .await + .unwrap(); + } + assert_eq!(dataset.version().version, baseline_version); + let before = capture_pre_enrollment_head(&dataset).await.unwrap(); + let defaults = expected_writer_defaults(enrollment_id); + initialize_with_defaults(&mut dataset, &defaults) + .await + .unwrap(); + assert_eq!(dataset.version().version, baseline_version + 1); + (before, defaults) +} + +/// Capture a witness from a Dataset already opened at the exact version named +/// by durable authority. This deliberately does not resolve latest. Phase A +/// must hold the exclusive base-HEAD gate across selecting that version and +/// consuming the witness; rereading a pinned Dataset is not a race detector. +async fn capture_witness_from_exact_dataset( + dataset: &Dataset, +) -> Result { + let branch_before = dataset + .branch_identifier() + .await + .map_err(|error| error.to_string())?; + let version = dataset.version().version; + let transaction = dataset + .read_transaction() + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "current HEAD has no transaction".to_string())?; + reject_if( + transaction.uuid.is_empty(), + "current HEAD has an empty transaction UUID", + )?; + let manifest_e_tag = dataset.manifest_location().e_tag.clone(); + let branch_after = dataset + .branch_identifier() + .await + .map_err(|error| error.to_string())?; + reject_if( + branch_before != branch_after || dataset.version().version != version, + "exact-version witness is internally incoherent", + )?; + Ok(CurrentHeadWitness { + version, + branch_identifier: branch_before, + transaction_uuid: transaction.uuid, + manifest_e_tag, + }) +} + +async fn measure_exact_enrollment_probe( + uri: &str, + before: &PreEnrollmentHead, + expected_defaults: &HashMap, +) -> ExactEnrollmentProbeCost { + let tracker = AttemptTracker::default(); + let anchor = open_attempt_tracked_lance_dataset_at_version( + uri, + before.version, + Arc::new(Session::default()), + &tracker, + ) + .await + .unwrap(); + let _anchor_open_attempts = tracker.incremental_attempts(); + let receipt = classify_exact_enrollment_successor(before, &anchor, expected_defaults) + .await + .unwrap(); + let successor = anchor.checkout_version(before.version + 1).await.unwrap(); + let witness = capture_witness_from_exact_dataset(&successor) + .await + .unwrap(); + let attempts = tracker.incremental_attempts(); + assert_eq!(receipt.version, before.version + 1); + assert_eq!(witness.version, receipt.version); + assert_eq!( + witness.branch_identifier, + successor.branch_identifier().await.unwrap() + ); + assert_eq!( + witness.manifest_e_tag, receipt.manifest_e_tag, + "witness must carry the current manifest e_tag" + ); + let transaction = successor + .read_transaction() + .await + .unwrap() + .expect("current-head witness requires the current transaction"); + assert_eq!(witness.transaction_uuid, transaction.uuid); + assert!(matches!( + transaction.operation, + Operation::CreateIndex { .. } + )); + assert!( + !attempts.is_empty(), + "tracked exact-version classifier must issue reads" + ); + assert!( + attempts.iter().all(|attempt| { + attempt.outcome != AttemptOutcome::Pending && !attempt.method.starts_with("list") + }), + "known-version enrollment classification must complete every attempt without resolving latest/listing history: {attempts:?}" + ); + let mut attempt_shape = attempts + .iter() + .map(|attempt| ProbeAttemptShape { + method: attempt.method, + object_kind: witness_object_kind(attempt.path.as_ref()), + outcome: attempt.outcome, + }) + .collect::>(); + attempt_shape.sort(); + ExactEnrollmentProbeCost { + attempts: attempt_shape, + } +} + +#[tokio::test] +async fn exact_enrollment_marker_survives_reopen_ordinary_commit_and_merge_progress() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("marker-survival.lance"); + let uri = uri.to_str().unwrap(); + let mut dataset = fresh_dataset(uri).await; + let before = capture_pre_enrollment_head(&dataset).await.unwrap(); + let enrollment_id = ShardId::new_v4().to_string(); + let expected_defaults = expected_writer_defaults(&enrollment_id); + + initialize_with_defaults(&mut dataset, &expected_defaults) + .await + .unwrap(); + drop(dataset); + + // Reopen instead of trusting the mutating initializer handle. This is the + // local lost-ack shape: only the precondition and durable Lance state remain. + let mut reopened = DatasetBuilder::from_uri(uri).load().await.unwrap(); + let enrollment_receipt = + classify_exact_enrollment_successor(&before, &reopened, &expected_defaults) + .await + .unwrap(); + let reopened_again = DatasetBuilder::from_uri(uri).load().await.unwrap(); + assert_eq!( + enrollment_receipt, + classify_exact_enrollment_successor(&before, &reopened_again, &expected_defaults) + .await + .unwrap(), + "the observed enrollment receipt must be stable across unchanged reopens" + ); + + append_row(&mut reopened, "bob", 2).await; + let after_append = reopened.mem_wal_index_details().await.unwrap().unwrap(); + assert_eq!(after_append.writer_config_defaults, expected_defaults); + assert!(after_append.merged_generations.is_empty()); + assert_eq!( + current_mem_wal_index_uuid(&reopened).await, + enrollment_receipt.mem_wal_index_uuid, + "ordinary data commits must preserve the MemWAL index UUID" + ); + assert!( + classify_exact_enrollment_successor(&before, &reopened, &expected_defaults) + .await + .is_err(), + "the enrollment classifier must stay one-successor-only after an ordinary commit" + ); + + let shard_id = ShardId::new_v4(); + let merged_generation = MergedGeneration::new(shard_id, 7); + let reopened = + merge_row_and_mark_generation(reopened, "carol", 3, merged_generation.clone()).await; + let after_merge = reopened.mem_wal_index_details().await.unwrap().unwrap(); + assert_eq!(after_merge.writer_config_defaults, expected_defaults); + assert_eq!(after_merge.merged_generations, vec![merged_generation]); + assert_ne!( + current_mem_wal_index_uuid(&reopened).await, + enrollment_receipt.mem_wal_index_uuid, + "merged-generation updates replace MemWAL index metadata, so enrollment identity cannot be its UUID" + ); + + let reopened_after_merge = DatasetBuilder::from_uri(uri).load().await.unwrap(); + let persisted = reopened_after_merge + .mem_wal_index_details() + .await + .unwrap() + .unwrap(); + assert_eq!(persisted.writer_config_defaults, expected_defaults); + assert_eq!( + persisted + .merged_generations + .iter() + .find(|generation| generation.shard_id == shard_id) + .map(|generation| generation.generation), + Some(7) + ); +} + +#[tokio::test] +async fn exact_enrollment_classifier_fails_closed_on_wrong_and_intervening_states() { + let dir = tempfile::tempdir().unwrap(); + + let foreign_uri = dir.path().join("foreign-marker.lance"); + let foreign_uri = foreign_uri.to_str().unwrap(); + let mut foreign = fresh_dataset(foreign_uri).await; + let foreign_before = capture_pre_enrollment_head(&foreign).await.unwrap(); + let actual_defaults = expected_writer_defaults("foreign-enrollment"); + initialize_with_defaults(&mut foreign, &actual_defaults) + .await + .unwrap(); + let expected_defaults = expected_writer_defaults("our-enrollment"); + let wrong_marker = + classify_exact_enrollment_successor(&foreign_before, &foreign, &expected_defaults) + .await + .unwrap_err(); + assert!(wrong_marker.contains("marker/config mismatch")); + + let wrong_operation_uri = dir.path().join("wrong-operation.lance"); + let wrong_operation_uri = wrong_operation_uri.to_str().unwrap(); + let mut wrong_operation = fresh_dataset(wrong_operation_uri).await; + let wrong_operation_before = capture_pre_enrollment_head(&wrong_operation).await.unwrap(); + append_row(&mut wrong_operation, "bob", 2).await; + let wrong_operation_error = classify_exact_enrollment_successor( + &wrong_operation_before, + &wrong_operation, + &expected_defaults, + ) + .await + .unwrap_err(); + assert!(wrong_operation_error.contains("Operation::CreateIndex")); + + let foreign_index_uri = dir.path().join("foreign-real-index.lance"); + let foreign_index_uri = foreign_index_uri.to_str().unwrap(); + let mut foreign_index = fresh_dataset(foreign_index_uri).await; + let foreign_index_before = capture_pre_enrollment_head(&foreign_index).await.unwrap(); + foreign_index + .create_index_builder(&["value"], IndexType::BTree, &ScalarIndexParams::default()) + .await + .unwrap(); + let foreign_index_transaction = foreign_index.read_transaction().await.unwrap().unwrap(); + assert!(matches!( + foreign_index_transaction.operation, + Operation::CreateIndex { .. } + )); + assert_eq!( + foreign_index.version().version, + foreign_index_before.version + 1 + ); + let foreign_index_error = classify_exact_enrollment_successor( + &foreign_index_before, + &foreign_index, + &expected_defaults, + ) + .await + .unwrap_err(); + assert!(foreign_index_error.contains("expected \"__lance_mem_wal\"")); + + let intervening_uri = dir.path().join("intervening-version.lance"); + let intervening_uri = intervening_uri.to_str().unwrap(); + let mut intervening = fresh_dataset(intervening_uri).await; + let intervening_before = capture_pre_enrollment_head(&intervening).await.unwrap(); + append_row(&mut intervening, "bob", 2).await; + initialize_with_defaults(&mut intervening, &expected_defaults) + .await + .unwrap(); + let intervening_error = + classify_exact_enrollment_successor(&intervening_before, &intervening, &expected_defaults) + .await + .unwrap_err(); + assert!(intervening_error.contains("enrollment effect is buried")); +} + +#[tokio::test] +async fn exact_enrollment_classifier_rejects_wrong_sharding_tag_and_properties() { + let dir = tempfile::tempdir().unwrap(); + let defaults = expected_writer_defaults("exact-negative-operation-shape"); + + let wrong_sharding_uri = dir.path().join("wrong-sharding.lance"); + let wrong_sharding_uri = wrong_sharding_uri.to_str().unwrap(); + let mut wrong_sharding = fresh_dataset(wrong_sharding_uri).await; + let wrong_sharding_before = capture_pre_enrollment_head(&wrong_sharding).await.unwrap(); + initialize_bucket_with_defaults(&mut wrong_sharding, &defaults) + .await + .unwrap(); + let wrong_sharding_error = + classify_exact_enrollment_successor(&wrong_sharding_before, &wrong_sharding, &defaults) + .await + .unwrap_err(); + assert!(wrong_sharding_error.contains("fresh unsharded enrollment")); + + // Reuse the initializer's exact public CreateIndex operation so tag and + // transaction-properties are the only differences in the two successors. + let template_uri = dir.path().join("operation-template.lance"); + let template_uri = template_uri.to_str().unwrap(); + let mut template = fresh_dataset(template_uri).await; + initialize_with_defaults(&mut template, &defaults) + .await + .unwrap(); + let enrollment_operation = template + .read_transaction() + .await + .unwrap() + .unwrap() + .operation + .clone(); + assert!(matches!( + enrollment_operation, + Operation::CreateIndex { .. } + )); + + let tagged_uri = dir.path().join("tagged-successor.lance"); + let tagged_uri = tagged_uri.to_str().unwrap(); + let tagged = fresh_dataset(tagged_uri).await; + let tagged_before = capture_pre_enrollment_head(&tagged).await.unwrap(); + let tagged_transaction = + TransactionBuilder::new(tagged_before.version, enrollment_operation.clone()) + .tag(Some("foreign-tag".to_string())) + .build(); + let tagged = CommitBuilder::new(Arc::new(tagged)) + .execute(tagged_transaction) + .await + .unwrap(); + let tagged_error = classify_exact_enrollment_successor(&tagged_before, &tagged, &defaults) + .await + .unwrap_err(); + assert!(tagged_error.contains("tag or transaction_properties")); + + let properties_uri = dir.path().join("properties-successor.lance"); + let properties_uri = properties_uri.to_str().unwrap(); + let properties = fresh_dataset(properties_uri).await; + let properties_before = capture_pre_enrollment_head(&properties).await.unwrap(); + let properties_transaction = + TransactionBuilder::new(properties_before.version, enrollment_operation) + .transaction_properties(Some(Arc::new(HashMap::from([( + "foreign-key".to_string(), + "foreign-value".to_string(), + )])))) + .build(); + let properties = CommitBuilder::new(Arc::new(properties)) + .execute(properties_transaction) + .await + .unwrap(); + let properties_error = + classify_exact_enrollment_successor(&properties_before, &properties, &defaults) + .await + .unwrap_err(); + assert!(properties_error.contains("tag or transaction_properties")); +} + +#[tokio::test] +async fn combined_classifier_rejects_raw_memwal_artifacts_in_every_state() { + let dir = tempfile::tempdir().unwrap(); + + let foreign_uri = dir.path().join("no-effect-foreign-shard.lance"); + let foreign_uri = foreign_uri.to_str().unwrap(); + let foreign = fresh_dataset(foreign_uri).await; + let foreign_before = capture_pre_enrollment_head(&foreign).await.unwrap(); + let foreign_defaults = expected_writer_defaults("no-effect-foreign-shard"); + let foreign_shard_id = ShardId::new_v4(); + shard_manifest_store(&foreign, foreign_shard_id) + .await + .initialize_shard(1, HashMap::new()) + .await + .unwrap(); + let foreign_error = classify_combined_enrollment_state( + &foreign_before, + &foreign, + &foreign_defaults, + ShardId::new_v4(), + ) + .await + .unwrap_err(); + assert!(foreign_error.contains("exact captured no-effect state")); + + let malformed_uri = dir.path().join("no-effect-malformed-prefix.lance"); + let malformed_uri = malformed_uri.to_str().unwrap(); + let malformed = fresh_dataset(malformed_uri).await; + let malformed_before = capture_pre_enrollment_head(&malformed).await.unwrap(); + let malformed_defaults = expected_writer_defaults("no-effect-malformed-prefix"); + put_raw_mem_wal_object(&malformed, "not-a-uuid/manifest/rogue.tmp").await; + let malformed_error = classify_combined_enrollment_state( + &malformed_before, + &malformed, + &malformed_defaults, + ShardId::new_v4(), + ) + .await + .unwrap_err(); + assert!(malformed_error.contains("exact captured no-effect state")); + + let loose_uri = dir.path().join("no-effect-loose-object.lance"); + let loose_uri = loose_uri.to_str().unwrap(); + let loose = fresh_dataset(loose_uri).await; + let loose_before = capture_pre_enrollment_head(&loose).await.unwrap(); + let loose_defaults = expected_writer_defaults("no-effect-loose-object"); + put_raw_mem_wal_object(&loose, "loose-object").await; + let loose_error = classify_combined_enrollment_state( + &loose_before, + &loose, + &loose_defaults, + ShardId::new_v4(), + ) + .await + .unwrap_err(); + assert!(loose_error.contains("exact captured no-effect state")); + + let unexpected_uri = dir.path().join("unexpected-manifest-key.lance"); + let unexpected_uri = unexpected_uri.to_str().unwrap(); + let mut unexpected = fresh_dataset(unexpected_uri).await; + let unexpected_before = capture_pre_enrollment_head(&unexpected).await.unwrap(); + let unexpected_defaults = expected_writer_defaults("unexpected-manifest-key"); + initialize_with_defaults(&mut unexpected, &unexpected_defaults) + .await + .unwrap(); + let expected_shard_id = ShardId::new_v4(); + let writer = unexpected + .mem_wal_writer( + expected_shard_id, + intended_shard_writer_config(expected_shard_id), + ) + .await + .unwrap(); + writer.close().await.unwrap(); + put_raw_mem_wal_object( + &unexpected, + &format!("{expected_shard_id}/manifest/rogue.tmp"), + ) + .await; + let unexpected_error = classify_combined_enrollment_state( + &unexpected_before, + &unexpected, + &unexpected_defaults, + expected_shard_id, + ) + .await + .unwrap_err(); + assert!( + unexpected_error.contains("unknown manifest objects"), + "unexpected manifest key must fail through the exact filename whitelist: {unexpected_error}" + ); +} + +#[tokio::test] +async fn strict_inventory_collection_rejects_a_partial_listing() { + let listing = futures::stream::iter(vec![ + Ok::<_, &'static str>("visible-object"), + Err("injected-list-page-failure"), + ]); + assert_eq!( + strict_try_collect(listing).await.unwrap_err(), + "injected-list-page-failure", + "a successful prefix followed by a listing error must never become partial clean evidence" + ); +} + +#[test] +fn exact_enrollment_chain_cannot_regress_to_latest_or_history_listing() { + let source = include_str!("memwal_enrollment_gate.rs"); + let start = source + .find("async fn exact_enrollment_successor_dataset") + .expect("exact enrollment-chain classifier must exist"); + let end = source[start..] + .find("/// Classify only the exact singleton") + .map(|offset| start + offset) + .expect("exact enrollment-chain classifier boundary must exist"); + let classifier = &source[start..end]; + for forbidden in [ + ".checkout_latest(", + ".latest_version_id(", + ".is_stale(", + ".versions(", + ] { + assert!( + !classifier.contains(forbidden), + "Gate E0's bounded classifier must use exact immediate-successor probes, not {forbidden}" + ); + } + assert!(classifier.contains(".has_successor_version()")); + assert!(classifier.contains(".checkout_version(before.version)")); + assert!(classifier.contains(".checkout_version(successor_version)")); +} + +#[tokio::test] +async fn empty_shard_classifier_rejects_cursor_and_flushed_generation_progress() { + let shard_id = ShardId::new_v4(); + let exact = lance_index::mem_wal::ShardManifest { + shard_id, + version: 1, + shard_spec_id: 1, + shard_field_values: HashMap::new(), + writer_epoch: 1, + replay_after_wal_entry_position: 0, + wal_entry_position_last_seen: 0, + current_generation: 1, + flushed_generations: Vec::new(), + status: ShardStatus::Active, + }; + validate_exact_fresh_shard_manifest(&exact, shard_id).unwrap(); + + let cursor = lance_index::mem_wal::ShardManifest { + wal_entry_position_last_seen: 1, + ..exact.clone() + }; + assert!(validate_exact_fresh_shard_manifest(&cursor, shard_id).is_err()); + + let replay_cursor = lance_index::mem_wal::ShardManifest { + replay_after_wal_entry_position: 1, + ..exact.clone() + }; + assert!(validate_exact_fresh_shard_manifest(&replay_cursor, shard_id).is_err()); + + let flushed = lance_index::mem_wal::ShardManifest { + flushed_generations: vec![FlushedGeneration { + generation: 1, + path: "orphan-generation-1".to_string(), + }], + ..exact + }; + assert!(validate_exact_fresh_shard_manifest(&flushed, shard_id).is_err()); +} + +#[tokio::test] +async fn combined_classifier_rejects_persisted_cursor_and_flushed_generation_progress() { + let dir = tempfile::tempdir().unwrap(); + + let cursor_uri = dir.path().join("persisted-cursor-progress.lance"); + let cursor_uri = cursor_uri.to_str().unwrap(); + let mut cursor_dataset = fresh_dataset(cursor_uri).await; + let cursor_before = capture_pre_enrollment_head(&cursor_dataset).await.unwrap(); + let cursor_defaults = expected_writer_defaults("persisted-cursor-progress"); + let cursor_shard_id = ShardId::new_v4(); + initialize_and_close_empty_shard(&mut cursor_dataset, &cursor_defaults, cursor_shard_id).await; + let cursor_store = shard_manifest_store(&cursor_dataset, cursor_shard_id).await; + cursor_store + .commit_update(1, |current| lance_index::mem_wal::ShardManifest { + version: current.version + 1, + wal_entry_position_last_seen: 1, + ..current.clone() + }) + .await + .unwrap(); + let persisted_cursor = cursor_store.read_latest().await.unwrap().unwrap(); + assert_eq!(persisted_cursor.version, 2); + assert_eq!(persisted_cursor.wal_entry_position_last_seen, 1); + assert_eq!( + strict_shard_object_inventory(&cursor_dataset, cursor_shard_id) + .await + .unwrap() + .manifest_versions, + vec![1, 2] + ); + let cursor_error = classify_combined_enrollment_state( + &cursor_before, + &cursor_dataset, + &cursor_defaults, + cursor_shard_id, + ) + .await + .unwrap_err(); + assert!(cursor_error.contains("non-initial manifest history")); + + let flushed_uri = dir.path().join("persisted-flushed-generation.lance"); + let flushed_uri = flushed_uri.to_str().unwrap(); + let mut flushed_dataset = fresh_dataset(flushed_uri).await; + let flushed_before = capture_pre_enrollment_head(&flushed_dataset).await.unwrap(); + let flushed_defaults = expected_writer_defaults("persisted-flushed-generation"); + let flushed_shard_id = ShardId::new_v4(); + initialize_and_close_empty_shard(&mut flushed_dataset, &flushed_defaults, flushed_shard_id) + .await; + let flushed_store = shard_manifest_store(&flushed_dataset, flushed_shard_id).await; + flushed_store + .commit_update(1, |current| lance_index::mem_wal::ShardManifest { + version: current.version + 1, + current_generation: 2, + flushed_generations: vec![FlushedGeneration { + generation: 1, + path: "generation-1.lance".to_string(), + }], + ..current.clone() + }) + .await + .unwrap(); + let persisted_flushed = flushed_store.read_latest().await.unwrap().unwrap(); + assert_eq!(persisted_flushed.version, 2); + assert_eq!(persisted_flushed.current_generation, 2); + assert_eq!(persisted_flushed.flushed_generations.len(), 1); + assert_eq!( + strict_shard_object_inventory(&flushed_dataset, flushed_shard_id) + .await + .unwrap() + .manifest_versions, + vec![1, 2] + ); + let flushed_error = classify_combined_enrollment_state( + &flushed_before, + &flushed_dataset, + &flushed_defaults, + flushed_shard_id, + ) + .await + .unwrap_err(); + assert!(flushed_error.contains("non-initial manifest history")); +} + +#[tokio::test] +async fn combined_classifier_rejects_a_corrupt_latest_shard_manifest() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("corrupt-latest-shard-manifest.lance"); + let uri = uri.to_str().unwrap(); + let mut dataset = fresh_dataset(uri).await; + let before = capture_pre_enrollment_head(&dataset).await.unwrap(); + let defaults = expected_writer_defaults("corrupt-latest-shard-manifest"); + let shard_id = ShardId::new_v4(); + initialize_and_close_empty_shard(&mut dataset, &defaults, shard_id).await; + assert!(matches!( + classify_combined_enrollment_state(&before, &dataset, &defaults, shard_id) + .await + .unwrap(), + CombinedEnrollmentState::ExactIndexAndExpectedEmptyShard(_) + )); + + put_raw_mem_wal_object(&dataset, &shard_manifest_object_relative(shard_id, 1)).await; + let inventory = strict_shard_object_inventory(&dataset, shard_id) + .await + .unwrap(); + assert_eq!(inventory.manifest_versions, vec![1]); + assert!(inventory.unexpected_objects.is_empty()); + let corrupt_error = classify_combined_enrollment_state(&before, &dataset, &defaults, shard_id) + .await + .unwrap_err(); + assert!( + corrupt_error.contains("decode") || corrupt_error.contains("protobuf"), + "corrupt latest manifest must fail closed after strict enumeration: {corrupt_error}" + ); +} + +#[tokio::test] +async fn combined_classifier_uses_exact_successor_probes_from_stale_handles() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("stale-handles.lance"); + let uri = uri.to_str().unwrap(); + let mut authority = fresh_dataset(uri).await; + let stale_at_n = DatasetBuilder::from_uri(uri).load().await.unwrap(); + let before = capture_pre_enrollment_head(&stale_at_n).await.unwrap(); + let defaults = expected_writer_defaults("stale-handles"); + let expected_shard_id = ShardId::new_v4(); + + initialize_with_defaults(&mut authority, &defaults) + .await + .unwrap(); + assert!(matches!( + classify_combined_enrollment_state(&before, &stale_at_n, &defaults, expected_shard_id,) + .await + .unwrap(), + CombinedEnrollmentState::ExactIndexOnly(_) + )); + + let stale_at_n_plus_one = DatasetBuilder::from_uri(uri).load().await.unwrap(); + assert_eq!(stale_at_n_plus_one.version().version, before.version + 1); + append_row(&mut authority, "intervening", 2).await; + let buried_error = classify_combined_enrollment_state( + &before, + &stale_at_n_plus_one, + &defaults, + expected_shard_id, + ) + .await + .unwrap_err(); + assert!(buried_error.contains("enrollment effect is buried")); +} + +#[tokio::test] +async fn high_level_writer_and_combined_enrollment_truth_table_are_exact() { + let dir = tempfile::tempdir().unwrap(); + + let positive_uri = dir.path().join("combined-positive.lance"); + let positive_uri = positive_uri.to_str().unwrap(); + let mut positive = fresh_dataset(positive_uri).await; + let positive_before = capture_pre_enrollment_head(&positive).await.unwrap(); + let positive_defaults = expected_writer_defaults("combined-positive"); + let expected_shard_id = ShardId::new_v4(); + assert_eq!( + classify_combined_enrollment_state( + &positive_before, + &positive, + &positive_defaults, + expected_shard_id, + ) + .await + .unwrap(), + CombinedEnrollmentState::ExactNoEffect + ); + + initialize_with_defaults(&mut positive, &positive_defaults) + .await + .unwrap(); + let index_only = classify_combined_enrollment_state( + &positive_before, + &positive, + &positive_defaults, + expected_shard_id, + ) + .await + .unwrap(); + assert!(matches!( + index_only, + CombinedEnrollmentState::ExactIndexOnly(_) + )); + + let writer = positive + .mem_wal_writer( + expected_shard_id, + intended_shard_writer_config(expected_shard_id), + ) + .await + .unwrap(); + assert_high_level_writer_is_exact_empty(&positive, &writer, expected_shard_id).await; + let index_and_shard = classify_combined_enrollment_state( + &positive_before, + &positive, + &positive_defaults, + expected_shard_id, + ) + .await + .unwrap(); + assert!(matches!( + index_and_shard, + CombinedEnrollmentState::ExactIndexAndExpectedEmptyShard(_) + )); + writer.close().await.unwrap(); + + let foreign_uri = dir.path().join("combined-foreign-shard.lance"); + let foreign_uri = foreign_uri.to_str().unwrap(); + let mut foreign = fresh_dataset(foreign_uri).await; + let foreign_before = capture_pre_enrollment_head(&foreign).await.unwrap(); + let foreign_defaults = expected_writer_defaults("combined-foreign"); + initialize_with_defaults(&mut foreign, &foreign_defaults) + .await + .unwrap(); + let foreign_shard_id = ShardId::new_v4(); + let expected_foreign_shard_id = ShardId::new_v4(); + let foreign_writer = foreign + .mem_wal_writer( + foreign_shard_id, + intended_shard_writer_config(foreign_shard_id), + ) + .await + .unwrap(); + let foreign_error = classify_combined_enrollment_state( + &foreign_before, + &foreign, + &foreign_defaults, + expected_foreign_shard_id, + ) + .await + .unwrap_err(); + assert!(foreign_error.contains("foreign, malformed, extra, or loose artifacts")); + foreign_writer.close().await.unwrap(); + + let data_uri = dir.path().join("combined-data-bearing.lance"); + let data_uri = data_uri.to_str().unwrap(); + let mut data_bearing = fresh_dataset(data_uri).await; + let data_before = capture_pre_enrollment_head(&data_bearing).await.unwrap(); + let data_defaults = expected_writer_defaults("combined-data-bearing"); + initialize_with_defaults(&mut data_bearing, &data_defaults) + .await + .unwrap(); + let data_shard_id = ShardId::new_v4(); + let data_writer = data_bearing + .mem_wal_writer(data_shard_id, intended_shard_writer_config(data_shard_id)) + .await + .unwrap(); + let write = data_writer + .put(vec![one_row(&data_bearing, "wal-row", 99)]) + .await + .unwrap(); + assert_eq!(write.batch_positions, 0..1); + assert!( + !unexpected_shard_objects(&data_bearing, data_shard_id) + .await + .unwrap() + .is_empty(), + "a durable put must leave an observable WAL object" + ); + assert!( + classify_combined_enrollment_state( + &data_before, + &data_bearing, + &data_defaults, + data_shard_id, + ) + .await + .is_err(), + "a data-bearing shard is never an admissible enrollment-only effect" + ); + data_writer.abort().await.unwrap(); +} + +#[tokio::test] +async fn exact_known_version_enrollment_probe_is_flat_across_history_depth() { + const SHALLOW_VERSION: u64 = 8; + const DEEP_VERSION: u64 = 80; + + let dir = tempfile::tempdir().unwrap(); + let shallow_uri = dir.path().join("exact-probe-shallow.lance"); + let shallow_uri = shallow_uri.to_str().unwrap(); + let (shallow_before, shallow_defaults) = + prepare_enrollment_at_baseline_version(shallow_uri, SHALLOW_VERSION, "exact-probe-shallow") + .await; + let shallow = + measure_exact_enrollment_probe(shallow_uri, &shallow_before, &shallow_defaults).await; + + let deep_uri = dir.path().join("exact-probe-deep.lance"); + let deep_uri = deep_uri.to_str().unwrap(); + let (deep_before, deep_defaults) = + prepare_enrollment_at_baseline_version(deep_uri, DEEP_VERSION, "exact-probe-deep").await; + let deep = measure_exact_enrollment_probe(deep_uri, &deep_before, &deep_defaults).await; + + assert_eq!( + shallow.attempts, deep.attempts, + "exact known-version enrollment/witness attempt shape, including the expected NotFound successor probe, must stay flat as retained history grows; shallow={shallow:?}, deep={deep:?}" + ); + assert!(shallow.attempts.len() <= 8); + eprintln!( + "MemWAL exact enrollment/witness probe: shallow_version={SHALLOW_VERSION} shallow={shallow:?}; \ + deep_version={DEEP_VERSION} deep={deep:?}" + ); +} + +#[cfg(unix)] +#[tokio::test] +async fn exact_successor_probe_does_not_enumerate_the_versions_directory() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + let dataset_path = dir.path().join("exact-probe-no-list.lance"); + let uri = dataset_path.to_str().unwrap(); + let mut current = fresh_dataset(uri).await; + for version in 2..=16 { + append_row( + &mut current, + &format!("no-list-history-{version}"), + version as i32, + ) + .await; + } + let predecessor = current.checkout_version(1).await.unwrap(); + let versions_dir = dataset_path.join("_versions"); + let original_permissions = std::fs::metadata(&versions_dir).unwrap().permissions(); + let mut execute_only = original_permissions.clone(); + execute_only.set_mode(0o111); + std::fs::set_permissions(&versions_dir, execute_only).unwrap(); + + // Capture both results before restoring permissions so a failed assertion + // cannot strand an unreadable temp directory. + let successor_probe = predecessor.has_successor_version().await; + let exact_successor = predecessor.checkout_version(2).await; + let mut latest_probe = predecessor.clone(); + let latest_resolution = latest_probe.checkout_latest().await; + std::fs::set_permissions(&versions_dir, original_permissions.clone()).unwrap(); + + assert_eq!(successor_probe.unwrap(), true); + assert_eq!(exact_successor.unwrap().version().version, 2); + assert!( + latest_resolution.is_err(), + "the pinned local latest resolver should be unable to enumerate an execute-only versions directory; this negative control proves the exact successor path did not list history" + ); + + let mut no_access = original_permissions.clone(); + no_access.set_mode(0o000); + std::fs::set_permissions(&versions_dir, no_access).unwrap(); + let denied_successor_probe = predecessor.has_successor_version().await; + std::fs::set_permissions(&versions_dir, original_permissions).unwrap(); + assert!( + denied_successor_probe.is_err(), + "an unreadable exact successor must be an error, never false/no-effect" + ); +} + +#[tokio::test] +async fn public_shard_lifecycle_can_seal_refuse_reopen_and_claim() { + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().join("shard-lifecycle.lance"); + let uri = uri.to_str().unwrap(); + let mut dataset = fresh_dataset(uri).await; + let defaults = expected_writer_defaults("shard-lifecycle-enrollment"); + initialize_with_defaults(&mut dataset, &defaults) + .await + .unwrap(); + + let shard_id = ShardId::new_v4(); + let object_store = dataset.object_store(None).await.unwrap(); + let branch_path = dataset.branch_location().path; + let store = ShardManifestStore::new(object_store, &branch_path, shard_id, 2); + let initial = store.initialize_shard(1, HashMap::new()).await.unwrap(); + assert_eq!(initial.version, 1); + assert_eq!(initial.writer_epoch, 0); + assert_eq!(initial.status, ShardStatus::Active); + assert!( + dataset + .list_mem_wal_latest_shard_ids() + .await + .unwrap() + .contains(&shard_id), + "the initialized shard must be discoverable through the public dataset API" + ); + + let sealed = store + .commit_update(0, |current| lance_index::mem_wal::ShardManifest { + version: current.version + 1, + status: ShardStatus::Sealed, + ..current.clone() + }) + .await + .unwrap(); + assert_eq!(sealed.status, ShardStatus::Sealed); + let before_refused_claim = store.read_latest().await.unwrap().unwrap(); + let claim_error = store.claim_epoch(1).await.unwrap_err(); + assert!(claim_error.to_string().contains("sealed")); + assert_eq!( + store.read_latest().await.unwrap().unwrap(), + before_refused_claim, + "a refused claim must not mutate a sealed shard" + ); + + let active = store + .commit_update(0, |current| lance_index::mem_wal::ShardManifest { + version: current.version + 1, + status: ShardStatus::Active, + ..current.clone() + }) + .await + .unwrap(); + assert_eq!(active.status, ShardStatus::Active); + let (epoch, claimed) = store.claim_epoch(1).await.unwrap(); + assert_eq!(epoch, 1); + assert_eq!(claimed.writer_epoch, 1); + assert_eq!(claimed.status, ShardStatus::Active); +} + +async fn remove_object_store_fixture(uri: &str) { + let (store, path) = lance_io::object_store::ObjectStore::from_uri(uri) + .await + .expect("configured object-store fixture must resolve for cleanup"); + store + .remove_dir_all(path) + .await + .expect("configured object-store fixture cleanup must succeed"); +} + +/// Bucket-gated RC.1 cell for the exact backend where enrollment recovery +/// matters. CI's RustFS job sets `OMNIGRAPH_S3_TEST_BUCKET`, making this test +/// non-vacuous there; local runs without the fixture skip explicitly. +#[tokio::test(flavor = "multi_thread")] +async fn s3_memwal_enrollment_gate_positive_and_listing_negatives() { + let Ok(bucket) = std::env::var("OMNIGRAPH_S3_TEST_BUCKET") else { + eprintln!( + "SKIP s3_memwal_enrollment_gate_positive_and_listing_negatives: \ + OMNIGRAPH_S3_TEST_BUCKET unset" + ); + return; + }; + let prefix = std::env::var("OMNIGRAPH_S3_TEST_PREFIX") + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| "omnigraph-itests".to_string()); + let unique = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system clock must be after the Unix epoch") + .as_nanos(); + let base_uri = format!( + "s3://{bucket}/{prefix}/memwal-enrollment-gate/{}-{unique}", + std::process::id() + ); + let uri = format!("{base_uri}-positive.lance"); + + let mut dataset = fresh_dataset(&uri).await; + let before = capture_pre_enrollment_head(&dataset).await.unwrap(); + assert!( + before.manifest_e_tag.is_some(), + "configured S3/RustFS backend must expose the pre-enrollment manifest e_tag" + ); + let expected_defaults = expected_writer_defaults(&ShardId::new_v4().to_string()); + let expected_shard_id = ShardId::new_v4(); + assert_eq!( + classify_combined_enrollment_state( + &before, + &dataset, + &expected_defaults, + expected_shard_id, + ) + .await + .unwrap(), + CombinedEnrollmentState::ExactNoEffect + ); + initialize_with_defaults(&mut dataset, &expected_defaults) + .await + .unwrap(); + + // Simulate a lost acknowledgement: discard the only handle that observed + // `execute()` returning and reconstruct the result solely from durable S3. + drop(dataset); + let rustfs_probe = measure_exact_enrollment_probe(&uri, &before, &expected_defaults).await; + assert!( + rustfs_probe + .attempts + .iter() + .any(|attempt| attempt.method == "head" && attempt.outcome == AttemptOutcome::NotFound), + "RustFS exact-successor evidence must include the N+2 NotFound HEAD: {rustfs_probe:?}" + ); + eprintln!("RustFS MemWAL exact enrollment/witness probe: {rustfs_probe:?}"); + let first_reopen = DatasetBuilder::from_uri(&uri) + .load() + .await + .expect("lost-ack S3/RustFS enrollment must reopen"); + let first_receipt = + classify_exact_enrollment_successor(&before, &first_reopen, &expected_defaults) + .await + .unwrap(); + assert!(first_receipt.manifest_e_tag.is_some()); + assert!(matches!( + classify_combined_enrollment_state( + &before, + &first_reopen, + &expected_defaults, + expected_shard_id, + ) + .await + .unwrap(), + CombinedEnrollmentState::ExactIndexOnly(_) + )); + let writer = first_reopen + .mem_wal_writer( + expected_shard_id, + intended_shard_writer_config(expected_shard_id), + ) + .await + .expect("S3/RustFS must admit the pre-minted high-level shard"); + assert_high_level_writer_is_exact_empty(&first_reopen, &writer, expected_shard_id).await; + assert!(matches!( + classify_combined_enrollment_state( + &before, + &first_reopen, + &expected_defaults, + expected_shard_id, + ) + .await + .unwrap(), + CombinedEnrollmentState::ExactIndexAndExpectedEmptyShard(_) + )); + writer + .close() + .await + .expect("empty S3/RustFS shard writer must close cleanly"); + drop(first_reopen); + + let unchanged_reopen = DatasetBuilder::from_uri(&uri) + .load() + .await + .expect("unchanged S3/RustFS enrollment must reopen again"); + let unchanged_receipt = + classify_exact_enrollment_successor(&before, &unchanged_reopen, &expected_defaults) + .await + .unwrap(); + assert_eq!( + first_receipt, unchanged_receipt, + "lost-ack classification must be stable across an unchanged object-store reopen" + ); + assert!(matches!( + classify_combined_enrollment_state( + &before, + &unchanged_reopen, + &expected_defaults, + expected_shard_id, + ) + .await + .unwrap(), + CombinedEnrollmentState::ExactIndexAndExpectedEmptyShard(_) + )); + drop(unchanged_reopen); + + remove_object_store_fixture(&uri).await; + + // S3/RustFS must exercise the listing-dependent refusal paths too. Local + // negatives alone cannot prove delimiter/recursive-list behavior on the + // backend where recovery matters. + let foreign_uri = format!("{base_uri}-foreign-shard.lance"); + let mut foreign = fresh_dataset(&foreign_uri).await; + let foreign_before = capture_pre_enrollment_head(&foreign).await.unwrap(); + let foreign_defaults = expected_writer_defaults("s3-foreign-shard"); + initialize_with_defaults(&mut foreign, &foreign_defaults) + .await + .unwrap(); + let expected_foreign_shard = ShardId::new_v4(); + let actual_foreign_shard = ShardId::new_v4(); + let foreign_writer = foreign + .mem_wal_writer( + actual_foreign_shard, + intended_shard_writer_config(actual_foreign_shard), + ) + .await + .unwrap(); + let foreign_error = classify_combined_enrollment_state( + &foreign_before, + &foreign, + &foreign_defaults, + expected_foreign_shard, + ) + .await + .unwrap_err(); + assert!(foreign_error.contains("foreign, malformed, extra, or loose artifacts")); + foreign_writer.close().await.unwrap(); + drop(foreign); + remove_object_store_fixture(&foreign_uri).await; + + let malformed_uri = format!("{base_uri}-malformed-root.lance"); + let malformed = fresh_dataset(&malformed_uri).await; + let malformed_before = capture_pre_enrollment_head(&malformed).await.unwrap(); + let malformed_defaults = expected_writer_defaults("s3-malformed-root"); + put_raw_mem_wal_object(&malformed, "not-a-uuid/manifest/rogue.tmp").await; + put_raw_mem_wal_object(&malformed, "loose-object").await; + let malformed_error = classify_combined_enrollment_state( + &malformed_before, + &malformed, + &malformed_defaults, + ShardId::new_v4(), + ) + .await + .unwrap_err(); + assert!(malformed_error.contains("no-effect state has MemWAL artifacts")); + drop(malformed); + remove_object_store_fixture(&malformed_uri).await; + + let data_uri = format!("{base_uri}-durable-wal.lance"); + let mut data_bearing = fresh_dataset(&data_uri).await; + let data_before = capture_pre_enrollment_head(&data_bearing).await.unwrap(); + let data_defaults = expected_writer_defaults("s3-durable-wal"); + initialize_with_defaults(&mut data_bearing, &data_defaults) + .await + .unwrap(); + let data_shard = ShardId::new_v4(); + let data_writer = data_bearing + .mem_wal_writer(data_shard, intended_shard_writer_config(data_shard)) + .await + .unwrap(); + data_writer + .put(vec![one_row(&data_bearing, "s3-wal-row", 101)]) + .await + .unwrap(); + let data_error = + classify_combined_enrollment_state(&data_before, &data_bearing, &data_defaults, data_shard) + .await + .unwrap_err(); + assert!(data_error.contains("WAL/generation/unknown manifest objects")); + data_writer.abort().await.unwrap(); + drop(data_bearing); + remove_object_store_fixture(&data_uri).await; + + let cursor_uri = format!("{base_uri}-cursor-progress.lance"); + let mut cursor = fresh_dataset(&cursor_uri).await; + let cursor_before = capture_pre_enrollment_head(&cursor).await.unwrap(); + let cursor_defaults = expected_writer_defaults("s3-cursor-progress"); + let cursor_shard = ShardId::new_v4(); + initialize_and_close_empty_shard(&mut cursor, &cursor_defaults, cursor_shard).await; + shard_manifest_store(&cursor, cursor_shard) + .await + .commit_update(1, |current| lance_index::mem_wal::ShardManifest { + version: current.version + 1, + wal_entry_position_last_seen: 1, + ..current.clone() + }) + .await + .unwrap(); + let cursor_error = + classify_combined_enrollment_state(&cursor_before, &cursor, &cursor_defaults, cursor_shard) + .await + .unwrap_err(); + assert!(cursor_error.contains("non-initial manifest history")); + drop(cursor); + remove_object_store_fixture(&cursor_uri).await; + + let corrupt_uri = format!("{base_uri}-corrupt-manifest.lance"); + let mut corrupt = fresh_dataset(&corrupt_uri).await; + let corrupt_before = capture_pre_enrollment_head(&corrupt).await.unwrap(); + let corrupt_defaults = expected_writer_defaults("s3-corrupt-manifest"); + let corrupt_shard = ShardId::new_v4(); + initialize_and_close_empty_shard(&mut corrupt, &corrupt_defaults, corrupt_shard).await; + put_raw_mem_wal_object(&corrupt, &shard_manifest_object_relative(corrupt_shard, 1)).await; + let corrupt_error = classify_combined_enrollment_state( + &corrupt_before, + &corrupt, + &corrupt_defaults, + corrupt_shard, + ) + .await + .unwrap_err(); + assert!(corrupt_error.contains("decode") || corrupt_error.contains("protobuf")); + drop(corrupt); + remove_object_store_fixture(&corrupt_uri).await; +} diff --git a/docs/dev/canon.md b/docs/dev/canon.md index b2741782..92089d1a 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -595,8 +595,13 @@ Every publish appends one `graph_commit` row (ULID, parents, actor) and moves `entity_at`, historical queries, diffs via `diff_between`/`diff_commits`) reads older manifest states; retention is governed by `cleanup` (see [Maintenance](#maintenance-optimize-cleanup-repair)). Checkpoint-pinned -retention — named snapshots as authoritative retention roots — is -**(draft RFC-025)**. +retention — named snapshots as authoritative retention roots — remains the +target architecture in **RFC-025**, but its first in-manifest BTREE access +shape is **research-blocked**. Lance tag guards prove exact sparse pins and the +branch-delete caveat; the local 10→1,000 decision run rejects activation because +compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 +warm (exact show grows too). No checkpoint format or API is active; production +remains on internal schema v6. ### Three-way merge @@ -922,7 +927,9 @@ cross-version rebuild/refusal evidence, duplicate-repair runbook, and passed **Roadmap / research:** durable table heads / heads format **(RFC-024, research-blocked after Gate A rejected the first -physical access shape)**; checkpoint-pinned retention **(RFC-025, draft)**; +physical access shape)**; checkpoint-pinned retention **(RFC-025, +research-blocked after Gate 0 rejected the current compacted registry-access +shape)**; MemWAL streaming ingest **(RFC-026, draft)**; lineage-based merge deltas **(RFC-027, research-blocked)**; background reconciler; planner statistics/cost model; policy pushdown; ingest-time embeddings; per-query @@ -937,10 +944,10 @@ resource budgets. | **R1: Destructive recovery beside a live foreign process.** Lance restore orphans concurrent commits; no conditional ref primitives; gates are process-local. | High | Documented single-writer-process support boundary; exact ownership prevents false adoption; distributed fence (lease on the schema-apply lock branch) is the sketched close **(roadmap)**. Do not promote multi-process write topologies before it exists. | | **R2: Pre-stable Lance pin.** 9.0.0-rc.1 via git rev; prereleases have regressed mid-line before (blob reads broke in beta.13, fixed beta.15). Blocks crates.io publishing (v0.8.1 is binaries-only; v0.9.0 gated on 9.0.0 stable). | High | Full alignment audit per bump (all commits reviewed, findings in [lance.md](lance.md)); surface guards as first smoke check; `cargo test --workspace` as the alignment gate, never the build alone. | | **R3: RFC-023 consumes a route-dependent pinned-Lance key-filter primitive.** Lance's filter emission and filtered/unfiltered resolution remain directional on RC.1 (revalidated 2026-07-17). | Medium | v6 closes production insertion-bearing routes through exact-`id`, forced-v2 filtered staging and source-guards bare Append; the adapter verifies the emitted field-ID filter and effect-aware recovery refuses ambiguity. Guards pin both conflict orders so any upstream symmetry/route change forces an audit. Historical beta.21 release evidence passed the 1M forced-v2 50 ms median / 256 MiB max-RSS thresholds (29 ms / 243,875,840 bytes). | -| **R4: Manifest fold cost grows with commit count.** Current-state resolution folds history. | Medium | `optimize` compacts internal tables (keeps periodically-optimized graphs flat — cost-gated every PR). RFC-024 Gate A found flat exact-BTREE scan work at width 10, including observable/reconcilable uncovered tails, but rejected the format because representative RustFS 20→80 uncompacted cold reads/bytes (34→94 / 61,947→121,592) and compacted byte terms grow. RFC-024 is research-blocked; v6 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | +| **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v6 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6 preserves them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: No streaming ingest** — per-branch write throughput is capped by the `graph_head` CAS rate; high-frequency small writes are wasteful. | Medium | Deliberate: the interactive path's guarantees come first. MemWAL-based ingest with durable per-row ack + graph-atomic folds is the design **(RFC-026)**. MemWAL is the strategic substrate. RC.1 now carries the base dataset's store parameters and Session into derived MemWAL datasets, but its public initializer still commits opaquely and shard provisioning is separate; activation waits for a public exact enrollment receipt plus reversible admission seal rather than using private APIs. | +| **R7: No streaming ingest** — per-branch write throughput is capped by the `graph_head` CAS rate; high-frequency small writes are wasteful. | Medium | Deliberate: the interactive path's guarantees come first. MemWAL-based ingest with durable per-row ack + graph-atomic folds is the design **(RFC-026)**. MemWAL is the strategic substrate. RC.1 now carries the base dataset's store parameters and Session into derived MemWAL datasets, while its public initializer still commits opaquely and shard provisioning is separate. Gate E0 proved a bounded main-only, unsharded, single-live-writer-process path through exact successor classification plus exclusive-HEAD and local admission gates; Phase A remains unimplemented. A public exact enrollment receipt plus reversible admission seal gates only broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -959,10 +966,13 @@ Live design questions, each owned by an RFC or a known gap — not a wishlist: here: a deleted row is absent from the target snapshot, so version columns can't identify it, and `_row_last_updated_at_version` filtering is O(rows) without a substrate index or change log. -3. **What makes latest current-state authority physically history-flat?** +3. **What makes current-state authority physically history-flat?** RFC-024's first in-manifest BTREE candidate bounds exact row/range/page work, but cannot bound latest-manifest object reads and compacted byte ranges. A - successor access shape must pass the original cold/warm, + separate RFC-025 Gate 0 fixture reaches the same conclusion for checkpoint + list/show/cleanup authority after compaction, even though its uncompacted + structured work and bounded uncovered tail are flat. A successor access + shape must pass the original cold/warm, compacted/uncompacted local and object-store gate; the answer is not to weaken the gate or add a second authority dataset. 4. **Which capability owns the next rebuild boundary after v6?** RFC-028 @@ -971,10 +981,13 @@ Live design questions, each owned by an RFC or a known gap — not a wishlist: independently reviewable. Any later format activation requires its own export/init/load rebuild unless capabilities deliberately co-release after their combined matrix passes. -5. **What is the checkpoint/retention contract?** RFC-025: checkpoint rows as - logical authority, Lance tags as physical pins, and the asymmetric safe - ordering between them — plus how the Q8 resurrection watermark unlocks - internal-table cleanup. +5. **What is the checkpoint/retention contract and its bounded access shape?** + RFC-025 keeps checkpoint rows as logical authority, Lance tags as physical + pins, and the asymmetric safe ordering between them. Its tag substrate + evidence passes, but the first registry lookup candidate fails the physical + cost gate. Research now owns a history-flat current-authority lookup or a + revised evidence-backed operational contract before the Q8 resurrection + watermark can unlock internal-table cleanup. 6. **How does the background reconciler arrive without violating the recovery model?** It must serialize with writers through the same gates and stay roll-forward-only until the fence exists. @@ -995,7 +1008,7 @@ The plan of record is the RFC-022…028 family (all under | [0028 — Stable schema identity and table incarnation](../rfcs/0028-stable-schema-identity.md) | Graph-scoped rename-stable type/property IDs, table lifetimes, SchemaApply recovery, and the shared strict-rebuild prerequisite | **Implemented** (2026-07-15) | | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | -| [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags | Draft | +| [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | | [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: ack on WAL durability, asynchronous graph-atomic folds | Draft | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | diff --git a/docs/dev/index.md b/docs/dev/index.md index 932a2b61..5dd8859c 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -28,6 +28,7 @@ roadmap). The per-area docs below remain the mechanical authority. | System structure, L1/L2 framing, component diagrams | [architecture.md](architecture.md) | | On-disk layout, manifest schema, URI behavior | [storage.md](../user/concepts/storage.md) | | Direct-publish writes, D2, staged writes, recovery sidecars | [writes.md](writes.md) | +| Write-path state of affairs — shipped RFCs, current boundaries, blockers, upstream alignment, and next gates | [writing-path-state-of-affairs.md](writing-path-state-of-affairs.md) | | Query execution, mutation execution, loader flow | [execution.md](execution.md) | | Index lifecycle and graph topology indexes | [indexes.md](../user/search/indexes.md) | | Branch and commit internals | [branches-commits.md](../user/branching/index.md) | @@ -89,7 +90,7 @@ as durable disposition history after closure, so RFC backlinks stay valid. | Area | Read | |---|---| -| RFC-022–028 split architecture review — RFC-022 and RFC-028 implemented; RFC-023–027 blockers, dependency corrections, and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | +| RFC-022–028 split architecture review — RFC-022/023/028 implemented; RFC-024/025/027 research-blocked; RFC-026 draft; dependency corrections and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | ## Active Implementation Plans diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 6799418c..6673a8c0 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -182,7 +182,7 @@ converge the physical state. | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Keyed writes | Every v6 node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed/updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | Not implemented yet. RFC-026 adopts Lance MemWAL as the strategic substrate; activation waits on its stated public API and evidence gates. Once active, append acknowledgement means MemWAL-durable, while graph visibility still occurs only through graph-atomic RFC-022 folds | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | Not implemented. RFC-026 adopts Lance MemWAL as the strategic substrate and remains draft. Its production-neutral Gate E0 passed for an exact main-only, one-unsharded-shard, one-live-writer-process enrollment classifier without adding schema/API/format state: pinned immediate-successor probes reject buried effects with a history-flat six-attempt/zero-list shape, and strict local/RustFS negatives fail closed. The candidate separates stable enrollment identity from a mutable `CurrentHeadWitness` and would give an `OPEN` stream exclusive base-HEAD ownership; those are unimplemented support restrictions, not current guarantees. Green E0 authorizes only the next production foundation, never acknowledgement activation. Once streaming is eventually active, acknowledgement means MemWAL-durable while graph visibility still occurs only through graph-atomic RFC-022 folds | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -218,6 +218,27 @@ incomplete would let a legacy writer prepare from unresolved physical state. Do not hide these behind invariant wording. Either move them forward or keep them explicit. +- **Streaming enrollment is inactive; Gate E0 passed:** RC.1's public + initializer commits the singleton MemWAL index internally and shard claim is + a separate durable effect. RFC-026 no longer treats an upstream release as + the only calendar path: a production-neutral harness proved exact + no-effect / `N -> N + 1` index / pre-minted empty-shard classification, + mutable current-HEAD witness behavior, local/S3 ABA, and history-bounded + lookup under a main-only, unsharded, one-live-writer-process boundary. The + first `checkout_latest`/`IOTracker` attempt was rejected because local + `read_dir` escaped tracking. The accepted gate pins the public but + guide-hidden `has_successor_version` primitive from freshly ABA-verified exact + `N` and `N + 1` handles, rejects buried `N + 2`, records the same complete + six-attempt/zero-list shape at versions 8/80, and runs strict listing-dependent + negatives non-vacuously on RustFS. Exclusive HEAD and cleanup/version-GC + exclusion remain held through decision/publish; only `Ok(false)` means + absence, and errors, overflow, or detached boundaries fail closed. It may not + delete ambiguous artifacts. Until the later lifecycle, writer-exclusion, + recovery, format, and rebuild gates pass, internal schema + v6 remains authoritative and no `@stream`, enrollment, acknowledgement, or + public stream surface exists. The exact upstream receipt and cross-process + seal remain required to broaden topology. See + [RFC-026](../rfcs/0026-memwal-streaming-ingest.md). - **Storage abstraction:** `TableStorage` is present, sealed, and canonical for staged writes. MR-854 made `db.storage()` staged-only; the exact EnsureIndices adapter then migrated beta.21's full-table vector shape into the typed mixed diff --git a/docs/dev/lance.md b/docs/dev/lance.md index d641fc55..040f4772 100644 --- a/docs/dev/lance.md +++ b/docs/dev/lance.md @@ -177,29 +177,72 @@ Behavior-affecting findings in this audit: branch/tag/cleanup behavior, shared `Session` construction, merge-insert filter emission, the directional filtered/unfiltered conflict resolver, and full-table `CreateIndexBuilder::execute_uncommitted` retain the shapes - OmniGraph consumes. All 22 Lance surface guards pass on RC.1, including the - branch/ABA, blob-compaction, key-filter, shared-session, vector-staging, and - compiler-reservation cells. Search, writes, schema apply, branching, - maintenance, row-version, ordinary recovery, and all 129 runnable failpoint - tests also pass. A graph initialized and populated by the cached beta.21 CLI - was opened, queried, merge-written, and queried again by the RC.1 CLI, proving - the supported ordinary-schema V2_2 forward-open/write path rather than - relying only on fresh RC fixtures. This proof intentionally does not cover - the two newly reserved row-version names; their explicit upgrade path is - documented below. + OmniGraph consumes. All 23 Lance surface guards pass on RC.1, including the + exact tag/branch/ABA, blob-compaction, key-filter, shared-session, and vector + staging cells. Search, writes, schema apply, branching, maintenance, + row-version, ordinary recovery, and all 129 runnable failpoint tests also + pass. A graph initialized and populated by the cached beta.21 CLI was opened, + queried, merge-written, and queried again by the RC.1 CLI, proving the + supported ordinary-schema V2_2 forward-open/write path rather than relying + only on fresh RC fixtures. This proof intentionally does not cover the two + newly reserved row-version names; their explicit upgrade path is documented + below. - **Data Overlay is not an OmniGraph primitive.** RC.1 adds the experimental, opt-in `Operation::DataOverlay` and table feature flag 64. OmniGraph neither enables that flag nor emits the operation; its recovery classifiers keep unknown foreign effects on the fail-closed path. The explicit V2_2 pin does not silently opt a dataset into overlays. -- **MemWAL's direction improves, but RFC-026's activation blocker remains.** +- **MemWAL's direction improves; RFC-026 now owns a bounded decision gate.** Derived MemWAL datasets now inherit the base dataset's store parameters and `Session`, which is compatible with OmniGraph's shared-session design and remote credentials. The public initializer still commits internally and shard claiming remains a separate effect: there is still no caller-owned - exact enrollment receipt plus reversible shard-admission seal. MemWAL stays - the strategic substrate; RFC-026 stays draft rather than reaching through - private APIs. + combined enrollment receipt plus reversible cross-process shard-admission + seal. Source inspection confirms that `execute()` builds one + `Operation::CreateIndex` from the current manifest, commits it through + `CommitBuilder`, mutates the handle, and returns `Result<()>`; the public + builder persists arbitrary namespaced writer defaults, and the public writer + path later accepts a caller-selected shard UUID and claims an observable + epoch. Gate E0 uses a pre-minted enrollment/config-version default as + read-back evidence independent of the replaceable index UUID. The public + branch/version/current-transaction/e_tag tuple + is a mutable current-HEAD witness—ordinary commits change it—not a stable + enrollment incarnation. + + RC.1 persists writer defaults in the MemWAL index but does not apply them to + a caller-supplied `ShardWriterConfig`; any bounded adapter must reconstruct + the exact durable-write and buffer configuration rather than infer compatible + defaults. + + MemWAL stays the strategic substrate and RFC-026 stays draft. Its + production-neutral Gate E0 passed for the bounded profile. The first + history-cost result was rejected: local + `checkout_latest` may discover the tip through filesystem `read_dir`, which + bypasses `IOTracker`, so the observed tracked GET count omitted part of the + lookup. RC.1's public but guide-hidden `Dataset::has_successor_version` + provides the accepted replacement: from a freshly ABA-verified exact `N`, it + tests only `N + 1` through `CommitHandler::version_exists`; the exact `N + 1` + handle can then reject a buried `N + 2` without latest resolution or listing. + `AttemptTracker` records before forwarding—including failed/`NotFound` + HEADs—and observes the identical complete shape at baseline versions 8 and + 80: four successful manifest HEADs, one `NotFound` manifest HEAD, one + successful manifest GET, and zero lists. A Unix permissions tripwire proves + the exact probe works while latest enumeration fails and makes an unreadable + exact HEAD error. + + Fourteen substantive local cells cover exact initializer readback, + lost-result reopening, the pre-minted empty shard, buried-effect refusal, and + broad fail-closed classification. The configured RustFS exact cell passes + non-vacuously with the same six-attempt/zero-list shape and covers the + positive sequence plus foreign shard, malformed/loose root, durable WAL, + persisted cursor, and corrupt-manifest negatives. Surface guards pin + `has_successor_version`, flush/drain, merged-generation state, and S3 ABA; CI + rejects skipped E0/ABA cells. Exclusive HEAD and cleanup/version-GC exclusion + remain load-bearing; only `Ok(false)` means absence, while errors, overflow, + or detached boundaries fail closed. No private API, raw object emulation, + production schema, or stream acknowledgement is introduced. The exact + upstream receipt/seal remains the preferred simplification and the gate for + broader topology. - **Maintenance and index defaults preserve current behavior.** Compaction avoids a useless fragment-reuse-index optimization in a deferred-remap case; OmniGraph uses `defer_index_remap=false`, and Lance still exposes no stable @@ -221,12 +264,18 @@ Behavior-affecting findings in this audit: rebuilt before opening on RC.1. The audit minted a genuine beta.21 v6 graph with `_row_created_at_version`: beta.21 opened it successfully, while RC.1 refused it before any write with that exact recovery instruction. -- **RFC-024's research-blocked cost conclusion survives.** Its ignored - 10/100/1,000 instrument keeps exact heads and indexed row/range work flat, but - RC.1 adds a bounded one-operation boundary at the 1,000-commit endpoint on top - of the already-rejected byte curves. The local default and decision-scale - instruments pass with these current no-go facts asserted; S3/RustFS remains - bucket-gated and was not available for this audit. +- **The research-blocked cost conclusions survive, with RC-specific numbers.** + RFC-025's local 10→1,000 compacted reconciled list/cleanup scan bytes are now + 17,012→38,000 cold and 12,336→15,064 warm; exact show is 29,348→53,064 + and 24,672→30,128. Cold scan operations still cross 24→25 (show 34→35). + These are modest improvements over beta.21, not a change in asymptotic + disposition: the proposed in-manifest checkpoint registry remains rejected. + RFC-024's ignored 10/100/1,000 instrument likewise keeps exact heads and + indexed row/range work flat, but RC.1 adds a bounded one-operation boundary + at the 1,000-commit endpoint on top of the already-rejected byte curves. The + local default and decision-scale instruments pass with these current no-go + facts asserted; S3/RustFS remains bucket-gated and was not available for this + audit. ### Prior alignment audit: 2026-07-12 (Lance 9.0.0-beta.21 upstream; omnigraph pinned at 9.0.0-beta.21 via git rev) @@ -282,16 +331,19 @@ Behavior-affecting findings in this audit: maintenance-transaction API and OmniGraph has distributed recovery fencing; the latter is independently required before destructive recovery is safe against a live foreign process. -- **RFC-024 Gate A found a usable public ABA token but rejected the proposed - physical lookup shape.** The backend-portable candidate is the composite of - public `Dataset::branch_identifier()`, the current public - `Dataset::read_transaction()` UUID, and `Dataset::manifest_location().e_tag`. +- **RFC-024 Gate A found a usable current-HEAD witness with ABA sensitivity but + rejected the proposed physical lookup shape.** The backend-portable candidate + combines the public table version, `Dataset::branch_identifier()`, the + current public `Dataset::read_transaction()` UUID, and + `Dataset::manifest_location().e_tag`. Capture brackets transaction/e_tag collection with two branch-identifier reads and fails if the ref moves; a missing transaction or empty UUID also fails. Beta.21's local `current_manifest_path` synthesizes an inode-mtime-size e_tag, while S3/RustFS returns the object e_tag. Guards prove - stable unchanged reopens and distinguish main and named-ref delete/recreate - at the same numeric version on local and S3/RustFS. The local guard reuses the + stability across unchanged reopens and distinguish main and named-ref + delete/recreate at the same numeric version on local and S3/RustFS. An + ordinary commit changes the witness; it is not a stable dataset/enrollment + incarnation. The local guard reuses the original shared `Session`; the S3 guard also pins unchanged-incarnation reopen stability. Main's canonical empty `BranchIdentifier` makes the UUID and e_tag load-bearing; named refs additionally mint a new branch identifier. @@ -323,6 +375,40 @@ Behavior-affecting findings in this audit: `fragments_scanned`, `ranges_scanned`, and `rows_scanned` are beta.21 `all_counts` debug names explicitly subject to change, so every Lance bump must re-audit them rather than silently treating them as stable API. +- **RFC-025 Gate 0 validates Lance tags but rejects the current checkpoint- + registry access shape.** The pinned public tag surface targets an exact main + or named-branch version, creates/deletes auxiliary `_refs/tags/*.json` + metadata without advancing the dataset version, and exempts the tagged + version from cleanup. Deleting the tag makes the version reclaimable. A tag + on a named branch does **not** retain `tree/` after branch deletion, + so OmniGraph's proposed checkpoint-aware branch-delete refusal remains + load-bearing. `Tags::create` is an existence check followed by an + unconditional put on this revision, not a conditional create; RFC-025's + retention claim plus exact post-create verification therefore cannot be + replaced by the tag call itself. All 22 `lance_surface_guards` passed, + including `native_tags_pin_exact_main_and_named_branch_versions_through_cleanup` + and the extended `force_delete_branch_semantics` cell. + + The separate production-neutral `checkpoint_retention_cost.rs` fixture holds + three checkpoints and catalog width ten constant while unrelated journal + history grows. At the local 10→1,000 decision endpoints, reconciled + uncompacted list stays at 3 rows / 3 ranges / 1 fragment / 1 page and 24 scan + operations / 13,752 bytes; exact show stays at 12 / 2 / 2 / 3 and 34 / + 22,952; cleanup returns 44 rows with list-like cost. The eight-fragment tail + is exact and history-flat. Compaction rejects the candidate: list/cleanup + cold scan bytes grow 17,012→39,668 and warm bytes 12,336→16,736; exact-show + cold bytes grow 29,348→56,404 and warm bytes 24,672→33,472. At 1,000 commits + scan operations also cross 24→25 for the one-scan paths and 34→35 for show. + The default local 20/80 matrix passes its no-go-preservation assertions. The + bucket-gated S3/RustFS cost cell exists but was not run for this decision and + is not claimed. + + This result blocks the in-manifest BTREE access shape, not checkpoint rows as + logical authority or Lance tags as physical pins. RFC-025 is + research-blocked, no retention format ships, and internal schema v6 remains + production truth. A successor needs a history-flat current-authority lookup + or revised evidence-backed operational contract without adding a second + authority dataset. - **RFC-023 key-filter behavior remains route-dependent and directional, and v6 closes production routing around that fact with two distinct adapters.** A 2026-07-14 probe on this beta.21 pin shows that an explicitly selected v2 diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index 8acf17e3..b1156e3e 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -1,7 +1,7 @@ # Architecture review: RFC-022 through RFC-028 -**Status:** RFC-022, RFC-023, and RFC-028 implemented; RFC-024 and RFC-027 -research-blocked; RFC-025 and RFC-026 remain under review +**Status:** RFC-022, RFC-023, and RFC-028 implemented; RFC-024, RFC-025, and +RFC-027 research-blocked; RFC-026 remains under review **Date:** 2026-07-11 **Audience:** RFC authors, engine/storage maintainers, and release reviewers **Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at @@ -17,27 +17,41 @@ Stable V2_2, native branch/tag/cleanup semantics, exact full-table index staging, and RFC-023's route-dependent/directional key-filter behavior remain aligned. Data Overlay is opt-in and unused. MemWAL now inherits store/session configuration but still lacks RFC-026's exact enrollment receipt and reversible -admission seal. RC.1 therefore requires no RFC redesign or format activation. - +admission seal. RFC-026 therefore keeps the general multi-process profile +closed, while a production-neutral bounded Gate E0 tests the existing public +surface without activating a format. Its 2026-07-18 redesign is green: exact +`N`/`N + 1` immediate-successor probes replace latest resolution, complete +attempt tracking is flat at versions 8/80 with zero lists, and configured +RustFS runs the strict positive/negative matrix non-vacuously. +The local RFC-024/025 decision instruments were rerun: both remain +research-blocked, with the current RC-specific counters recorded in the Lance +audit and RFC-025. **Prior RFC-022 close-out revalidation:** Lance 9.0.0-beta.21 at `1aec14652dcbace23ac277fa8ced35000bea0c40`; see the [2026-07-12 alignment audit](lance.md#prior-alignment-audit-2026-07-12-lance-900-beta21-upstream-omnigraph-pinned-at-900-beta21-via-git-rev). -**Prior RFC-023 substrate evidence revalidated against:** the same beta.21 revision; +**RFC-023 substrate evidence revalidated against:** the same beta.21 revision; filter-shape and conflict-order probes are recorded in RFC-023 §2. Internal schema v6 now consumes that evidence through exact-`id` fenced production routing. Its final insertion-absence certificate/no-target-preflight route and predeclared 10K/100K production series now satisfy the remaining implementation and acceptance gates. -**Prior RFC-026 substrate contract revalidated against:** the same beta.21 revision; +**RFC-026 substrate contract revalidated against:** RC.1 at the current revision; the full MemWAL table and system-index specifications, including the durability/fencing behavior carried forward from beta.17, are reflected in the -RFC's survey and acceptance guards. -**Prior RFC-024/025/027 substrate claims revalidated against:** the same beta.21 +RFC's survey and acceptance guards. Gate E0 passed at the explicit bounded +profile and evidence split above; RFC-026 remains draft and production inactive. +**RFC-024/025/027 substrate claims revalidated against:** the current RC.1 revision and the full matching table/index/branch/tag/cleanup, transaction, and -row-lineage specification sections; their survey headers now record beta.21. +row-lineage specification sections; their survey headers now record RC.1. RFC-024 Gate A was subsequently measured on 2026-07-15: its exact BTREE scan work is flat, but the required RustFS latest-manifest/object-byte curves grow, so the candidate and format are research-blocked rather than accepted. +RFC-025 Gate 0 was measured on 2026-07-17: Lance tag semantics pass, but the +current in-manifest checkpoint-registry BTREE shape has history-sensitive +compacted scan bytes and crosses another scan-operation boundary at 1,000 +commits. RFC-025 is therefore also research-blocked and production remains on +internal schema v6. The bucket-gated S3/RustFS cost cell is checked in but was +not run for that decision. This is a review ledger, not a competing specification. The RFCs remain the normative design records: RFC-022/RFC-028 document implemented contracts; @@ -393,20 +407,22 @@ identity, at least: ```text (state, stable_table_id, incarnation_id, table_path, - table_branch, physical_ref_incarnation, table_version, schema_ir_hash) + table_branch, current_head_witness, table_version, schema_ir_hash) ``` The publisher may encode that token differently, but retry/revalidation must -not reduce it to `table_version`. `physical_ref_incarnation` is the public -`BranchIdentifier` + current `Transaction.uuid` + `ManifestLocation.e_tag` -composite. Capture brackets transaction/e_tag collection with branch-identifier -reads and rejects movement; missing/empty transaction identity fails closed. -This is distinct from the logical `incarnation_id`, which RFC-028 deliberately -preserves across an owner handoff. Add stale-writer races for both logical -drop/recreate and physical ref recreation at the same numeric version. - -**Disposition:** RFC-024 §3.2 now persists the separate physical-ref token and -refuses activation on a backend without a proven source. Beta.21 local manifest +not reduce it to `table_version`. `current_head_witness` is the public +`BranchIdentifier` + current table version + current `Transaction.uuid` + +`ManifestLocation.e_tag` composite. Capture brackets transaction/e_tag +collection with branch-identifier reads and rejects movement; missing/empty +transaction identity fails closed. It is intentionally mutable: every ordinary +table commit advances it. This is distinct from the logical `incarnation_id`, +which RFC-028 deliberately preserves across an owner handoff. Add stale-writer +races for both logical drop/recreate and physical ref recreation at the same +numeric version. + +**Disposition:** RFC-024 §3.2 now persists the separate current-HEAD witness and +refuses activation on a backend without a proven source. Pinned-Lance local manifest resolution supplies an inode-mtime-size e_tag; S3/RustFS supplies the object e_tag. Public-surface guards pass main and named-ref same-version ABA on both, unchanged reopen stability on S3, and original-shared-`Session` local @@ -426,10 +442,15 @@ integration. [§9](../rfcs/0026-memwal-streaming-ingest.md#9-fresh-read-cuts), and [§11–13](../rfcs/0026-memwal-streaming-ingest.md#11-format-activation-and-rebuild) -**Status:** Open. The format/refusal portion is specified, but RC.1 lacks the -public exact enrollment and reversible shard-admission surface needed by the -recovery/quiescence adapter. RFC-026 cannot be accepted or activated until that -API gate and its crash evidence close. +**Status:** Open; bounded Gate E0 passed; Phase A pending. RC.1 lacks the ideal +caller-owned enrollment receipt and cross-process shard-admission seal, but +upstream release timing is no longer the only decision path. RFC-026 now +specifies a production-neutral Gate E0 that may admit a main-only, unsharded, +single-live-writer-process profile if every exact-classification and ABA cell +passes. Those cells now pass using exact known-version probes and strict +object-store classification. RFC-026 remains draft and production inactive +regardless of E0 until its later format, recovery, writer-exclusion, lifecycle, +and crash gates close. Enrollment creates persistent MemWAL metadata and `stream_state` changes the correctness preconditions for schema, branch, maintenance, and data operations. @@ -459,18 +480,48 @@ logical pair, name, path, or a compatible-looking index. `InitializeMemWalBuilder::execute` internally commits the `CreateIndex` transaction and returns only `Result<()>`; `mem_wal_writer` separately creates or claims shard-manifest objects. That cannot satisfy a sidecar that pre-arms -one exact combined effect. RFC-026 §3 now rejects private-module/object-store -emulation and requires a public caller-controlled transaction plus idempotent -shard provision/classify/seal/reopen/reclaim APIs, or one recoverable enrollment -receipt covering both effects. §5/§8 require the physical admission seal that -closes the final lifecycle-check-to-put race. The same-path/ref ABA, multi- -effect crash, and paused-claimant tests remain red acceptance gates. - -Replica scope must also match RFC-023's recovery support boundary. Multiple -replicas may route acknowledgement traffic to one shard owner, but enrollment, -fold, and sidecar recovery remain single-writer-process operations until -foreign-process sidecar ownership is fenced. Do not advertise general replica -failover for those operations merely because MemWAL has a shard epoch. +one caller-minted combined effect, so the public receipt plus seal/reopen API +remains the preferred simplification and the gate for overlapping processes. +RFC-026 still rejects private-module/object-store emulation. + +The bounded candidate instead makes the support restriction load-bearing. It +grants an `OPEN` stream exclusive authority to advance that base-table HEAD; +every other writer/maintenance/control path touching the table must refuse +pre-effect or drain. Stable enrollment identity is kept separate from the +mutable `CurrentHeadWitness` (`BranchIdentifier`, current table version, +transaction UUID, manifest e_tag), which advances atomically with every +allowed table-pointer publish. A root-scoped admission lease closes the final +check-to-put race only inside the one live writer process. + +Gate E0 must classify exact no effect, the sole `N -> N + 1` singleton MemWAL +`CreateIndex` carrying the namespaced enrollment/config-version marker, and +that successor plus the pre-minted empty shard; wrong config, intervening HEAD, +foreign shard, data-bearing WAL/generation state, or ambiguity must yield +`RecoveryRequired` without deletion. It also proves unchanged-reopen +witness stability, ordinary-commit movement, local/S3 same-coordinate ABA, and +history-depth cost. The first cost attempt observed a flat tracked GET count but +missed local filesystem `read_dir` work outside `IOTracker` and was discarded. +The accepted replacement freshly ABA-verifies exact `N`, uses the public but +guide-hidden `Dataset::has_successor_version` to probe only `N + 1`, then uses +the exact `N + 1` handle to reject a buried `N + 2`. Exclusive HEAD and +cleanup/version-GC exclusion remain held through decision/publish; only +`Ok(false)` means absence, while errors, overflow, and detached boundaries fail +closed. + +The final local run has 14 substantive cells plus one explicit S3 skip. Its +attempt-counting wrapper includes failed/`NotFound` HEADs and records the same +six-attempt, zero-list shape at baseline versions 8 and 80; a Unix permissions +tripwire proves exact probes succeed when latest enumeration fails and that an +unreadable exact HEAD errors. The configured RustFS cell passes non-vacuously +with the same shape plus the declared positive and listing-dependent negative +matrix. Surface guards pin the doc-hidden successor, flush/drain, +merged-generation, and S3 ABA shapes; CI rejects skipped E0/ABA cells. This is +green evidence for the bounded profile, not production activation. + +Replica scope must match RFC-023's recovery support boundary. The bounded +profile does not permit overlapping replicas: a crash successor proceeds only +after external exclusivity, recovery, and a higher shard epoch. Do not +advertise general replica failover merely because MemWAL has a shard epoch. ### BLOCKER-07 — stable identity ownership contradicts sibling dependencies @@ -509,9 +560,12 @@ adopted by implication, and drop/re-add cannot claim a predecessor's rows. The same correction keeps logical identity separate from physical ABA proof. RFC-025 now owns a heads-independent exact manifest/table version token, backend refusal, recovery/reconciler checks, and local/S3 same-coordinate ABA -tests. RFC-026 owns a binding-stable physical-ref incarnation token and remains -blocked under BLOCKER-06 until its backend/API guards pass. Neither silently -imports RFC-024's head storage merely to obtain those proofs. +tests. RFC-026 owns a stable enrollment identity plus a separately mutable +`CurrentHeadWitness`; it does not pretend one token can be both stable across +fold commits and sensitive to current-HEAD movement. It remains draft under +BLOCKER-06 despite green Gate E0 because Phase A and later production gates are +still open. Neither silently imports RFC-024's head storage merely to obtain +those proofs. > 💬 **Concur; supersedes a 2026-07-11 correction (2026-07-11):** the earlier > pass placed identity ownership *inside* RFC-024 §3.1, which created exactly @@ -564,8 +618,8 @@ metadata selects its lifecycle: RFC-022 through RFC-028 now identify the maintainer design-series track and an owner. RFC-022 and RFC-028 are `implemented` at their documented support -boundaries, RFC-023 is `implemented`, RFC-024 and RFC-027 are -`research-blocked`, and RFC-025/RFC-026 remain `draft`. +boundaries, RFC-023 is `implemented`, RFC-024, RFC-025, and RFC-027 are +`research-blocked`, and RFC-026 remains `draft`. All formal RFC filenames in the central directory are normalized to four digits. Legacy `docs/dev/rfc-00N-*` files remain in place with their existing lifecycle; they are internal design and implementation records, not members of @@ -741,6 +795,53 @@ acceptance and a combined initialization/recovery matrix. The rebuild's loss of unselected branches and history is explicit rather than hidden behind the word “migration.” +### BLOCKER-13 — checkpoint authority lookup is not physically history-flat + +**Affected:** [RFC-025 §0.1](../rfcs/0025-checkpoint-retention.md#01-pre-activation-gate-0), +[§9](../rfcs/0025-checkpoint-retention.md#9-observability-and-bounds), and +[§11](../rfcs/0025-checkpoint-retention.md#11-phasing) + +**Status:** Research-blocking no-go recorded on 2026-07-17. Production remains +on internal schema v6; no retention format or checkpoint API is activated. + +The production-neutral `checkpoint_retention_cost.rs` fixture holds three live +checkpoints and catalog width ten fixed while unrelated journal history grows. +It measures complete list, exact show, and cleanup-root reads from cold open and +warm repeat across compacted/uncompacted layouts and absent, reconciled, and +eight-fragment uncovered-tail index states. + +The local 10→100→1,000 decision run separates a real access-shape win from the +required history-slope claim: + +- reconciled uncompacted list is flat at 3 rows / 3 ranges / 1 fragment / 1 + index page and 24 scan operations / 13,752 bytes; exact show is flat at + 12 / 2 / 2 / 3 and 34 operations / 22,952 bytes; cleanup returns 44 rows with + list-like cost, and the bounded tail remains exact and history-flat; +- after compaction, list/cleanup cold scan bytes grow 17,012→38,000 and warm + bytes 12,336→15,064; exact-show cold bytes grow 29,348→53,064 and warm bytes + 24,672→30,128. At depth 1,000 the one-scan paths cross 24→25 scan operations + and show crosses 34→35. + +The default 20/80 test passes its no-go-preservation assertions. That is a +successful instrument reproducing a failed design gate, not acceptance. The +S3/RustFS cost cell exists and remains bucket-gated; it was skipped locally and +is not credited as evidence. One required local compacted cell already fails, +so the missing S3 run cannot turn this result green. + +The substrate itself is not the blocker. All 23 RC.1 Lance surface guards pass; the +RFC-025 cells prove exact main/named-branch tag targets, sparse cleanup pin and +unpin, and that tags do not protect a named branch tree. This supports the +checkpoint-row/Lance-tag architecture and keeps the branch-deletion guard +load-bearing. It does not make the surviving registry authority read +history-flat. + +**Required disposition:** retain `__manifest` as the only checkpoint authority +and find a history-flat current-authority lookup shape, or revise the +operational contract with measured evidence and explicit user-visible cost. +Do not weaken the physical-I/O gate, infer success from flat row/range counters, +or add a second authority dataset whose rows cannot share the main-manifest +CAS. + ### TIGHTENING-01 — symmetric Lance conflicts are not obviously an activation gate **Status:** Closed in implementation and acceptance on 2026-07-15. The beta.21 @@ -793,10 +894,13 @@ protected by symmetry in either case. These are smaller than the blockers above, but should be resolved before the affected RFC is accepted or implemented: -1. **Checkpoint-name normalization:** define allowed bytes, Unicode - normalization, case sensitivity, maximum encoded length, reserved prefixes, - and normalization-version compatibility. Test collisions and reuse across - versions. +1. **Checkpoint-name normalization (closed in specification and reference test + on 2026-07-17):** RFC-025 defines V1 as an ASCII-only, case-sensitive, + identity transform over 1..=128 bytes with slash-separated nonempty + `[A-Za-z0-9._-]+` segments, explicit path-like refusals, reserved `__` and + `ogcp_` prefixes, a versioned reservation key, and no trim, Unicode + normalization, or case folding. Future versions must use bounded exact + cross-version collision checks and may not reinterpret V1 reservations. 2. **RFC lifecycle values (closed 2026-07-13):** [`docs/rfcs/README.md`](../rfcs/README.md) now uses one four-digit filename namespace and distinguishes the public contribution lifecycle from the @@ -906,11 +1010,16 @@ The review does not require all RFCs to land together. A safe order is: the exact BTREE's scan work is flat, but uncompacted RustFS cold object reads/bytes and compacted byte terms grow, so the format is research-blocked and production remains on internal schema v6; -5. accept RFC-025 after physical protection of both checkpoints and live graph - branches plus strict-rebuild implementation evidence; -6. accept RFC-026 only after Lance exposes the public exact-enrollment and - admission-seal surface, then its capability/refusal/rebuild, multi-effect - recovery, and writer-ownership evidence match the specified scope; +5. keep RFC-025 research-blocked after its 2026-07-17 Gate 0 no-go; reconsider + only after a history-flat current-authority lookup shape or revised + evidence-backed operational contract passes the full physical-I/O boundary, + then require physical protection of checkpoints/live graph branches and + strict-rebuild implementation evidence; +6. retain RFC-026 Gate E0's green exact-version evidence without production + activation, then implement and verify only Phase A's bounded + main/unsharded/single-live-writer enrollment recovery, writer exclusion, + lifecycle, and admission-lease foundation; retain the public exact-enrollment + receipt and cross-process admission seal as the gate for broader topology; 7. keep RFC-027 in research until deletion-delta discovery passes its stated correctness and flat-cost gates. diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 43f3b2b8..883d07dd 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -25,8 +25,10 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); RFC-023 pins the inclusive 8,192-row keyed input ceiling, the same exact/+1 boundary on streamed mutation-update matches, no-effect state for both refusals, and oversized stored-Blob rejection before payload read. Crate-internal pending-scan cells pin inclusive/+1 32 MiB accounting plus pending-key shadow-before-charge. The lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal staged primitives. RFC-023 pins one exact target preflight for general StrictInsert, durable v1 mint/commit/reopen/history persistence, exact-`id` filter emission, typed `KeyConflict`, and missing/wrong PK refusal. `all_new_upsert_certifies_insert_absence_and_persists_it_in_history` proves an all-new completed Upsert receives the optional certificate, a mixed/update Upsert does not, unrelated transaction properties survive, and UUID rebinding does not erase it. Proven-insert cells show the opaque path performs zero strict preflights; stages with `InsertBuilder` but commits the full pure-insert `Update` shape (exact parent and `id` filter, `RewriteRows`, no updates/removals, full nested schema preorder, physical rows); persists/re-admits its own output for proof composition; leaves new fragments outside old index coverage; and fails same-key races loudly in proven/proven and proven/general orders. The in-source `exec/merge.rs` certificate unit table rejects missing/unknown properties, wrong parent/filter/full-preorder/mode/offsets, rewrite/removal shapes, missing `physical_rows`, and Append. Source-interval cells pin exact selection, lazy retained-parent splitting, coalescing, and pinned Lance's approximate raw-emission boundary while every normalized/writer chunk remains hard-capped. Generic `stage_append`/`stage_merge_insert` remain primitive tests only. The file also owns index staging and `commit_staged{,_exact}` | | `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. This is intentionally not a Rust macro-expander or general alias analysis; `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | -| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current `Transaction.uuid` + `ManifestLocation.e_tag` physical-ref candidate; the local, shared-`Session`, and RustFS tests require unchanged-reopen stability and distinguish main/named-ref delete/recreate ABA at the same numeric version. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection; every Lance bump still audits upstream source for additions. All 22 guards pass on RC.1. These prove substrate tokens, not an accepted heads format or publisher. They remain substrate-upgrade tripwires; v6 production route, recovery, format, and cost evidence live at their own boundaries | +| `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These prove substrate shapes/tokens, not an accepted heads/checkpoint/stream format or production publisher | +| `memwal_enrollment_gate.rs` | RFC-026's green production-neutral Gate E0 harness, isolated from schema v6 and the graph writer. Fourteen substantive local cells plus one explicit unconfigured-S3 skip cover exact no-effect / `N + 1` index / pre-minted empty-shard classification, buried-effect refusal, marker survival, strict inventory/error handling, and the broad fail-closed matrix. The rejected first instrument used `checkout_latest` plus `IOTracker`, which missed local `read_dir`. The accepted exact-version classifier pins doc-hidden `has_successor_version`; its `AttemptTracker` records failed/`NotFound` attempts before forwarding and proves the identical complete six-attempt shape at baseline versions 8/80: four successful manifest HEADs, one `NotFound` manifest HEAD, one successful manifest GET, zero lists. A Unix execute-only `_versions` tripwire proves exact probing works when latest enumeration fails and an unreadable exact HEAD errors. The configured RustFS exact cell passes non-vacuously with the same zero-list shape and owns the positive lost-result/index/empty-shard/reopen sequence plus foreign shard, malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest negatives. S3 ABA remains in `lance_surface_guards.rs`; CI rejects skipped E0/ABA cells. This file never mutates production manifest/schema state or deletes ambiguous artifacts; green E0 authorizes Phase A only | | `durable_head_lookup_cost.rs` | RFC-024 Gate A decision instrument, isolated from the production manifest schema/publisher. At fixed catalog width 10 it runs the full absent/reconciled/one-uncovered/eight-uncovered/reconciled-after-tail matrix over compacted and uncompacted histories, with cold-open and warm-repeat measurements on local FS and bucket-gated S3/RustFS. Default depths are 20/80; the ignored decision-scale cell runs 10/100/1,000. Correct exact heads, flat indexed `rows_scanned`/range work, an index-absent growing negative control, and observable bounded tails all pass; after the eight-fragment tail, `optimize_indices` returns coverage to zero uncovered and representative `rows_scanned`/range work from 27→10 / 17→10. The test deliberately pins the no-go: uncompacted RustFS cold object reads/bytes and compacted byte terms grow, while RC.1 also crosses a bounded one-operation boundary by 1,000 commits, so RFC-024 remains research-blocked. `rows_scanned` is an RC.1 debug proxy, not a universal decoded-row counter. Object-store wrapper bytes and Lance execution-summary bytes are separate fixture-owned metrics and are not additive | +| `checkpoint_retention_cost.rs` | RFC-025 Gate 0 decision instrument, isolated from production schema v6. It models three live checkpoints at catalog width 10 and measures complete list, exact show, and cleanup-root authority reads across absent/reconciled/eight-uncovered index states, compacted/uncompacted layouts, and cold/warm access. It also owns the reference V1 name-normalization matrix. Default local depths 20/80 pass the checked-in **no-go-preservation** assertions; the RC.1 ignored 10/100/1,000 run shows reconciled uncompacted work and the bounded tail flat, but rejects the current format shape after compaction: list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; show grows 29,348→53,064 and 24,672→30,128; scan operations add one at 1,000. The S3/RustFS cell is bucket-gated and was not run for this decision. The result keeps RFC-025 research-blocked and production on v6 | | `warm_read_cost.rs` | Cost-budget tests for the warm read/control path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); a cold other-branch resolution derives snapshot state and lineage from one coherent manifest open/scan; native branch create and create-from each use one post-gate open/scan, while delete uses one target capture plus one native-ref opener and only one row scan; stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. | | `write_cost.rs` | Cost-budget tests for the WRITE path (RFC-013), the latency twin of `warm_read_cost.rs` on the **shared `helpers::cost` harness** (`measure`/`IoCounts`/`assert_flat`/`local_graph`). Runs on **local FS**; gates the **internal-table** term (`__manifest` scans flat in commit-history depth, lineage rows included — `internal_table_scans_are_flat_in_history`, now **green every-PR** since RFC-013 step 2 brought the internal tables into `optimize`; the test compacts at each depth before measuring), graph-visible maintenance arbitration (`ensure_indices_manifest_reads_are_flat_in_history` and `optimize_manifest_reads_are_flat_in_history`), plus green every-PR guards (single-insert `data_writes` bounded, a per-write read-op ceiling that fails the moment a round-trip is added, and a `measure_with_staged` fitness assert that a keyed insert routes through the exact-`id` fenced adapter once with no bare `stage_append`/vector-index build). Also gates the batched committed `@unique` probe: `unique_probe_io_is_flat_in_delta_rows` sweeps DELTA size (4 vs 64 rows) at fixed shallow history and asserts `data_open_count`/`data_scan_reads` flat — red when the cross-version probe regresses to per-row scans/opens. The **data-table opener** term is S3-only — see `write_cost_s3.rs` and the backend-split note in "Cost-budget tests" below. RFC-023's representative row-count and peak-RSS decision measurements use the scenario harness, not this every-PR I/O budget | | `write_cost_s3.rs` | Bucket-gated (skips without `OMNIGRAPH_S3_TEST_BUCKET`) twin of `write_cost.rs` on the same `helpers::cost` harness: gates the **data-table opener** term (per-write latest-version resolution flat across commit depth on a real object store — per-version GETs are invisible on local FS). A cost gate, not a correctness test — run on demand, not in the every-merge `rustfs_integration` job (see the backend-split note in "Cost-budget tests" below) | @@ -107,7 +109,7 @@ sidecar arm or graph movement. CI runs these S3-backed **correctness** tests against a containerized RustFS server (`.github/workflows/ci.yml` → `rustfs_integration` job, sharded one suite per runner): - `cargo test -p omnigraph-engine --test s3_storage` (lifecycle/branching + the e_tag-present CSR topology cache-key reuse test — the path local FS can't reach since its e_tag is `None`) -- `cargo test -p omnigraph-engine --test lance_surface_guards public_physical_ref_token_rejects_s3_same_version_aba -- --exact` (RFC-024's public physical-ref token across unchanged reopen plus main/named same-version ABA; the workflow additionally rejects a zero-test/vacuous match) +- `cargo test -p omnigraph-engine --test lance_surface_guards public_physical_ref_token_rejects_s3_same_version_aba -- --exact` (RFC-024's public current-HEAD witness across unchanged reopen plus main/named same-version ABA; the workflow additionally rejects a zero-test/vacuous match) - `cargo test -p omnigraph-server --test s3` (single-graph serving + config-free `--cluster s3://` boot) - `cargo test -p omnigraph-cluster --test s3_cluster` (full control-plane lifecycle on the bucket) - `cargo test -p omnigraph-cli --test system_local local_cli_s3_end_to_end_init_load_read_flow` @@ -119,6 +121,19 @@ RFC-024's S3 **cost** matrix is deliberately not in this correctness job. Run it on demand with `OMNIGRAPH_S3_TEST_BUCKET=… cargo test -p omnigraph-engine --test durable_head_lookup_cost s3_durable_head_lookup_matrix_is_correct_and_observable -- --exact --nocapture`. +RFC-025's S3 **cost** matrix is likewise on demand and was not run for the +2026-07-17 local no-go decision: +`OMNIGRAPH_S3_TEST_BUCKET=… cargo test -p omnigraph-engine --test checkpoint_retention_cost s3_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go -- --exact --nocapture`. + +RFC-026 Gate E0's configured RustFS cell is both classifier and complete +exact-probe evidence. It owns the positive lost-result/index/empty-shard/reopen +sequence, listing-dependent foreign/malformed/loose/data/cursor/corrupt +negatives, and the six-attempt zero-list shape. The RustFS CI job rejects a +`SKIP` or zero-test match; run it explicitly with +`OMNIGRAPH_S3_TEST_BUCKET=… cargo test -p omnigraph-engine --test memwal_enrollment_gate s3_memwal_enrollment_gate_positive_and_listing_negatives -- --exact --nocapture`. +Outside that configured job the test skips explicitly when the bucket is +absent. CI separately rejects a skipped S3 ABA surface guard. + ## Cross-version upgrade (genuine old binaries → v6) `crates/omnigraph-cli/tests/crossversion_upgrade.rs` contains genuine-binary @@ -324,6 +339,8 @@ Correctness bugs fail loudly in tests; cost-scaling bugs pass every test and deg realistic history depth, describe the result as reduced amplification, not history-flat authority lookup. - **Keep decision instruments honest when the answer is no.** RFC-024's `durable_head_lookup_cost.rs` attaches tracking before the cold dataset load through `open_tracked_lance_dataset`, then reports object-store wrapper I/O separately from Lance execution-summary I/O. Its reconciled BTREE row/range curve is flat, but its required RustFS cold-open and compacted-byte curves grow; those red design facts are asserted as the current result rather than erased because some counters pass. Run the default local 20/80 matrix with `cargo test -p omnigraph-engine --test durable_head_lookup_cost local_durable_head_lookup_matrix_is_correct_and_observable -- --exact --nocapture`; run the ignored 10/100/1,000 local matrix with `cargo test -p omnigraph-engine --test durable_head_lookup_cost local_durable_head_lookup_matrix_at_one_thousand_commits -- --ignored --exact --nocapture`. The bucket-gated S3 command is in the RustFS section above and remains on demand. +- **Apply the same rule to RFC-025.** `checkpoint_retention_cost.rs` keeps live checkpoint count and catalog width fixed while unrelated journal history grows, and counts complete list/show/cleanup-root authority reads. The uncompacted reconciled counters and bounded tail are flat; compacted scan bytes and the 1,000-commit operation boundary are not, so the assertions preserve a no-go. Run the default local matrix with `cargo test -p omnigraph-engine --test checkpoint_retention_cost local_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go -- --exact --nocapture`; run the ignored decision scale with `cargo test -p omnigraph-engine --test checkpoint_retention_cost local_checkpoint_retention_matrix_at_one_thousand_commits -- --ignored --exact --nocapture`. A green test means the known result was reproduced, not that RFC-025 passed Gate 0. +- **Keep RFC-026 Gate E0 reproducible.** The first `checkout_latest`/`IOTracker` instrument was false-green because local `read_dir` escaped tracking; it is not acceptance evidence. The green harness uses the public but guide-hidden `Dataset::has_successor_version` from freshly ABA-verified exact `N`, probes only `N + 1`, then uses exact `N + 1` to reject buried `N + 2`. `AttemptTracker` records before forwarding, including failed/`NotFound` HEADs, and versions 8/80 must retain the identical four-success-HEAD + one-NotFound-HEAD + one-success-GET shape with zero lists. The Unix execute-only `_versions` tripwire must keep exact probing green while latest enumeration fails, and an unreadable exact HEAD must error. Run the 14-substantive-cell local file with `cargo test -p omnigraph-engine --test memwal_enrollment_gate -- --nocapture`; its fifteenth bucket-gated cell logs an explicit skip when unconfigured. Run the exact configured RustFS command above for its positive plus listing-dependent negative matrix. Green E0 authorizes only Phase A, never schema/API/format or acknowledgement activation. - **Count on the handle that does the reads, not just the one a measured op opens.** Lance's IO-counted tests attach the `IOTracker` to the (warm, cached) dataset and read `incremental_stats()` per request — the tracker MUST be on the handle performing the reads, or warm-handle reads escape. A per-op tracker installed at measure time cannot see reads on a long-lived handle opened earlier (the warm coordinator's `__manifest` handle, reused across writes), so such reads were silently undercounted. Wrap a depth-swept body in `cost_harness` so the manifest tracker is installed before the graph opens and `manifest_reads` is **ground truth** (handle-age-irrelevant). The `version_probes` counter is the freshness-probe *call* count; ground truth additionally reveals that a write's probe does ~3 object-store RPCs (a read's probe is a 0-IO cache hit). `manifest_reads_capture_warm_probe` is the guard that this stays true. - This is the testing companion to invariant 15 in [docs/dev/invariants.md](invariants.md) (hot-path cost is bounded by work, not history). diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md new file mode 100644 index 00000000..d229a467 --- /dev/null +++ b/docs/dev/writing-path-state-of-affairs.md @@ -0,0 +1,346 @@ +# Write Path: State of Affairs + +**Type:** living architecture and execution summary +**Status:** current as of 2026-07-18 +**Surveyed:** OmniGraph 0.8.1 development, internal manifest schema v6, +Lance 9.0.0-rc.1 at `cec0b7df` +**Scope:** the direct-publish graph write path, its RFC-022–028 family, +adjacent control and maintenance operations, known blockers, and the next +decision points + +**Change-set boundary:** this page describes the target state formed by current +main plus this follow-up. Merged PR #364 (`2864681`) moved the substrate to +Lance RC.1 but intentionally left RFC-025 Draft and excluded Gate 0. This +separate Gate-0 follow-up records the production-neutral no-go and moves RFC-025 +to Research-blocked. Neither state activates checkpoints; internal schema v6 +remains production truth. This revision also records RFC-026 Gate E0 as a +green production-neutral bounded-enrollment decision after replacing its +invalid untracked latest-tip instrument with exact-version attempt tracking. +The gate does not activate `@stream`, lifecycle rows, MemWAL enrollment, public +APIs, or durable acknowledgements. + +This page answers four practical questions: + +1. What write-path architecture is actually implemented? +2. Which RFC decisions shipped, and which remain inactive? +3. What did implementation and measurement teach us? +4. What should we do next—and what should we deliberately not build yet? + +It is an orientation document, not a second specification. Use +[invariants.md](invariants.md) for hard architectural rules and support +boundaries; the individual [RFCs](../rfcs/README.md) and +[review ledger](rfc-022-027-architecture-review.md) for decisions and acceptance +gates; [writes.md](writes.md) for current mechanics; [canon.md](canon.md) for +the narrative mental model; [lance.md](lance.md) for the pinned upstream +contract; and [testing.md](testing.md) for evidence ownership and commands. + +## Executive judgment + +The architecture is on the right track. The original write-path problem did +not require a new database inside OmniGraph; it required a disciplined +coordination layer over Lance: + +- Lance owns table transactions, versions, branches, fragments, indexes, + compaction, cleanup, and MemWAL. +- OmniGraph owns graph-wide authority, cross-table visibility, schema identity, + validation, and recovery across independently committed Lance datasets. +- One `__manifest` commit is the only graph-content visibility point. +- Durable recovery ownership covers the gap between a Lance table effect and + that manifest publication. +- Immutable, version-pinned state may be cached. Mutable authority is read or + arbitrated durably; it is never supplied to a commit by an in-memory + `GraphState`-like shadow. + +The central chassis is implemented: [RFC-022](../rfcs/0022-unified-write-path.md), +[RFC-023](../rfcs/0023-key-conflict-fencing.md), and +[RFC-028](../rfcs/0028-stable-schema-identity.md) are complete at the documented +support boundary. The remaining RFCs are not a backlog to implement in numeric +order. RFC-024 and RFC-027 were already research-blocked; this follow-up records +RFC-025's Gate-0 blocker. RFC-026 remains strategic and draft, but is no longer +passively calendar-blocked: its production-neutral Gate E0 confirms that RC.1's +public state supports a deliberately bounded main-only, unsharded, +single-live-writer-process enrollment classifier. Its final evidence is green: +the replacement proof uses Lance's public `has_successor_version` only from +freshly verified exact `N` / `N + 1` handles, under uninterrupted HEAD and +cleanup/version-GC exclusion, with complete attempt and list tracking. No +streaming format or path is active; Phase A is now the next authorized +production foundation slice. + +The largest remaining correctness boundary is topology: destructive recovery +is serialized across every handle in one process, but not against a live writer +in another process. OmniGraph must not advertise general multi-process writers +on one graph until a distributed fence closes that gap. + +## The implemented correctness protocol + +There is one correctness protocol with writer-specific physical adapters. It +does **not** mean every writer is forced through one identical Lance operation. + +```text +resolve or refuse relevant recovery + -> capture one immutable authority token and accepted catalog + -> prepare and validate the complete operation + -> acquire schema -> branch -> sorted-table gates + -> relist recovery and revalidate authority plus physical baselines + -> durably arm an identity-bearing recovery intent + -> apply writer-specific Lance effects + -> exact adapters durably confirm the achieved outcome; + Optimize retains a bounded complete-set classifier + -> publish one graph-visible __manifest transition when needed + -> delete the intent; a recovery resolution records an audit first +``` + +For the exact adapters, the authority token binds the facts that made planning +valid: the native Lance branch incarnation, exact optional graph head, accepted +schema identity, and relevant table versions/physical refs. A publisher may +retry a transient manifest CAS against that captured precondition. A semantic +full reprepare starts a new attempt instead of refreshing versions under an old +validation result; Optimize likewise reopens and replans a maintenance retry. + +The five active graph-visible writer classes emit schema-v9 recovery envelopes +before their first independently durable effects. Table ownership is keyed by +`(stable_table_id, table_incarnation_id)`, never inferred from an alias, path, +Lance version, or field ID. Mutation, Load, BranchMerge, SchemaApply, and +EnsureIndices use exact protocols in which `Armed` is rollback-only and +`EffectsConfirmed` may roll forward only under the captured authority. Optimize +is the deliberate exception: its v9 envelope carries identity-bound pins and a +bounded complete-set maintenance classifier, with no exact authority/fixed- +lineage confirmation phase, because Lance exposes no caller-minted maintenance +transaction. Mutation and Load share one writer adapter but retain distinct +sidecar/audit kinds, so the five writer classes below map to six `SidecarKind`s. +Foreign, buried, or ambiguous effects fail closed. + +The sidecar is not a second WAL or transaction log. Lance remains the owner of +table transactions and durable data. The sidecar records the authority, +ownership, intended graph delta, and evidence needed to resolve the temporary +gap between those Lance effects and graph visibility. + +### Active graph-visible writers + +| Writer | Physical adapter | Visibility and recovery posture | +|---|---|---| +| Mutation / load | One staged keyed or overwrite transaction per touched table; deletes stage too | Exact transaction identities, fixed lineage, and one final manifest publish. Effect-free insert/upsert contention may fully reprepare only under the typed rules; any owned or ambiguous effect becomes `RecoveryRequired`. | +| Branch merge | A bounded ordered chain of filtered inserts/upserts/deletes per changed table | One v9 intent covers physical effects, pointer-only updates, first-touch refs, and the complete merge delta. The target is never silently reparented after planning. | +| SchemaApply | Metadata-only pure type rename, exact staged rewrite for property/schema changes, or strict first-touch create | The fixed schema identity, table delta, lineage, and source/IR/state promotion are one recoverable outcome. Every supported rename preserves logical identity and table lifetime; only a pure type rename also preserves Lance version/index history. Drop/re-add does neither. | +| EnsureIndices | One staged mixed `CreateIndex` transaction per productive table | Indexes remain derived state. Exact transactions and the complete pointer delta publish once; untrainable vector work stays pending instead of breaking logical writes. | +| Optimize | Lance compaction, incremental index optimization, and missing-index materialization across productive tables | One graph-wide v9 envelope and at most one monotonic manifest/lineage publish. Provenance is bounded rather than exact because Lance does not expose a caller-minted maintenance transaction surface. | + +### Explicit adjacent paths + +These operations do not create a side door around the protocol: + +| Operation | Why it is different | +|---|---| +| Native graph-branch create/delete | Lance `BranchContents` is the sole logical authority. Create/delete use a smaller authority-derived control protocol, emit no graph lineage, and reconcile clone-tree/ref crash gaps within the single-writer-process boundary. | +| Repair | Repair is main-only and does not manufacture a new table effect. It classifies already-uncovered HEADs and publishes selected adoptions together in one manifest/lineage commit after explicit operator confirmation; suspicious or unverifiable drift additionally requires force. | +| Cleanup | Cleanup is destructive physical version GC, not graph publication. It refuses unresolved recovery and uncovered main-HEAD drift, protects lazy-branch inherited main versions, and performs graph-wide preflight before fault-isolated per-table GC. The CLI requires confirmation; the public engine method executes the requested cleanup directly. | + +## What is done + +| Area | Implemented state | +|---|---| +| Publication surface | The historical Run state machine and `__run__*` staging branches are gone. The graph-write storage surface is crate-private, sealed, and staged-only; source guards make a new durable gateway an explicit review event. | +| Lineage | `graph_commit` and `graph_head` live in `__manifest` and land in the same CAS as table pointers. The former secondary commit-graph datasets are retired. | +| Mutation / load | Multi-statement writes accumulate in `MutationStaging`, provide read-your-writes, stage deletes as well as constructive writes, and publish once after complete validation. D2 deliberately keeps one query constructive or destructive. | +| Identity / schema | Accepted SchemaIR v2 owns graph-scoped, monotonic, no-reuse type/property/incarnation IDs. Supported renames preserve logical identity and table lifetime; a property rename rewrites that lifetime, while only a pure type rename also preserves physical/index history. Drop/re-add mints a new lifetime. The strict strand serves only internal schema v6 and upgrades by export/init/load rebuild. | +| Key fencing | Every graph table has exact non-null physical `id` as Lance's unenforced PK. Strict insert/upsert use the sealed exact-`id` filtered adapter; bare keyed Append is forbidden; effect-free conflicts have typed reprepare or `KeyConflict` outcomes. | +| BranchMerge | The ordinary ordered diff remains the correctness path. A completely verified `omnigraph.insert_absence = "v1"` history permits the proven-insert shortcut without committing Append or weakening the final filter. | +| Validation | Value, enum, uniqueness, edge-RI, and cardinality use one catalog-derived, delta-scoped evaluator across mutation, load, and merge. Committed non-key uniqueness probes are batched by constraint group and bounded chunks. | +| Recovery | Same-process handles share ordered queues for the same canonical local root or identical normalized opaque remote/custom URI. Mutation/load, SchemaApply, BranchMerge, EnsureIndices, and `refresh` heal roll-forward-only; Optimize, Repair, and Cleanup refuse pending recovery, while branch controls use specialized barriers. Read-write open performs the quiesced full sweep; read-only open never repairs. A resolved intent is audited internally with the original actor when present; no-effect cleanup need not create graph lineage. | +| Lance access | One process-wide `ObjectStoreRegistry` reuses clients. Each `Omnigraph` handle owns its cached data-table `Session`; one process-wide zero-cache control `Session` opens mutable tips. Only the object-store registry is shared between the data and control sessions. This is “cache the past, never the present,” not one global cached session. | +| Maintenance | EnsureIndices stages exact missing-index transactions. Optimize coordinates graph-wide compaction/index work under one bounded recovery envelope. Periodic optimize compacts `__manifest`, but unmaintained history-dependent paths are not globally flat. | + +Selected enforced limits are 8,192 rows / 32 MiB per keyed Mutation/Load table, +at most 32 pre-effect full reprepares, 8,192 rows / 32 MiB per BranchMerge +chunk, one aggregate 32-MiB retained merge-validation budget, at most 1,024 +logical data transactions per merged table, and a 1,026-version exact-recovery +scan bound. Optimize defaults to eight physical tasks and a five-attempt +compaction retry budget. This is not the complete constants catalog. The +important loud outcomes are `KeyConflict`, `ReadSetChanged`, +`RecoveryRequired`, and `ResourceLimitExceeded`; none reports partial success. +See [writes.md](writes.md) and [constants.md](../user/reference/constants.md) for +retry rules, recovery classification, and the full owned limits. + +## RFC implementation state + +| RFC | Current disposition | What that means now | +|---|---|---| +| [022 — Unified graph-write protocol](../rfcs/0022-unified-write-path.md) | **Implemented** | The shared correctness state machine, per-writer adapters, recovery barrier, one visibility point, control exceptions, and test lattice are the stable chassis. Completion is scoped to one writer process per graph for destructive recovery. | +| [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 activates exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence. | +| [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | +| [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked in the Gate-0 follow-up; PR #364 left it Draft** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active; production remains schema v6. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; bounded Gate E0 passed; Phase A pending** | MemWAL is the strategic substrate and the acknowledgement/fold/quiescence contract is specified. RC.1 lacks the ideal receipt/seal API, but exact `N`/`N + 1` successor classification, buried-effect refusal, flat attempt shape, and strict object-store negatives pass for the bounded profile. This is evidence only: no stream capability, lifecycle row, format, API, or acknowledgement path is active. | +| [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | +| [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation are active in v6. | + +## Blockers and constraints discovered + +| Frontier | Evidence and current consequence | Exit condition | +|---|---|---| +| Distributed recovery fence | Process-local queues cannot stop a live foreign process; Lance restore may orphan its commits and native refs lack conditional compare-delete. Supported destructive recovery remains one writer process per graph. | A separately designed and adversarially tested distributed fence before multi-process writers, background compensation, or cross-process exact maintenance recovery. | +| History-flat authority | RFC-024/025 show flat BTREE rows/ranges/pages can coexist with history-growing manifest discovery or compacted bytes. No heads/checkpoint format is active; mutable tip caches and a second authority remain rejected. | A new Lance-native access shape—or revised measured operational contract—passes the original cold/warm, compacted/uncompacted, local/object-store gate. | +| Internal history GC | Safe live-writer cleanup needs a durable resurrection/retention boundary; otherwise a stalled writer can recreate a collected version. | An evidence-backed cleanup watermark/fence before automated `__manifest` version GC. | +| MemWAL enrollment | RC.1 initializes the system index and claims shards as separate effects without a caller-minted combined receipt or cross-process seal. Gate E0 passed for main-only, one-shard, one-live-writer-process, exclusive-base-HEAD ownership: pinned `has_successor_version` checks exact `N + 1`/`N + 2` without latest/list, and strict local/RustFS state classification fails closed. | Implement production enrollment recovery, writer exclusion, lifecycle/admission-lease, crash, format, and rebuild gates for the bounded profile. The public receipt/seal remains the exit for broader topology. | +| O(delta) merge | A version-column predicate is still O(rows) without a selective source, and deleted rows have no live version columns. Full ID differencing remains correct. | Bounded live-row and deletion/change discovery, exact shadow agreement, and a table-size-flat one-row-delete gate. | +| Optimize provenance | Compaction/reindex has no stable caller-minted transaction covering the complete effect. Optimize therefore uses bounded, not exact, provenance. | Both an upstream maintenance transaction API and distributed recovery fencing. | +| Remaining bounds/operations | Some long-running operations lack complete memory/time budgets; rollback waits for quiesced read-write open; recovery audit has no public query. | Incremental, independently owned hardening without widening format or topology. | + +RFC-026's bounded candidate separates two facts that earlier wording +incorrectly collapsed. The enrollment binding is stable: logical +table/incarnation, location/main ref, never-reused enrollment/shard IDs, and +configuration. The public Lance composite of branch identifier, current table +version, transaction UUID, and manifest e_tag is a mutable +`CurrentHeadWitness`; every ordinary commit changes it. While a bounded stream +is `OPEN`, only fold/recovery may advance that base HEAD, and the next witness +must publish atomically with the table pointer. Other writer/control/ +maintenance paths touching the table refuse pre-effect or drain first. Gate E0 +proved the classifier and witness model with complete direct-probe and +object-store evidence; the next production slice must enforce this reversible +support restriction. It is not a claim about code already shipped. + +The full known-gap ledger, including adjacent local-CAS and unsupported +multi-version-topology details, remains in +[invariants.md#known-gaps](invariants.md#known-gaps). + +## How Lance 9.0.0-rc.1 influenced the plan + +RC.1 mostly validated the direction and sharpened gates. It did not justify a +write-path redesign. + +| RC.1 finding | Planning effect | +|---|---| +| Lance surfaces consumed by RFC-022/023—transactions, branches, key filters, staged indexes, compaction, and shared sessions—remain compatible; separately surveyed tag/cleanup behavior is also unchanged | Keep the current architecture. PR #364 passed 22 surface guards and 129 runnable failpoint tests; the Gate-0 follow-up adds the 23rd guard plus checkpoint cost evidence. No format redesign is needed. | +| Derived MemWAL datasets now inherit the base store parameters and `Session` | Strengthens the shared-session and remote-credential design and removes one integration concern. It does not create an exact combined enrollment receipt or cross-process seal. MemWAL is strategic, not experimental; Gate E0 passed the narrower exact-version public-state classifier without treating upstream timing as a production plan. | +| Experimental, opt-in `DataOverlay` was added | Do not adopt it now. OmniGraph does not enable feature flag 64 or emit the operation; unknown foreign overlay effects remain fail-closed. DataOverlay's experimental status says nothing about MemWAL's status. | +| RC.1 expands Lance write rejection from the three row-address names to all five surveyed virtual system-column names | OmniGraph now rejects `_rowid`, `_rowaddr`, `_rowoffset`, `_row_created_at_version`, and `_row_last_updated_at_version` during parsing and accepted-IR validation. A beta.21 development graph using a newly reserved row-version name must be exported with beta.21, renamed, and rebuilt. | +| A genuine ordinary-schema beta.21 V2_2 graph forward-opened, queried, and merge-wrote under RC.1 | No general storage-format migration. The reserved-name exception is explicit rather than hidden behind a format bump. | +| RFC-024's 10/100/1,000 run added a bounded one-operation boundary while the rejected compacted-byte slope remained | Durable heads stay research-blocked; do not weaken the gate. The RC made the no-go slightly clearer, not the design more attractive. | +| A separate RFC-025 Gate-0 follow-up run on the RC.1 pin—not part of PR #364—found the same complete-I/O problem for checkpoint authority | Keep the logical checkpoint/tag ordering, reject the current access shape, and stop before production format/API work. | +| Maintenance still has no caller-controlled exact transaction | Optimize's bounded adapter and single-writer-process recovery boundary remain necessary. | +| DataFusion moved 53 -> 54; Arrow and explicit V2_2 stayed stable | Mechanical dependency/type alignment, not an architecture change. | +| RC.1 requires Rust 1.91+ and remains a git-pinned prerelease | Continue the full alignment audit on every bump. Crates.io/stable release work remains gated on Lance 9 stable. | + +The matched merge-all-changed diagnostics were effectively neutral: RC.1 was +about 0.6% slower at 10K and 1.9% at 100K in single matched runs, with similar +or slightly lower incremental RSS. That is not a roadmap signal. + +## Next logical steps + +### Now + +1. **Stop RFC-025 at the Gate-0 no-go.** Keep only the production-neutral + substrate guards, cost instrument, and recorded decision; do not add + checkpoint rows, production `ogcp_` tags, format stamps, or public commands. +2. **Keep production on internal schema v6.** RFC-024/025/027 blockers and + RFC-026's green production-neutral Gate E0 are independent research results, + not reasons to reopen the implemented v6 correctness work. +3. **Implement only RFC-026 Phase A's production foundation next.** Add exact + roll-forward enrollment recovery, central `OPEN`-table writer exclusion, + lifecycle/current-witness state, and admission-lease races. Preserve the + proven exact-version/cleanup-exclusion classifier while keeping `@stream`, + public APIs, and acknowledgements inactive. +4. **Give a distributed recovery fence its own design and evidence gate.** + Define authority, expiry/renewal, fencing tokens, and crash semantics before + implementation; require adversarial multi-process tests on local and object + storage. +5. **Continue low-risk v6 hardening.** Add missing resource/time budgets, + preserve cost-at-history-depth gates, and reduce constant factors only where + the existing authority model remains intact. +6. **Coordinate upstream without making it the calendar.** The useful + Lance asks are recoverable MemWAL enrollment/admission, conditional native + ref operations, exact maintenance transaction provenance, and a bounded + deletion/change-lineage source. + +### Only when an evidence trigger fires + +- **MemWAL activation:** Gate E0 is green, but durable acknowledgements and + upsert/fold activate only after Phase A's enrollment recovery, + writer-exclusion, lifecycle/admission, format/refusal/rebuild, and full crash + gates are green. The exact upstream receipt/seal remains the preferred + simplification and the broader-topology gate. +- **Durable heads or checkpoints:** when a new current-authority access shape + exists, run the full decision instrument before adding production rows or a + format stamp. +- **Lineage merge deltas:** when both live-row and deletion discovery are + selective, shadow the new classifier against the existing truth-table path + and prove cost flat in table size. +- **Exact Optimize provenance:** when Lance exposes the required maintenance + transaction and the distributed fence exists, replace the bounded adapter + rather than layering another classifier beside it. +- **Lance 9 stable:** repeat the full upstream diff/spec audit, surface guards, + compatibility proof, failpoint suite, and cost instruments. Do not infer + stable behavior from RC.1 by name alone. + +## Explicit non-goals for the next slices + +Do not: + +- build a custom WAL, lock table, transaction manager, or buffer pool; +- restore a mutable in-memory graph tip as commit authority; +- create a second heads/checkpoint authority merely to escape the measured + manifest-access cost; +- implement RFC-024, RFC-025, RFC-026, or RFC-027 production paths behind a + nominal feature flag before their blocking gate closes; +- treat a green RFC-026 Gate E0 as format or acknowledgement activation; +- permit another base-table writer to advance an `OPEN` bounded stream's HEAD, + or use a long-lived enrollment tag without first measuring retained files and + bytes across rewrite/compaction/cleanup; +- treat process-local gates or exact transaction UUIDs as a distributed fence; +- infer table ownership from aliases, paths, matching versions, or compatible + schemas; +- give mutable control datasets the same cached-session policy as immutable + version-pinned data; or +- use private Lance APIs to make an upstream lifecycle gap appear closed. + +## Evidence map + +| Contract | Primary checked-in evidence | +|---|---| +| Lance surfaces and upgrade tripwires | [`lance_surface_guards.rs`](../../crates/omnigraph/tests/lance_surface_guards.rs) | +| No graph-write side doors | [`forbidden_apis.rs`](../../crates/omnigraph/tests/forbidden_apis.rs) | +| Mutation/load atomicity, conflicts, and resource limits | [`writes.rs`](../../crates/omnigraph/tests/writes.rs), [`consistency.rs`](../../crates/omnigraph/tests/consistency.rs) | +| Recovery and crash windows | [`failpoints.rs`](../../crates/omnigraph/tests/failpoints.rs), [`recovery.rs`](../../crates/omnigraph/tests/recovery.rs) | +| Schema identity and schema publication | [`schema_apply.rs`](../../crates/omnigraph/tests/schema_apply.rs), [RFC-028](../rfcs/0028-stable-schema-identity.md) | +| Key fencing and bounded branch adoption | [`merge_fast_forward.rs`](../../crates/omnigraph/tests/merge_fast_forward.rs), [`staged_tests.rs`](../../crates/omnigraph/src/table_store/staged_tests.rs), [RFC-023](../rfcs/0023-key-conflict-fencing.md) | +| Unified validation and merge semantics | [`validators.rs`](../../crates/omnigraph/tests/validators.rs), [`merge_truth_table.rs`](../../crates/omnigraph/tests/merge_truth_table.rs) | +| Native graph-branch controls | [`branching.rs`](../../crates/omnigraph/tests/branching.rs) | +| Maintenance and derived-index visibility | [`maintenance.rs`](../../crates/omnigraph/tests/maintenance.rs) | +| Warm read/write and merge cost at history depth | [`warm_read_cost.rs`](../../crates/omnigraph/tests/warm_read_cost.rs), [`write_cost.rs`](../../crates/omnigraph/tests/write_cost.rs), [`merge_cost.rs`](../../crates/omnigraph/tests/merge_cost.rs) | +| Durable-head decision gate | [`durable_head_lookup_cost.rs`](../../crates/omnigraph/tests/durable_head_lookup_cost.rs), [RFC-024](../rfcs/0024-durable-table-heads.md) | +| Checkpoint-retention Gate 0 | [`checkpoint_retention_cost.rs`](../../crates/omnigraph/tests/checkpoint_retention_cost.rs), [RFC-025](../rfcs/0025-checkpoint-retention.md) | +| MemWAL bounded-enrollment Gate E0 (green; production inactive) | [`memwal_enrollment_gate.rs`](../../crates/omnigraph/tests/memwal_enrollment_gate.rs), [`lance_surface_guards.rs`](../../crates/omnigraph/tests/lance_surface_guards.rs), [RFC-026 §12.1](../rfcs/0026-memwal-streaming-ingest.md) | +| Cross-version refusal/rebuild | [`crossversion_upgrade.rs`](../../crates/omnigraph-cli/tests/crossversion_upgrade.rs) | + +Use [testing.md](testing.md) to find the existing owner before adding coverage. +Decision instruments and production correctness tests serve different purposes: +a green no-go-preservation test means the rejected result was reproduced, not +that the candidate became shippable. RFC-026 Gate E0 is green at a precise +boundary. Fourteen substantive local cells cover the state/negative matrix, +buried-effect refusal, and the exact-version cost proof; baseline versions 8 +and 80 have the same complete six-attempt shape (four successful HEADs, one +`NotFound` HEAD, one successful GET) with zero lists. A Unix permissions +tripwire distinguishes exact probing from latest enumeration and forces an +unreadable exact HEAD to error. The configured RustFS cell passes non-vacuously +with the same zero-list shape and the declared positive plus listing-dependent +negative matrix; separate surface guards own S3 ABA and the doc-hidden Lance +surfaces. This decision authorizes Phase A only—no format or acknowledgement +activation follows. + +## Updating this page + +Update this page when any of the following changes: + +- a new graph-visible writer or control-path exception is added; +- recovery ownership, support topology, or the manifest visibility boundary + changes; +- the internal schema or strict rebuild contract changes; +- an RFC moves between draft, research-blocked, accepted, and implemented; +- a Lance bump changes a consumed surface or closes an upstream gate; +- a cost instrument reverses a no-go or reveals a new history-dependent term; + or +- a known gap is closed or promoted into the public support contract. + +Keep detailed mechanics in [writes.md](writes.md), numeric evidence in the +owning RFC/test, and public behavior in the relevant [user docs](../user/index.md). +This page should remain the shortest accurate answer to “where is the write path +now, and what should we do next?” diff --git a/docs/rfcs/0024-durable-table-heads.md b/docs/rfcs/0024-durable-table-heads.md index 75d78199..5c66e77a 100644 --- a/docs/rfcs/0024-durable-table-heads.md +++ b/docs/rfcs/0024-durable-table-heads.md @@ -17,7 +17,7 @@ owner: OmniGraph maintainers `cec0b7dffe2d85c7e66dbe9d1f3891c297903a1d`; full Lance table layout, transaction, branching, indexing, compaction, cleanup, and object-store specifications -**RC.1 evidence status (2026-07-17):** the public physical-ref and BTREE +**RC.1 evidence status (2026-07-17):** the public current-HEAD-witness and BTREE surfaces remain aligned. The local 10/100/1,000 decision instrument was rerun: exact indexed rows, ranges, result fragments, and pages remain flat; RC.1 adds one bounded range-read operation and at most 128 scan bytes at the deepest @@ -164,16 +164,24 @@ TableHeadMetadata { state: "live" | "tombstoned", stable_table_id: u64, incarnation_id: u64, - physical_ref_incarnation: String, + current_head_witness: CurrentHeadWitness, schema_ir_hash: String, head_graph_commit_id: Option, } + +CurrentHeadWitness { + branch_identifier: BranchIdentifier, + table_version: u64, + transaction_uuid: UUID, + manifest_e_tag: String, +} ``` -`physical_ref_incarnation` is the opaque encoding of one public Lance composite: +`current_head_witness` is the encoding of one public Lance composite: ```text -(BranchIdentifier, current Transaction.uuid, ManifestLocation.e_tag) +(BranchIdentifier, current table version, current Transaction.uuid, + ManifestLocation.e_tag) ``` Capture reads `BranchIdentifier` before and after the current transaction and @@ -183,9 +191,12 @@ e_tag fails closed. On pinned Lance, local `current_manifest_path` synthesizes t e_tag from inode, mtime, and size; S3/RustFS returns the object e_tag. Main's `BranchIdentifier` is canonically empty, so its transaction UUID and e_tag are load-bearing; a named-ref recreation additionally changes `BranchIdentifier`. -Logical `incarnation_id` cannot substitute for this composite because a -physical owner or ref may be replaced while logical table identity is -preserved. +This is deliberately a **mutable current-HEAD witness**, not a stable physical +or enrollment incarnation. An ordinary successful table commit changes the +version, transaction UUID, and manifest e_tag, so the publisher must replace +the witness with the exact achieved value in the same head-row update. Logical +`incarnation_id` cannot substitute for the witness because a physical owner or +ref may be replaced while logical table identity is preserved. The public-surface guards prove stable unchanged reopens and same-version delete/recreate detection for main and named refs on local FS and S3/RustFS, @@ -231,8 +242,8 @@ transition then follows that accepted schema outcome; RFC-024 never invents it. The mutable head is not the only place that records identity. Every heads-format table-version, registration, rename, and tombstone journal event carries -`stable_table_id`, `incarnation_id`, and the exact physical location/ref token -for its table version; otherwise drop/recreate followed by a new physical +`stable_table_id`, `incarnation_id`, and the exact `current_head_witness` for +its table version; otherwise drop/recreate followed by a new physical dataset whose Lance versions restart cannot be replayed unambiguously or repair the full head token. @@ -281,7 +292,7 @@ complete expected token: ```text (state, stable_table_id, incarnation_id, location, table_branch, - physical_ref_incarnation, table_version, schema_ir_hash) + current_head_witness, table_version, schema_ir_hash) ``` Any difference returns control to full RFC-022 revalidation before effects; a @@ -311,7 +322,7 @@ A heads-format current-state read: 4. includes only rows whose authoritative state is `live`; 5. validates schema identity from the head payload; 6. opens the exact pinned physical table/ref and validates - `physical_ref_incarnation` before exposing it; + `current_head_witness` before exposing it; 7. returns one immutable `Snapshot` used for the operation's lifetime. Missing, duplicate, unknown-state, or schema-mismatched **live** heads fail @@ -548,8 +559,10 @@ No migration claimant, per-branch conversion ledger, old-format writer mode, or dataset restarts Lance version numbering; - rename preserves identity/incarnation and changes the public key only; - owner-branch handoff at an equal table version updates the head; +- every ordinary table commit advances `current_head_witness` together with + `table_version` in the same manifest publish; - delete/recreate of a dataset or native ref at the same path, branch, and - numeric version changes `physical_ref_incarnation` and rejects a stale + numeric version changes `current_head_witness` and rejects a stale writer, on local FS and S3/RustFS; public token detection is proven, while publisher-level stale-writer rejection remains unimplemented; - a current read refuses that same replacement until an authoritative publish @@ -568,7 +581,7 @@ No migration claimant, per-branch conversion ledger, old-format writer mode, or - rollback and roll-forward assertions include head payloads, not only table versions; - publisher retry compares the complete expected token and never reparents a - prepared effect across a physical-ref incarnation change; + prepared effect across a current-HEAD-witness change; - a stale sidecar converges exactly once with one audit record. ### 12.3 Format and rebuild @@ -620,7 +633,7 @@ coverage and does run against RustFS in CI. - exact heads-format metadata schema and object IDs; - one head row per stable identity; -- RFC-028 stable-ID/incarnation types plus `physical_ref_incarnation` in head +- RFC-028 stable-ID/incarnation types plus `current_head_witness` in head and identity-bearing journal event schemas; - RFC-023 PK metadata on node and edge tables when the release combines them; - heads-format publisher source always pairs a journal/tombstone event with a head row. diff --git a/docs/rfcs/0025-checkpoint-retention.md b/docs/rfcs/0025-checkpoint-retention.md index 3f0a76f8..cba750fe 100644 --- a/docs/rfcs/0025-checkpoint-retention.md +++ b/docs/rfcs/0025-checkpoint-retention.md @@ -2,7 +2,7 @@ type: spec title: "RFC-025 — Checkpoint-pinned retention" description: Makes named graph checkpoints authoritative retention roots, materializes them as Lance-native manifest and data-table tags, and defines crash-safe reconciliation and cleanup ordering on the RFC-022 unified write path. -status: draft +status: research-blocked tags: [eng, rfc, retention, checkpoint, cleanup, manifest, lance, omnigraph] timestamp: 2026-07-10 owner: OmniGraph maintainers @@ -10,8 +10,10 @@ owner: OmniGraph maintainers # RFC-025 — Checkpoint-pinned retention -**Status:** Draft / for team review +**Status:** Research-blocked — Gate 0 rejected the current in-manifest BTREE +access shape; no retention format is active **Date:** 2026-07-10 +**Gate 0 evaluated:** 2026-07-17 **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s publisher and recovery-sidecar protocol and @@ -29,10 +31,11 @@ Findings marked **BLOCKER** must be dispositioned before acceptance. ## 0. Decision -OmniGraph adopts checkpoint-pinned retention. A checkpoint is a durable, named -graph snapshot: one reference set in the reserved main-manifest registry -containing every physical manifest/table lineage and version needed to -reconstruct that graph state. +OmniGraph retains checkpoint-pinned retention as the target architecture, but +activation is research-blocked on a history-flat current-authority access +shape. A checkpoint is a durable, named graph snapshot: one reference set in +the reserved main-manifest registry containing every physical manifest/table +lineage and version needed to reconstruct that graph state. The checkpoint rows are the **logical authority**. Lance tags are the **physical pins** that make that authority effective. This distinction is @@ -48,6 +51,49 @@ The safe ordering is asymmetric: Every crash window consequently over-retains. None under-retains. +### 0.1 Pre-activation Gate 0 + +Retention is an irreversible storage-format capability, so its substrate and +access-shape claims are proved before any production activation work starts. +Throughout Gate 0, production remains on internal schema v6: `CURRENT` and +`MIN_SUPPORTED` do not move, fresh graphs gain no checkpoint or GC-boundary +rows, and no production path creates an `ogcp_` tag. Gate 0 may add only +decision fixtures, substrate guards, and their recorded measurements. + +Gate 0 must establish all of the following on the pinned Lance release: + +- deterministic internal tag names pass Lance validation and resolve the exact + main or named-branch version recorded by the proposed checkpoint row; +- a sparse tagged version survives cleanup while adjacent eligible versions + are removed, deleting the tag makes that version collectable, and a tag on a + named branch does not protect the branch tree from deletion; +- the proposed physical-ref token is stable across an unchanged reopen and + changes after same-path, same-branch, same-version delete/recreate ABA on + local storage and S3/RustFS; +- the storage adapter's atomic create-if-absent operation admits exactly one + retention-claim owner, a mismatched token cannot release the claim, and a + paused live owner cannot be taken over; and +- checkpoint list, exact lookup, and cleanup-root planning pass §9's complete + physical-I/O gate across cold/warm, compacted/uncompacted, index-absent, + reconciled, and bounded-uncovered-tail cells at realistic history depth. + +Lance tag creation on the surveyed release is an existence check followed by +an unconditional put, not a conditional create. The retention claim and exact +post-create target verification are therefore load-bearing even when the tag +surface tests pass. + +**Gate 0 result (2026-07-17): no-go for the current in-manifest BTREE access +shape.** The Lance tag/cleanup substrate guards pass, but §9's decision-scale +local matrix shows history-sensitive compacted scan bytes and an additional +scan operation at depth 1,000. One required physical-I/O cell failing is +sufficient to block activation; the unrun S3/RustFS cost cell is not needed to +turn that result into a pass or a stronger claim. Internal schema v6 remains +authoritative. This result rejects the proposed access shape, not checkpoint +rows as logical authority or Lance tags as physical pins. A successor needs a +history-flat current-authority lookup shape or a revised evidence-backed +operational contract, without weakening the gate or adding a second authority +dataset. + ## 1. Scope and non-goals This RFC specifies: @@ -81,17 +127,58 @@ global registry, not branch-local graph state: deleting the source branch must not delete the authority that protects its snapshot. One main-manifest publish writes: -- `checkpoint_name:` — unique name reservation pointing at the - immutable checkpoint ID; +- `checkpoint_name:v1:` — unique, normalization-versioned name + reservation pointing at the immutable checkpoint ID; - `checkpoint:` — header containing `name`, source logical branch, `graph_commit_id`, source physical manifest branch and version, source-manifest `physical_ref_incarnation`, manifest schema stamp, - `created_at`, and `actor`; + `name_normalization_version`, `tag_encoding_version`, `created_at`, and + `actor`; - `checkpoint_table:::` — one row per pinned table containing stable table identity, table key, `table_path`, `table_branch`, `table_version`, and a separately named `physical_ref_incarnation` token (e_tag or the proven backend fallback). +#### 2.1.1 CheckpointName normalization v1 + +Checkpoint-name normalization is persisted equality semantics, not display +cleanup. Both the name-reservation row and header store +`name_normalization_version = 1` and the canonical name. The reservation key is +`checkpoint_name:v1:`. + +V1 is a validation plus identity transform over UTF-8 input: + +1. The input is not trimmed, case-folded, locale-folded, or Unicode-normalized. + Its canonical value is its exact input byte sequence after validation. +2. The canonical value is 1 through 128 bytes inclusive and contains ASCII + only. Non-ASCII UTF-8, invalid UTF-8 at a decoding boundary, whitespace, + controls, NUL, and backslash are rejected rather than rewritten. +3. Slash separates non-empty segments. A name cannot start or end with `/` or + contain `//`. +4. Every segment matches `[A-Za-z0-9._-]+`. The complete name cannot contain + `..`, and no segment may end in `.lock`. +5. Canonical names beginning with the reserved `__` or `ogcp_` byte prefixes + are rejected. These prefix checks are byte-exact; V1 name equality remains + case-sensitive, so `Release` and `release` are distinct names. + +Every name-taking engine, CLI, and future transport entry point runs this one +parser before authorization or durable effects. Responses return the canonical +stored spelling; they do not retain a second display spelling. The same +canonical bytes are used for reservation lookup, ordering, policy scope, audit, +and conflict details. + +The V1 mapping is immutable once shipped. A future normalization rule uses a +new persisted version and format capability; it never reinterprets stored V1 +bytes or changes V1's tag names. That later specification must define its +cross-version equivalence relation. Before admitting a name, creation computes +the finite candidate keys for every supported normalization version and uses +bounded exact lookups to find an equivalent live **or tombstoned** reservation. +A live reservation conflicts; a tombstone participates in its existing +generation-CAS reuse rather than allowing an independent reservation under the +new version. A later format may instead use the strand rebuild and begin with +no checkpoints, but it cannot silently merge, split, or revive V1 +reservations. + Because RFC-024 heads are optional, RFC-025 owns this token contract for both the pinned manifest and every pinned table. The token identifies the immutable physical version object and its dataset/native-ref lineage for the exact @@ -106,7 +193,7 @@ activate the retention format. The name reservation, header, and every table row land in one RFC-022 publisher CAS on main. First creation is insert-only; reuse requires the exact tombstoned reservation generation in the `ReadSet`. Two concurrent attempts for the same -normalized name therefore conflict even when they captured different source +V1 canonical name therefore conflict even when they captured different source branches. A missing or duplicate table row is an invalid checkpoint, not a partial checkpoint. @@ -141,10 +228,21 @@ and manifest version; it never implies that the checkpoint tracks a moving tip. Every checkpoint owns deterministic Lance tags of two kinds: ```text -ogcp__manifest_ -ogcp__ +ogcp_v1__m_ +ogcp_v1__t_ ``` +`tag_encoding_version = 1` fixes this ASCII spelling. The digest is lowercase +hex SHA-256 over a domain-separated, length-prefixed encoding of the exact +physical target. The manifest domain includes its identity-derived +dataset-relative path, branch, version, and physical-ref token. The table +domain additionally includes stable table ID and incarnation plus the +identity-derived table path, branch, version, and physical-ref token. +Length-prefixing makes field boundaries unambiguous; the implementation test +vectors are part of the format gate. The user-visible checkpoint name is +deliberately absent from the tag: name-normalization evolution cannot rename a +physical pin. + The manifest tag targets the exact `(manifest_branch, manifest_version)` named by the header. Each table tag targets the exact `(table_branch, table_version)` recorded by its table row. The spelling stays within Lance tag-name constraints @@ -417,6 +515,13 @@ introduced here. ## 8. Format activation and rebuild compatibility +Gate 0 was production-neutral and returned a no-go for the surveyed access +shape. The shipped format therefore remains internal schema v6 and none of the +activation or rebuild behavior below exists. The remainder of this section is +the contingent format contract for a successor that first clears the same +physical-I/O boundary; the Gate 0 result itself authorizes no implementation or +format bump. + This RFC owns its own internal-format activation. RFC-022 authorizes no format bump, and RFC-024 explicitly excludes checkpoint rows from its heads format. Retention may therefore be the next independently accepted format capability @@ -455,6 +560,14 @@ another independently accepted capability, the first valid target state contains all of them and the combined initialization/recovery matrix must pass. Separate format releases imply separate rebuilds. +The retention stamp must not ship as an inert capability. The first activated +format must include, in one releasable slice, its row initialization and +strict-rebuild/refusal boundary plus functional engine create, list, show, and +delete, the retention claim, exact tag verification, recovery, and pin +reconciliation. Development-only pieces may land behind test-only seams before +that slice, but production must stay on v6 until the minimum usable contract is +complete. + ## 9. Observability and bounds Expose: @@ -479,6 +592,42 @@ still reads history-sized manifest fragments does not pass this RFC's cost gate; a separate checkpoint dataset is rejected because its authority rows could not share the main-manifest CAS. +### 9.1 Gate 0 evidence — current access shape rejected + +The checked-in `checkpoint_retention_cost.rs` fixture holds three live +checkpoints and catalog width ten constant while unrelated manifest-journal +history grows. It measures the complete list, exact show, and cleanup-root +authority reads from a tracked cold open and a warm repeat over absent, +reconciled, and eight-fragment uncovered-tail index states, with both compacted +and uncompacted layouts. Object-store I/O and Lance execution-summary I/O are +reported separately. + +The ignored local decision run at 10, 100, and 1,000 real commits found: + +- the reconciled **uncompacted** path is history-flat at the 10→1,000 + endpoints. List reports 3 rows / 3 ranges / 1 fragment / 1 index page and 24 + scan operations / 13,752 scan bytes. Exact show reports 12 rows / 2 ranges / + 2 fragments / 3 pages and 34 operations / 22,952 bytes. Cleanup returns 44 + rows with the list-like physical cost. The bounded eight-fragment tail stays + exact, observable, and history-flat; +- the reconciled **compacted** path fails the required physical-I/O slope. + List and cleanup cold scan bytes grow from 17,012 to 38,000, and warm bytes + grow from 12,336 to 15,064. Exact-show cold bytes grow from 29,348 to 53,064, + and warm bytes from 24,672 to 30,128. At depth 1,000 the one-scan operations + also cross from 24 to 25 scan operations, while show crosses from 34 to 35. + +The default local 20/80 matrix passes its checked-in **no-go preservation** +assertions; that means it reproduces the current blocker, not that Gate 0 +passes. The S3/RustFS matrix is checked in and bucket-gated but was not run for +this decision, so this RFC makes no S3 cost claim. + +Separately, all 23 Lance surface guards pass on RC.1. The RFC-025 cells prove exact +main and named-branch tag targets, sparse cleanup pin/unpin behavior, and that +a tag does not protect the named branch tree. These results validate the +physical-pin architecture but cannot compensate for the failed registry-access +gate. Production remains on schema v6 until a successor access shape or revised +operational contract passes the full boundary. + ## 10. Acceptance gates - A checkpoint on an old sparse version survives cleanup while adjacent @@ -493,7 +642,7 @@ share the main-manifest CAS. path, branch name, and numeric version with a different physical-ref token; checkpoint create, recovery, and reconciliation all refuse the replacement. A backend without a proven token refuses retention-format activation. -- Concurrent creation of the same normalized name yields exactly one authority +- Concurrent creation of the same V1 canonical name yields exactly one authority record; the losing attempt leaves only safely reclaimable tags. - After deletion, concurrent attempts to reuse the tombstoned name generation yield exactly one fresh checkpoint ID; the old checkpoint ID never revives. @@ -530,7 +679,7 @@ share the main-manifest CAS. | Phase | Content | Gate | |---|---|---| -| A | row schemas, engine DTOs, strict-format/rebuild activation, deterministic tag encoding | schema, refusal, and tag surface guards | -| B | create sidecar, delete-operation fields in the authority CAS, and reconciler | crash matrix; sparse-pin correctness | -| C | offline cleanup integration and GC boundary enforcement | quiescence/refusal tests; cost budgets | -| D | CLI, policy, audit, docs, and rebuild runbook | CLI outputs; genuine upgrade test | +| Gate 0 — pre-activation decision | production-neutral tag/cleanup semantics, physical-ref ABA, retention-claim, and checkpoint-registry physical-I/O instruments | **No-go (2026-07-17):** tags pass; compacted local registry scan bytes grow at 10→1,000 and scan operations add one boundary. S3 cost cell not run. Production remains schema v6 | +| A — minimum usable activation | row schemas and engine DTOs; deterministic V1 name/tag encoding; retention claim; create/list/show/delete; create sidecar, delete-operation marker, pin reconciler; strict-format/refusal/rebuild activation | **Blocked on a successor Gate 0 access shape or revised evidence-backed operational contract.** Then: schema and deterministic encoding vectors; old/new refusal; complete crash matrix; sparse-pin correctness; no inert format stamp | +| B — destructive retention | offline cleanup integration and GC-boundary enforcement | blocked with A; then quiescence/refusal tests, complete root proof, and cost budgets | +| C — operator surface | CLI, policy, audit, observability, docs, and rebuild runbook | blocked with A; then CLI outputs, policy parity, and genuine upgrade test | diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index ab04083f..e289605c 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -10,8 +10,9 @@ owner: OmniGraph maintainers # RFC-026 — MemWAL streaming ingest -**Status:** Draft / for team review +**Status:** Draft / Gate E0 passed; Phase A pending; production inactive **Date:** 2026-07-10 +**Gate E0 evaluated:** 2026-07-18 **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and generic recovery-sidecar protocol, plus @@ -45,14 +46,19 @@ small adapter, compile/runtime surface guards, a quiescence requirement before Lance upgrades, and a fresh full-spec alignment audit on every bump. It is not a reason to fork or reimplement MemWAL. -One RC.1 API gap is a ship blocker, not evidence against the architecture: +One RC.1 API gap prevents the ideal enrollment adapter, not the bounded +evidence work: `InitializeMemWalBuilder::execute` internally commits the MemWAL `CreateIndex` transaction and returns only `Result<()>`, while initial shard-manifest creation is a separate object-store effect reached through `mem_wal_writer`. The public -surface therefore cannot yet provide the exact pre-arm effect identities and -reversible admission seal required by §3/§8. OmniGraph does not reach through -private Lance modules or hand-roll those objects. Streaming activation waits -for the public recoverable-enrollment/seal gate defined below. +surface therefore cannot provide the caller-minted transaction identity and +reversible cross-process admission seal required by the general profile in +§3/§8. OmniGraph does not reach through private Lance modules or hand-roll +those objects. It also does not make upstream release timing a calendar +prerequisite: Gate E0 below confirms that a deliberately narrower profile can +classify the public effects exactly under OmniGraph's existing +single-live-writer-process boundary. The exact upstream receipt/seal remains +the preferred simplification and the gate for broader topology. The contract is: @@ -62,6 +68,31 @@ The contract is: - a fold is an ordinary RFC-022 graph writer and is the sole visibility point; - fresh reads are explicit and never claim cross-table atomicity. +### 0.1 Gate E0: bounded-enrollment decision passed before activation + +Gate E0 is a production-neutral decision harness, not streaming activation. It +uses the pinned public Lance surface to establish that one exact first +enrollment can be classified after success, failure, and lost acknowledgement +without deleting or adopting ambiguous state. Its evidence-backed initial +profile is: + +- `main` only; +- one unsharded keyed-upsert shard per enrolled table; +- one live OmniGraph writer process for the graph, with a crash successor only + after external exclusivity has been established; +- no raw Lance writer and no overlapping second OmniGraph writer process; and +- exclusive ownership of the enrolled table's base HEAD while the stream is + `OPEN`: only the stream fold/recovery adapter may advance it. Every other + graph writer, branch/schema operation, repair, index/optimize path, or cleanup + that touches the table must refuse before effect or first drain the stream. + +Gate E0 added no manifest rows, sidecars, `@stream`, public APIs, WAL +acknowledgements, or a format stamp. Internal schema v6 remains the only served +production format. Its green result authorizes only the next implementation +slice (enrollment recovery, writer exclusion, and lifecycle integration); it +does not authorize durable stream admission. RFC-026 remains draft and +production inactive until the later gates close. + ## 1. Scope and non-goals This RFC specifies enrollment, the public stream API, acknowledgement semantics, @@ -74,6 +105,12 @@ default snapshot isolation. Stream-mode deletes remain out of the first delivery and require the Lance tombstone surface plus a separate acceptance pass. +The bounded initial profile additionally excludes non-main streams, +multi-shard ownership, overlapping writer processes, raw Lance writers, +cross-process `Fresh`, and concurrent interactive/maintenance HEAD movement on +an `OPEN` enrolled table. These are support-boundary refusals, not implicit +eventual-consistency modes. + ## 2. Stream mode and key semantics Initial delivery exposes @@ -101,7 +138,9 @@ may consume MemWAL append semantics under its own schema/API decision. Stream ordering intentionally differs from the interactive fence: -- concurrent interactive same-key writes serialize or fail/retry loudly; +- interactive same-key writes outside an `OPEN` bounded stream serialize or + fail/retry loudly; an `OPEN` bounded stream refuses such a write before + effect until it is drained; - same-key stream entries resolve by MemWAL generation/position order; - duplicate keys inside one bulk-load input retain the existing load error. @@ -113,40 +152,71 @@ Schema apply records `@stream` intent only. First stream use enrolls the physica table by creating the singleton `__lance_mem_wal` system index and its sharding configuration. -RFC-024 heads are optional, so this RFC carries its own exact physical binding -in every lifecycle row: +RFC-024 heads are optional. Every lifecycle row therefore carries two distinct +classes of evidence: ```text StreamPhysicalBinding { + stable_table_id, + incarnation_id, table_location, - table_branch, - physical_ref_incarnation, + table_branch, // exactly main in the bounded profile enrollment_id, // pre-minted UUID, never reused shard_ids, // sorted UUID namespace, never reused stream_config_hash, } + +CurrentHeadWitness { + branch_identifier, + table_version, + transaction_uuid, + manifest_e_tag, +} ``` -The binding is valid only when the current manifest table entry resolves to the -exact location/ref incarnation and that physical table contains the expected -MemWAL system index plus shard manifests for the recorded namespace. -`physical_ref_incarnation` is an opaque Lance/backend proof that remains stable -under ordinary HEAD advances but changes when the dataset or native ref is -deleted and recreated at the same path/name. For a non-main ref it includes the -native `BranchIdentifier`; for main it requires an equivalent dataset-root -incarnation proof. Path, branch name, numeric version, timestamp, and the -RFC-028 logical incarnation are not substitutes. A backend without a token -that passes local and S3/RustFS same-path/ref ABA guards cannot activate -streaming. +`StreamPhysicalBinding` is stable for one physical enrollment. The +`CurrentHeadWitness` is not: it is the exact public Lance composite at the +currently accepted base HEAD, using the same capture discipline as RFC-024. +Every ordinary Lance commit changes at least the table version and current +transaction UUID and normally the manifest e_tag. Calling that composite a +stable “physical-ref incarnation” is incorrect. + +The lifecycle binding is valid only when the current manifest table entry +resolves to the same stable table/incarnation and location/main ref, its +persisted `CurrentHeadWitness` equals the physical table's current witness, and +the table contains the expected singleton MemWAL index plus exactly the +recorded shard namespace/configuration. Path, branch name, numeric version, +timestamp, or the RFC-028 logical incarnation alone is never sufficient. Local +and S3/RustFS same-path/ref delete-recreate guards must change the witness; a +backend without that proof cannot activate streaming. + +The bounded profile makes the changing witness tractable by granting the +`OPEN` stream exclusive authority to advance that base-table HEAD. Every fold +captures the prior witness and publishes the achieved table pointer and next +witness in the same `__manifest` CAS. Any unexpected movement is foreign or +ambiguous and fails closed. A path that needs to mutate or maintain the table +must drain first and participate in the lifecycle/witness transition; it may +not silently coexist with an `OPEN` stream. A long-lived Lance tag is not the +default enrollment anchor: it would pin the enrollment-time table snapshot and +can retain an old full-table file set across rewrites/compaction, while also +adding another auxiliary effect to enrollment. It remains only a measured +fallback if a later profile requires concurrent base writers. The MemWAL index UUID is not used as enrollment identity because Lance may replace that metadata entry while advancing merged-generation state. A mutable table name, the RFC-028 logical pair alone, or a compatible-looking index is never enough. RFC-024 may project related table/ref facts into a durable head, but RFC-026 persists and validates this binding without depending on heads. +For the bounded profile, the pre-minted `enrollment_id` is also persisted as a +namespaced MemWAL `writer_config_defaults` marker together with the exact +configuration version. RC.1 explicitly permits arbitrary persisted default +keys, and merge-progress/index-metadata replacement must preserve them. This is +classifier evidence independent of the replaceable MemWAL index UUID; the +manifest lifecycle row remains the logical stream authority. -Enrollment advances Lance HEAD and provisions shard-manifest objects. Before -implementation, Lance must expose one of these public, surface-guarded shapes: +Enrollment advances Lance HEAD and provisions shard-manifest objects. The +preferred general-profile substrate remains one of these public, +surface-guarded shapes: 1. a caller-controlled staged/uncommitted MemWAL initialization transaction with an exact transaction identity, plus idempotent public APIs to provision, @@ -156,57 +226,95 @@ implementation, Lance must expose one of these public, surface-guarded shapes: classify/roll-forward/rollback semantics. The RC.1 `execute() -> Result<()>` initializer plus separately claimed shard -writer satisfies neither shape. Private-module access, compatible-looking -index inference, and direct object-store emulation are rejected. Until this -gate lands, `@stream` may be parsed/planned in draft work but the streaming -format and first enrollment do not activate. +writer satisfies neither general shape. Private-module access, +compatible-looking index inference, and direct object-store emulation remain +rejected. Gate E0 established that the public effects are nevertheless exact +enough for the bounded profile. Until the following production adapter's later +activation gates pass, `@stream`, the streaming format, first enrollment, and +acknowledgement all remain inactive. -Once that public gate exists, enrollment uses one RFC-022 multi-effect sidecar, -not an ad-hoc state machine: +With Gate E0 green, the proposed bounded enrollment uses one RFC-022 +multi-effect sidecar, not an ad-hoc state machine: 1. run and await RFC-022's synchronous recovery barrier; 2. authorize, pin the manifest/schema/table state, and prepare a complete - `ReadSet` containing schema identity, stable table ID and incarnation, exact - physical-ref incarnation and pre-enrollment HEAD, PK metadata, stream - intent/configuration, and lifecycle-row absence for a first enrollment or + `ReadSet` containing schema identity, stable table ID and incarnation, + location/main ref, exact pre-enrollment `CurrentHeadWitness`, PK metadata, + stream intent/configuration, and lifecycle-row absence for a first enrollment or the exact `SEALED` prior binding for a physical rebind; 3. acquire any global claims and then the `(table, branch)` write queue in RFC-022 order, then freshly revalidate the complete `ReadSet`; a mismatch restarts before any physical effect; 4. verify RFC-023's already-installed PK; enrollment never performs a first-use PK migration; validate the sharding configuration; -5. pre-mint a never-reused enrollment UUID and shard UUID namespace, then arm a - sidecar with writer kind `stream_enrollment` or `stream_rebind`, the exact - target/ref token, the index transaction/receipt, the deterministic initial - shard object set in physically non-admitting `SEALED` state, and the complete - intended `StreamPhysicalBinding`; -6. commit the exact index effect, then provision the exact `SEALED` shard - objects; neither effect admits appends; -7. classify both effects and mark the sidecar `EffectsConfirmed` only when the - complete intended set exists with exact ownership; -8. publish the achieved table version, exact physical binding, per-shard epoch - floors, and `stream_state = OPEN` in one manifest CAS, including the - table-head row when RFC-024 is active; -9. activate the bound shard claims idempotently, then delete the sidecar - best-effort. Ack paths run the recovery barrier and refuse a still-covered - enrollment, so `OPEN` plus an unresolved sidecar cannot acknowledge. - -Recovery rolls forward only after classifying both exact effects. Rollback -reclaims the exact empty owned shard objects first, then restores/removes the -owned index transaction; a foreign object, a nonempty shard, or an effect buried -by another transaction fails closed. It never infers ownership from the -preserved logical pair, a table name/path, or a compatible-looking index. -Repeating enrollment with the same exact binding is a no-op. A different PK, -physical binding, sharding spec, maintained-index set, or writer-default -configuration is a typed conflict. +5. pre-mint a never-reused enrollment UUID and one shard UUID, then arm a + sidecar with writer kind `stream_enrollment`, the exact baseline witness, + fixed intended binding/configuration (including the namespaced persisted + enrollment marker), and the only allowed successor shape; +6. call the public initializer only while the base-HEAD/exclusive-admission + gate is held. After success or a lost result, use the manifest-pinned direct + physical URI opener at `VersionResolution::At(N)` (the evidence harness uses + `DatasetBuilder::with_version(N)`), never latest resolution, and verify the + exact captured `N` witness. Then use the pinned public + `Dataset::has_successor_version` primitive to ask only whether `N + 1` + exists. `Ok(false)` means no effect only while the same gate and cleanup/GC + exclusion remain held. `Ok(true)` permits an exact `checkout_version(N + 1)`; + that handle must report no `N + 2` successor before its transaction is + accepted as exactly one singleton MemWAL `CreateIndex` reading `N`, with the + intended unsharded configuration and no unrelated change. A probe error, + overflow, detached-version boundary, same-version ABA, or existing `N + 2` + is ambiguous and returns `RecoveryRequired`; +7. pass the pre-minted UUID to the public shard-writer path and accept only its + exact empty initial shard state: expected shard/spec identity, observable + epoch, no flushed generation, and no data-bearing WAL entry. A deterministic + data-less fence sentinel is admissible only if Gate E0 first pins it as part + of that exact state; +8. once either allowed effect exists, recover only by rolling forward to the + fixed outcome. The bounded adapter never deletes/reclaims shard artifacts or + restores the base table from inferred ownership. Intervening HEAD movement, + a wrong index/config, a foreign shard, unexpected WAL data, or any ambiguity + retains the sidecar and returns `RecoveryRequired`; +9. publish the achieved table version, `CurrentHeadWitness`, stable physical + binding, exact pre-claim epoch floor, and `stream_state = OPEN` in one + manifest CAS, including the table-head row when RFC-024 is active; and +10. resolve the enrollment sidecar before admitting `put`. Ack paths run the + recovery barrier, so `OPEN` plus an unresolved enrollment cannot + acknowledge. + +An exact no-effect intent may finalize without publication. The exact +index-only and index-plus-empty-shard states roll forward. No other state is +silently repaired, adopted, or rolled back. Repeating enrollment with the same +complete binding is a no-op only after exact validation. A different PK, +binding, sharding spec, maintained-index set, writer-default configuration, or +current-HEAD chain is a typed conflict. No row is acknowledged until enrollment is manifest-committed, every sidecar effect is resolved, and the exact bound shard is physically admitting writes. -Initial delivery supports one unsharded shard per `(table, main)`. Non-main -branches remain refused until the Lance branch-scoping question is proven by a -surface guard and end-to-end test. Later `bucket(id, N)` sharding must preserve -the one-key-to-one-shard rule. +The exclusive base-HEAD gate and cleanup/version-GC exclusion are load-bearing +from the final pre-effect check through classification and publication. Lance's +`has_successor_version` intentionally reports only whether the immediate next +manifest exists and may return false if an intermediate manifest was removed; +therefore recovery cannot interpret `Ok(false)` as no effect after cleanup was +allowed to race. The method is public but absent from the rendered guide, so a +compile/runtime surface guard pins it on every Lance bump. + +RC.1 persists arbitrary writer defaults in the MemWAL index but does not apply +them to a caller-supplied `ShardWriterConfig`. The bounded adapter must therefore +reconstruct the exact persisted durable-write and buffer configuration before +opening the pre-minted shard; compatible defaults are not inferred. Recovery +classification refreshes the durable table tip and performs a read-only exact +inventory of the documented `_mem_wal/` layout. A foreign or malformed +prefix, loose object, unknown shard-manifest object, WAL entry, cursor movement, +or flushed generation fails closed. This inventory creates or mutates no raw +Lance object and is pinned by the Gate E0 guards on every Lance bump. + +Initial delivery supports one unsharded shard per `(table, main)` and one live +writer process for the graph. Non-main branches remain refused until the Lance +branch-scoping question is proven by a surface guard and end-to-end test. Later +`bucket(id, N)` sharding must preserve the one-key-to-one-shard rule. General +overlapping-process enrollment/failover still requires the upstream receipt / +admission lifecycle or a separately accepted distributed fence. ## 4. New public API @@ -276,35 +384,44 @@ a bound backpressures with a typed retryable response; it never drops a row. Admission is keyed by the exact `(stable_table_id, incarnation_id, enrollment_id, shard_id)` binding. Before claiming a writer, the append path captures `OPEN`, the complete physical -binding, and that shard's epoch floor. After `mem_wal_writer` claims an epoch, -it re-reads the lifecycle row and physical shard status immediately before the -first `put`: the binding must still be identical, state must still be `OPEN`, -the shard must be admitting, and the claimed epoch must exceed the recorded -floor for that same shard. A mismatch closes the writer and returns a typed -retry without appending. - -That check is paired with §8's substrate admission seal. After publishing -`DRAINING`, the drainer atomically prevents later claims and advances the epoch -of each existing shard. A writer that claimed before the seal either put first -and is included in the in-flight durability drain, or is fenced before its put; -a claimant after the seal is refused by the substrate. A check-then-put against -manifest state alone is insufficient. The public claim-seal/reopen primitive is -therefore part of §3's ship gate, not an optional optimization. +binding, `CurrentHeadWitness`, and that shard's epoch floor. After +`mem_wal_writer` claims an epoch, it re-reads the lifecycle row, physical base +HEAD, and shard status immediately before the first `put`: the binding and +witness must still be identical, state must still be `OPEN`, the shard must be +active, and the claimed epoch must exceed the recorded floor for that same +shard. A mismatch closes the writer and returns a typed retry without +appending. + +The bounded profile closes the final check-to-put race with one root-scoped +process-local admission lease. Every append holds a shared lease from its final +`OPEN`/binding/witness/epoch check through durability resolution. Drain takes +the lease exclusively, waits out existing acknowledgements, and keeps it +closed through `DRAINING` and `SEALED`. On restart, a lifecycle state other than +`OPEN` is reconstructed as closed before requests are served. This is valid +only because the support boundary excludes an overlapping writer process; the +lease is not advertised as distributed fencing. + +The general multi-process profile still requires the substrate admission seal: +after `DRAINING`, later claims must be refused across processes and existing +writers fenced before a post-check put. The exact upstream seal/reopen surface +therefore remains the expansion gate, even if Gate E0 accepts the bounded +profile. Initial topology has one active ingest owner for each `(graph, table, main)` -shard. MemWAL's epoch fence makes restart/failover safe; it is not a load -balancer. A deployment with multiple server replicas must route a shard to its -current owner (or return a typed retry/redirect) instead of letting replicas -reclaim the epoch per request. General multi-owner routing waits for the -multi-shard phase and its ownership protocol. +shard and one live writer process for the graph. MemWAL's epoch fence permits a +crash successor to replay after external exclusivity; it is not a load +balancer, a distributed OmniGraph recovery fence, or permission for two server +replicas to overlap. Multi-replica routing/failover waits for the multi-shard +phase plus an accepted ownership protocol. ## 6. Fold protocol The fold consumes flushed generations in ascending order. Embeddings and base-dependent validation run outside the table queue. With or without RFC-024, the `ReadSet` carries schema identity, the complete stream -binding/configuration/generation cut, every probed table, and the conservative -branch authority token `(native branch incarnation, optional graph_head)`. +binding/configuration/generation cut, the base table's exact +`CurrentHeadWitness`, every probed table, and the conservative branch authority +token `(native branch incarnation, optional graph_head)`. Absence of `graph_head` on a fresh branch is part of that token. Any publisher retry compares the captured token and returns to full fold revalidation on a change; it never reparents a validation-sensitive fold around a concurrent @@ -319,8 +436,9 @@ correctness dependency. The commit phase then: 4. writes one generic RFC-022 recovery sidecar before the first `commit_staged` call; 5. commits every staged Lance transaction; -6. publishes all data/internal table versions and lineage in one `__manifest` - CAS, including table heads when RFC-024 is active; +6. captures the achieved base-table witness and publishes all data/internal + table versions, the next lifecycle `CurrentHeadWitness`, and lineage in one + `__manifest` CAS, including table heads when RFC-024 is active; 7. deletes the sidecar after successful publication. The sidecar is mandatory even though merge-insert is staged. After @@ -390,27 +508,30 @@ real barrier, not an empty check. Each enrolled table has a durable `stream_state::` row in its manifest branch with `OPEN | DRAINING | SEALED`, configuration hash, an -`epoch_floor_by_shard: Map`, and the §3 physical binding. Epochs +`epoch_floor_by_shard: Map`, the §3 physical binding, and the +current base-table `CurrentHeadWitness`. Epochs are comparable only within the same enrollment and shard ID; a fresh shard in a new binding may start at 1 and is fenced from its predecessor by enrollment and shard identity, not by a larger number. The row is the logical lifecycle -authority and is updated by an RFC-022 CAS; MemWAL shard epochs and admission -status are the physical writer fence. Neither an in-memory registry nor an -empty-generation observation can substitute for both. +authority and is updated by an RFC-022 CAS. MemWAL shard epochs are the shard +writer fence; the bounded profile pairs them with the process-local admission +lease and exclusive base-HEAD ownership. The general profile additionally +requires the cross-process substrate admission seal. Neither an empty- +generation observation nor an unscoped in-memory writer registry can +substitute for the profile's complete authority/fence pair. Lifecycle-only transitions are audited manifest metadata transactions; they do not create graph-content commits or move `graph_head`. -The shared drain sequence is: - -1. publish stream intent `OPEN -> DRAINING` for the exact binding with a target - epoch floor for every current shard; -2. through the public substrate primitive required by §3, seal admission for - each shard and advance its epoch to at least that target, fencing stale - writers and refusing later claims across processes; -3. reject or backpressure new appends; §5's post-claim check rejects a claimant - that entered between steps 1 and 2, while the physical seal closes the final - check-to-put race; -4. wait for in-flight durability waiters, flush active MemTables, and fold every +The bounded-profile drain sequence is: + +1. acquire the root-scoped admission lease exclusively. This prevents a new + final check/claim and waits until every append that passed its final check + has resolved durability; +2. revalidate the exact `OPEN` binding, current-HEAD witness, and shard epoch, + then publish `OPEN -> DRAINING` with the target epoch floor; +3. keep admission exclusively closed, claim/confirm the next shard epoch to + fence a stale owner, and reject or backpressure every new append; +4. flush active MemTables and fold every generation to empty; 5. verify shard manifests and base `merged_generations` agree on emptiness; 6. publish `DRAINING -> SEALED` with the verified generation cut and exact @@ -420,32 +541,41 @@ The shared drain sequence is: `OPEN -> DRAINING` and `DRAINING -> SEALED` are RFC-022 authority-first metadata writes; each fold is a separate normal RFC-022 graph write. -`DRAINING` fully encodes every target per-shard floor, so interrupted physical -seals are idempotently resumed from that row. A `SEALED -> OPEN` transition is -different: an exact `stream_resume` sidecar covers the physical reopen and is -resolved before any ack path proceeds, so no naked admitting effect precedes -the CAS. The drain itself is not one giant sidecar spanning multiple commits. +`DRAINING` fully encodes every target per-shard floor and current-HEAD witness, +so restart reconstructs the admission gate closed and resumes the drain. A +`SEALED -> OPEN` transition is different: an exact `stream_resume` sidecar +covers the higher-epoch claim while the gate remains exclusively closed and is +resolved before any ack path proceeds. The drain itself is not one giant +sidecar spanning multiple commits. + +For the general multi-process profile, step 3 must instead include the public +cross-process seal required by §3; it atomically refuses later claims and +fences a claimant that crossed another process's lifecycle check. The +process-local sequence is not evidence for that topology. There are two dispositions after the drain reaches `SEALED`: - **operation-scoped drain** — branch/schema maintenance automatically publishes `SEALED -> OPEN` only after the guarded operation succeeds, the stream contract remains compatible, and the §3 physical binding still names the - exact table/ref. Under an exact resume sidecar, reopening the same binding + exact table/ref and the guarded operation has published the table's new + `CurrentHeadWitness` with its table pointer. Under an exact resume sidecar, + reopening the same binding advances each same-shard epoch above its recorded floor before the CAS; a - pre-CAS failure reseals those exact shards. If the operation rematerialized - the table or changed/recreated its native ref incarnation, it must complete - `stream_rebind` instead of applying this transition directly; + pre-CAS failure keeps admission closed and recovery resumes or fails closed. + If the operation rematerialized the table or changed/recreated its native + ref, it must complete `stream_rebind` instead of applying this transition + directly; - **persistent quiesce** — the public `quiesce` command leaves the stream `SEALED`. It never auto-reopens. `stream resume` explicitly revalidates schema, - PK, configuration, MemWAL format, physical binding, and every same-shard - epoch, then publishes `OPEN`. Stream teardown deletes intent only from - `SEALED`. + PK, configuration, MemWAL format, physical binding, current-HEAD witness, and + every same-shard epoch, then publishes `OPEN`. Stream teardown deletes intent + only from `SEALED`. The barrier never holds the table write queue while waiting for a fold that -needs that queue. State transition and epoch fencing happen first; fold commit -then acquires the normal queue. Crash recovery resumes from the durable state -and per-shard epoch map. +needs that queue. The separate admission gate closes first; fold commit then +acquires the normal table queue. Crash recovery resumes from the durable state, +current-HEAD witness, and per-shard epoch map. Schema apply must drain every affected enrolled type before changing fields, constraints, PK, embeddings, or `@stream` and resumes only when compatible. @@ -494,7 +624,8 @@ Fresh At query planning, `Fresh` captures one `FreshReadCut` containing: - the ordinary manifest snapshot; -- each selected table's exact lifecycle state and `StreamPhysicalBinding`; +- each selected table's exact lifecycle state, `StreamPhysicalBinding`, and + `CurrentHeadWitness`; - each selected shard-manifest version and writer epoch; - included flushed-generation paths and maximum generation; - the active same-process MemTable row-position watermark, when available; @@ -504,18 +635,20 @@ At query planning, `Fresh` captures one `FreshReadCut` containing: Capture uses a retrying handshake: 1. read each selected lifecycle row and require `OPEN`; capture its exact - physical binding, then read only that binding's shard manifests/epochs, + physical binding and current-HEAD witness, then read only that binding's + shard manifests/epochs, acquire Lance generation retention guards for the flushed files in the tentative cut, and under one same-process writer snapshot capture/pin any active-MemTable watermark; 2. pin the graph manifest snapshot and require it to select the same lifecycle - binding and physical base table/ref captured in step 1; read + binding, current-HEAD witness, and physical base table/ref captured in step + 1; read `merged_generations` from each exact base-table version it selects. A `DRAINING`, `SEALED`, or different enrollment restarts the whole capture; -3. re-read the lifecycle rows, complete physical bindings, shard manifest - versions, and per-shard epochs. Any enrollment, ref-incarnation, - configuration, state, shard-set, manifest-version, or epoch change restarts - the whole capture; +3. re-read the lifecycle rows, complete physical bindings, current-HEAD + witnesses, shard manifest versions, and per-shard epochs. Any enrollment, + witness, configuration, state, shard-set, manifest-version, or epoch change + restarts the whole capture; 4. if a generation from step 1 disappeared, accept that only when the pinned base's `merged_generations` proves it is included; otherwise release guards, discard the whole graph snapshot, and retry from step 1; @@ -561,8 +694,12 @@ physical MemWAL index and exact lifecycle row but does not change the graph format. The capability stamp is not assigned merely because RC.1 MemWAL APIs are -present. Initialization can emit a stream-capable first state only after §3's -public exact-enrollment and admission-seal gate passes; otherwise both new-root +present or because Gate E0's production-neutral harness passes. Initialization +can emit a stream-capable first state only after Gate E0 is green **and** the +bounded production enrollment/recovery, writer-exclusion, lifecycle, +refusal/rebuild, and crash gates below pass. The public exact-enrollment and +cross-process admission-seal surface remains required before expanding beyond +the bounded profile. Until the applicable profile is complete, both new-root activation and first enrollment remain disabled. Old binaries refuse a stream-capable graph before reading or writing any table. @@ -611,19 +748,87 @@ capabilities require another rebuild; co-release with RFC-023, RFC-024, RFC-025, or RFC-028 is allowed only after every participating RFC is independently accepted and their combined init, refusal, recovery, and rebuild matrix passes. -Enrollment, fold, and recovery retain RFC-022's single-writer-process support -boundary. MemWAL's shard epoch permits an explicitly routed acknowledgement -owner; it does not turn OmniGraph sidecar recovery into distributed fencing or -authorize general replica failover. +Enrollment, fold, and recovery retain RFC-022's single-live-writer-process +support boundary, strengthened for the bounded profile by exclusive base-HEAD +ownership while `OPEN`. MemWAL's shard epoch permits a crash successor after +external exclusivity; it does not turn OmniGraph sidecar recovery into +distributed fencing or authorize overlapping replica failover. ## 12. Acceptance gates +### 12.1 Gate E0 — bounded enrollment decision (green) + +Gate E0 is deliberately isolated from the production manifest schema and write +path. Its checked-in evidence suite must prove all of the following on the +pinned public RC.1 surface before the bounded profile can proceed: + +- From exact baseline HEAD `N` with no MemWAL index, initializer success yields + exactly `N + 1`, whose transaction reads `N` and contains only one singleton + `__lance_mem_wal` `CreateIndex` with the requested unsharded configuration, + namespaced enrollment ID, and configuration-version marker. The marker + is classified independently of the index UUID and survives ordinary commits + and merged-generation metadata updates. +- Discarding the initializer result and reopening produces the same exact + allowed-successor classification; an index-only crash is therefore + distinguishable from no effect without relying on the mutable Dataset + handle. +- Passing a pre-minted UUID through the public shard-writer path produces only + that shard with the expected unsharded spec, observable claimed epoch, empty + generations, and no data-bearing WAL. Any deterministic data-less fence + artifact is enumerated explicitly rather than treated as “empty enough.” +- The classifier truth table accepts only: exact no effect (finalize), the + exact index-only successor (roll forward), and that successor plus the exact + empty pre-minted shard (roll forward). Wrong configuration, an intervening + HEAD, another index transaction, a foreign shard, unexpected WAL/generation + data, a buried effect, or unreadable/ambiguous evidence yields + `RecoveryRequired`; the harness never deletes or reclaims an artifact. +- `CurrentHeadWitness` is stable across an unchanged reopen, changes after an + ordinary commit, and distinguishes same-path/same-version recreation on + local FS and S3/RustFS. Starting from a freshly verified exact-`N` handle, + the immediate `N + 1` and buried-`N + 2` successor probes are bounded in + history depth; flatness is measured, not inferred from the tuple shape. + +**Gate E0 result (2026-07-18): green for the bounded profile.** The final local +run reports 14 substantive tests plus one explicit S3 skip when no bucket is +configured. It covers the exact no-effect/index-only/index-plus-empty-shard +progression, high-level pre-minted `mem_wal_writer`, lost-result +reclassification, durable marker survival, buried-effect refusal, strict +inventory/error handling, and the complete local fail-closed matrix. + +The earlier `checkout_latest`/`IOTracker` result was discarded because local +filesystem `read_dir` bypassed that tracker. The accepted classifier never +resolves latest. It opens exact `N`, uses the doc-hidden public +`Dataset::has_successor_version` for `N + 1`, and repeats it on exact `N + 1` to +reject a buried `N + 2`. Its `AttemptTracker` records every attempt before +forwarding, including failed and `NotFound` HEADs. At baseline versions 8 and +80 the complete shape is identical: four successful manifest HEADs, one +`NotFound` manifest HEAD, one successful manifest GET, and zero `list` or +`list_with_delimiter` calls. A Unix execute-only `_versions` tripwire proves +the exact probe succeeds while latest enumeration fails; an unreadable exact +HEAD returns an error rather than false/no-effect. + +The configured RustFS exact cell passes non-vacuously. It covers no effect, +opaque initializer lost result, exact index-only state, the pre-minted empty +shard, unchanged reopen, and fail-closed foreign-shard, malformed-plus-loose- +root, durable-WAL, persisted-cursor, and corrupt-manifest states. It observes +the same six-attempt, zero-list exact-probe shape. Separate surface guards pin +`has_successor_version`, flush/drain, merged-generation state, and object-store +same-coordinate ABA; CI rejects a skipped Gate E0 or ABA cell. + +This green decision authorizes only Phase A's production foundation: exact +roll-forward enrollment recovery, central `OPEN`-table writer exclusion, +lifecycle/current-witness state, and admission-lease races. It does not change +internal schema v6, parse `@stream`, expose an API, or acknowledge a WAL row. +The general upstream receipt/seal path remains the preferred simplification and +the gate for broader topology. + +### 12.2 Production activation gates after E0 + - Surface guards pin claim, append, durability waiter, flush, per-shard epoch fencing, staged merge with `merged_generations`, and index catchup. They also - pin RC.1's blocking fact that initialization commits internally and shard - creation is separate. Acceptance requires the public exact enrollment - receipt/staging plus admission seal/reopen surface from §3; until then this - gate remains red. + pin RC.1's internal-commit initializer and separate shard effect so a Lance + bump cannot silently invalidate the E0 classifier. The public exact receipt / + seal/reopen surface remains required for the general multi-process profile. - A WAL append failure emits no durable acknowledgement. Every acknowledged row survives crash, replay, fold, and recovery. - Failpoints cover enrollment's index/shard multi-effect gap and every fold participant @@ -632,11 +837,17 @@ authorize general replica failover. the exact sidecar, returns `RecoveryRequired`, and starts no replan until the recovery barrier resolves it; the no-effect case finalizes the empty intent before its bounded full replan. -- Enrollment restarts without an inline effect when schema, PK, table head, or +- Enrollment restarts without an inline effect when schema, PK, table HEAD + witness, or stream configuration changes between prepare and gated revalidation. -- Local and S3/RustFS same-path/ref delete-recreate races change the physical-ref - incarnation and make enrollment, ack, fold, recovery, and fresh capture refuse - the replacement. A backend without a proven token refuses activation. +- Local and S3/RustFS same-path/ref delete-recreate races change the + `CurrentHeadWitness` and make enrollment, ack, fold, recovery, and fresh + capture refuse the replacement. A backend without a proven token refuses + activation. +- While a bounded-profile lifecycle is `OPEN`, every non-fold table writer and + maintenance/control path touching that table refuses before effect. After an + operation-scoped drain, any allowed table commit advances the lifecycle + witness atomically with its manifest table pointer before resume. - A future rematerializing type rename preserves the RFC-028 logical pair, leaves the target `SEALED`, and cannot acknowledge until an exact sidecar-covered rebind publishes a fresh enrollment/shard namespace. Crash tests cover every old- @@ -647,14 +858,17 @@ authorize general replica failover. - Without RFC-024, a concurrent graph commit that changes a probed-but-untouched table after fold validation changes the captured branch token, forces a full fold revalidation, and cannot be accepted by publisher reparenting. -- Two server replicas do not epoch-ping-pong one shard; owner failover fences - the old process before the new owner acknowledges. +- A second overlapping writer process is outside the bounded support contract + and cannot be credited as safe merely because epochs prevent silent dual + ownership. A crash successor acknowledges only after external exclusivity, + recovery, a higher epoch, and complete lifecycle revalidation. - Quiescence tests race appends with branch and schema operations across two coordinators and prove no post-drain tail appears; failpoints cover every - lifecycle-row, per-shard epoch-fence, admission-seal, fold, and `SEALED` + lifecycle-row, admission-lease, per-shard epoch-fence, fold, and `SEALED` boundary. A claimant paused before its final lifecycle check and one paused - after that check are both either included before the drain cut or fenced - before `put`. + after that check are both either included before the drain cut or refused + before `put`. The general profile repeats this with the cross-process + substrate admission seal. - Persistent quiesce never auto-reopens; explicit same-binding resume advances each same-shard epoch, while a rebind starts an unrelated epoch namespace. Upgrade tests cover every retained MemWAL artifact through declared @@ -684,9 +898,11 @@ authorize general replica failover. | Phase | Content | Gate | |---|---|---| -| A | graph-format capability/refusal gate, strict rebuild path, MemWAL adapter, surface guards, enrollment sidecar | public exact-enrollment plus admission-seal API gate; genuine cross-version/refusal and rebuild suite; multi-effect crash matrix | +| E0 | production-neutral public-surface enrollment/witness classifier; no schema, API, sidecar, or format activation | **Passed 2026-07-18:** 14 substantive local cells, complete six-attempt zero-list 8/80 cost shape, Unix no-list/error tripwire, and one non-vacuous configured RustFS positive-plus-negative cell (§12.1); production remains v6 | +| A | bounded main/unsharded/single-live-writer enrollment adapter, central OPEN-table exclusion, lifecycle/admission lease, then graph-format capability/refusal and strict rebuild | green E0; exact no-effect/roll-forward-only crash matrix; witness-chain and writer-refusal races; genuine cross-version/refusal and rebuild suite | | B | new ingest route/CLI, durable ack, strict fold | ack durability; API compatibility; cost budget | | C | atomic dead letter, audit provenance, status/bounds | reject crash matrix; backpressure tests | | D | epoch-fenced drain, persistent quiesce/resume, schema/branch/upgrade integration, rematerialization rebind | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | | E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | | F | multi-shard upsert and stream deletes | one-key-one-shard proof; Lance re-audit | +| G | overlapping-process ownership/failover | public exact enrollment receipt plus cross-process seal/reopen, or a separately accepted distributed fence; adversarial multi-process recovery evidence |