docs: cover v0.4.0 — atomic cross-shard transactions, CRDT collections, sparse vectors

docs/architecture.md

@@ -142,17 +142,29 @@ Each kind has independent leader election. A sequencer leader failure does not a
 
 Write transactions that touch multiple vShards go through the **Calvin sequencer** rather than two-phase commit. The sequencer Raft group produces a globally-ordered log of transaction batches (epochs, default 20 ms). Within each epoch, the scheduler derives a deterministic lock order and the executor runs all writes concurrently without cross-shard coordination.
 
+**Interactive transactions are atomic across shards.** A `BEGIN ... COMMIT` block whose statements span multiple vShards (or nodes) flushes at COMMIT as one Calvin transaction bound by a durable **Vote/Verdict barrier**: each participant replicates a commit/abort vote through the sequencer log; when the tally completes, a replicated verdict commits or drops every participant's staged writes together. Expected-participant counts are seeded from the replicated epoch batch, so the barrier survives sequencer failover. Read-only participants count too — a transaction that wrote shard X but read shard Y validates and votes on both.
+
+**Serializable OCC.** Reads recorded during the transaction (point reads, predicate scans, index equality/range probes, both sides of gathered cross-node JOINs) are validated at COMMIT against per-key, per-collection, and per-index-value write LSNs. A stale read aborts the transaction with SQLSTATE `40001` (retryable) instead of committing silently.
+
+**Durability.** A committed transaction's writes are persisted as a single atomically-replayable `TransactionRedo` WAL record appended before the commit is acknowledged. WAL-only recovery replays it in LSN order, rebuilding even in-memory secondary indexes (vector HNSW, FTS postings) that base storage cannot reconstruct alone.
+
 For value-dependent predicates (e.g., `WHERE balance > 0`), the executor uses **OLLP** (Optimistic Lock Location Prediction): optimistically proceed, then re-validate and retry on mismatch. A circuit breaker opens when the retry ratio exceeds 50% for a predicate class.
 
 ```sql
 -- Require atomic cross-shard writes (default)
 SET cross_shard_txn = 'strict';
 
--- Opt out of atomicity for bulk loads (each shard commits independently)
+-- Opt out of atomicity for bulk loads: writes are grouped per vShard and each
+-- group commits as an independent single-vShard transaction; a failure does NOT
+-- roll back vShards that already committed
 SET cross_shard_txn = 'best_effort_non_atomic';
 ```
 
-Single-shard writes bypass the sequencer entirely and go directly through the relevant data-group Raft.
+(Bare `best_effort` is deliberately rejected; invalid values return SQLSTATE `22023`.)
+
+**Single-node deployments run Calvin by default** (`[server] single_node_calvin = true`): a standalone node synthesizes a one-node sequencer group so transactions spanning multiple cores (vShards) commit atomically instead of being rejected. Set it `false` to force the legacy fast path. Uncontended single-shard point writes bypass the sequencer entirely and go directly through the relevant data-group Raft; contended or predicate/bulk writes route through the deterministic scheduler.
+
+**Overlay hygiene.** Per-transaction staging overlays are kept alive by every staged write/read; overlays orphaned by vanished clients are reaped after a 6-hour lease. The `nodedb_active_txn_overlays` Prometheus gauge tracks live overlays. Data-Plane resource rejection surfaces as SQLSTATE `53200` (backpressure — retry when pressure subsides).
 
 ## Distributed Query Execution
 

docs/bitemporal.md

@@ -164,6 +164,8 @@ ORDER BY _ts_system ASC;
 
 `AS OF SYSTEM TIME NULL` returns every system-time version of each matching row, ordered ascending. Each version row carries the user columns plus three synthetic temporal columns: `_ts_system`, `_ts_valid_from`, and `_ts_valid_until`. Unbounded valid-time ends surface as the raw `i64::MIN` / `i64::MAX` sentinels (matching the Columnar and Timeseries engines). It is supported on the Document (strict and schemaless), Columnar, and Timeseries engines. It is not supported on the Graph or Array engines, nor when reading through a database clone — those return a typed error rather than collapsing to a single version.
 
+`AS OF` reads run at parity with non-temporal reads: `WHERE` predicates (including on strict Binary-Tuple collections), `ORDER BY`, computed columns, and window functions all apply to temporal queries.
+
 **Reserved columns never leak.** Bitemporal collections store their temporal envelope in reserved columns (`__system_from_ms`, `__valid_from_ms`, `__valid_until_ms`). These are hidden from user projections: `SELECT *` on a bitemporal collection returns exactly the declared user columns. Temporal data is only exposed through the synthetic `_ts_*` columns of audit queries.
 
 ## Index Engines and Temporal Composition

docs/databases.md

@@ -337,6 +337,8 @@ ALTER DATABASE analytics SET IDLE_TIMEOUT 0;     -- disabled
 
 Sessions idle longer than the configured time are automatically closed with `SESSION_IDLE_TIMEOUT`. Tracks are monitored per-database; timeout is checked at request boundaries.
 
+Separately, a process-global pgwire listener watchdog force-closes idle connections after `auth.idle_timeout_secs` (default 3600s; `auth.session_absolute_timeout_secs` caps total session age, default disabled). The watchdog never kills a connection mid-statement — the idle window starts when a statement completes — and its teardown reclaims any idle-in-transaction staging overlay. The per-database `SET IDLE_TIMEOUT` and the global watchdog are independent mechanisms; the stricter one wins.
+
 ### View and Kill Sessions
 
 ```sql

docs/documents.md

@@ -137,6 +137,43 @@ FROM orders
 GROUP BY status;
 ```
 
+## CRDT Document Collections
+
+Declare a document collection CRDT-backed at creation time and plain SQL DML converges via last-writer-wins instead of overwriting:
+
+```sql
+CREATE COLLECTION crdt_notes (
+    id TEXT PRIMARY KEY,
+    title TEXT,
+    body TEXT
+) WITH (crdt=true);
+
+-- Full-replace write (untouched keys pruned)
+INSERT INTO crdt_notes (id, title, body) VALUES ('a', 'v1', 'text');
+UPSERT INTO crdt_notes (id, title) VALUES ('a', 't2');
+
+-- PK-targeted UPDATE is a per-field LWW merge: only the provided
+-- fields are written; untouched fields survive concurrent writers
+UPDATE crdt_notes SET title = 't3' WHERE id = 'a';
+
+-- PK-targeted DELETE writes a tombstone
+DELETE FROM crdt_notes WHERE id = 'a';
+
+-- RETURNING works on CRDT UPDATE/DELETE
+UPDATE crdt_notes SET title = 't4' WHERE id = 'a' RETURNING id, title;
+DELETE FROM crdt_notes WHERE id = 'a' RETURNING id;
+```
+
+`crdt=true` is only valid on document collections. DML on a CRDT collection routes through the CRDT engine — there is no silent fallthrough to the plain document path (that would bypass convergence). Statement shapes that cannot be expressed as a CRDT operation are rejected with a typed error rather than downgraded:
+
+- Predicate (non-primary-key) `UPDATE` / `DELETE`
+- `UPDATE` with a non-literal right-hand side (`SET count = count + 1`)
+- `INSERT ... ON CONFLICT DO UPDATE`
+
+The `crdt` flag is part of the collection descriptor: it replicates across the cluster and travels in sync `CollectionSchema` announcements, so Lite/WASM peers see the same convergence semantics.
+
+**Movable lists (SDK).** The client SDK exposes CRDT movable-list operations on documents: `list_insert(collection, doc_id, list_path, index, fields)`, `list_delete(...)`, and `list_move(collection, doc_id, list_path, from_index, to_index)` — dispatched as native opcodes and merged conflict-free across devices.
+
 ## Choosing Between Modes
 
 |               | Schemaless                            | Strict                               |

docs/offline-sync-patterns.md

@@ -117,7 +117,7 @@ Columnar, Vector, FTS, and Spatial collections now participate in **inbound sync
 
 **Schema announce.** A sync peer announces each collection's schema (`CollectionSchema` message) before sending any shape or delta data. Collections unknown to the receiver are materialized into its catalog and propagate cluster-wide via Raft — create-only, so an existing collection is never clobbered. A Lite client can therefore bootstrap collections on Origin simply by announcing their schema.
 
-**Constraint validation and rejects.** UNIQUE, FK, required-field, and CHECK constraints are validated when deltas apply on Origin. Because CRDT deltas are commutative and already merged on import, enforcement is a post-hoc pass: a violating row is quarantined and the client receives a `DeltaReject` carrying a typed `CompensationHint` (e.g., `UniqueViolation { field, conflicting_value }`, `RetryWithDifferentValue`, `ManualIntervention`) telling it precisely what to fix. A rejected delta does not wedge the stream — later deltas continue to apply.
+**Constraint validation and rejects.** UNIQUE, FK, required-field, and CHECK constraints are validated when deltas apply on Origin. Because CRDT deltas are commutative and already merged on import, enforcement is a post-hoc pass: a violating row is quarantined and the client receives a `DeltaReject` carrying a typed `CompensationHint` (e.g., `UniqueViolation { field, conflicting_value }`, `RetryWithDifferentValue`, `ManualIntervention`) telling it precisely what to fix. A rejected delta does not wedge the stream — later deltas continue to apply. A delta must also carry exactly **one document** — cross-engine identity binds one surrogate per delta — so a delta writing rows outside its frame target is rejected with `RejectedConstraint { constraint: "crdt_single_document_delta" }`.
 
 **Durability.** Acknowledged sync writes are quorum-durable: the delta commits through the data group's Raft log before the ack, so it survives leader failover. Producers carry `(producer_id, epoch, seq)` provenance; replays are acknowledged as duplicates rather than double-applied.
 

docs/query-language.md

@@ -49,6 +49,8 @@ SELECT * FROM users WHERE role IN ('admin', 'editor');
 SELECT * FROM users WHERE deleted_at IS NULL;
 ```
 
+Timestamp-valued strings compare by instant, not bytes: a stored ISO-8601 value (`'2026-07-02T13:00:00.000000Z'`) matches a SQL-style literal (`'2026-07-02 13:00:00'`) in `=` and range predicates when both sides parse as datetimes. Non-datetime strings keep lexicographic comparison.
+
 ### Aggregates
 
 ```sql
@@ -722,6 +724,21 @@ WHERE category = 'machine-learning'
   );
 ```
 
+### Sparse Vector Search
+
+```sql
+-- Dimensionless sparse-vector column; values are '{dimension: weight}' literals
+CREATE TABLE sparse_docs (id TEXT PRIMARY KEY, terms SPARSEVECTOR);
+INSERT INTO sparse_docs (id, terms) VALUES ('a', '{3: 1.0, 7: 1.0}');
+
+-- Dot-product top-k ranking; LIMIT sets k
+SELECT id FROM sparse_docs
+ORDER BY sparse_score(terms, '{3: 1.0, 7: 0.5}') DESC
+LIMIT 3;
+```
+
+The inverted index is maintained automatically on document writes (insert, update, delete). Documents sharing no dimension with the query are never scanned. `sparse_score` is a standalone surface — it is not fused into `rrf_score` hybrid ranking.
+
 ### Full-Text Search
 
 ```sql
@@ -856,13 +873,18 @@ SELECT * FROM events WHERE doc_array_contains(payload, '$.tags', 'important');
 ### CRDT
 
 ```sql
--- Read CRDT state
-SELECT crdt_state('collab_docs', 'doc123');
+-- Declare a document collection CRDT-backed: plain DML converges via LWW
+CREATE COLLECTION notes (id TEXT PRIMARY KEY, title TEXT, body TEXT) WITH (crdt=true);
+UPDATE notes SET title = 't2' WHERE id = 'a';   -- per-field LWW merge
+DELETE FROM notes WHERE id = 'a' RETURNING id;  -- tombstone
 
--- Apply delta
+-- Low-level delta path
+SELECT crdt_state('collab_docs', 'doc123');
 SELECT crdt_apply('collab_docs', 'doc123', '<delta_bytes>');
 ```
 
+`WITH (crdt=true)` is document-collection-only. Non-PK-targeted UPDATE/DELETE, non-literal SET values, and `ON CONFLICT DO UPDATE` are rejected on CRDT collections (no representable CRDT operation). See [Documents — CRDT Document Collections](documents.md#crdt-document-collections).
+
 ### Array (NDArray)
 
 ```sql
@@ -967,6 +989,10 @@ COMMIT;
 
 Isolation level: **Snapshot Isolation (SI)**. Reads see a consistent snapshot from `BEGIN` time. Write conflicts detected at `COMMIT` and surfaced as SQLSTATE `40001` (`could not serialize access due to concurrent update`).
 
+### Cross-Shard Transactions
+
+An interactive `BEGIN ... COMMIT` block whose statements span multiple vShards or nodes commits **atomically** by default — the whole block flushes through the Calvin sequencer's durable vote/verdict barrier at COMMIT. Reads taken during the transaction (point reads, predicate scans, index probes, both sides of distributed JOINs) are OCC-validated at COMMIT; a stale read aborts with `40001` (retry the transaction). `SET cross_shard_txn = 'best_effort_non_atomic'` opts bulk loads out of cross-shard atomicity. Single-node deployments run the same path by default (`single_node_calvin = true`), so transactions spanning cores commit atomically too. See [Architecture — Cross-Shard Transactions](architecture.md#cross-shard-transactions).
+
 ### Read-Your-Own-Writes
 
 Statements inside `BEGIN ... COMMIT` are staged in a per-transaction overlay on the Data Plane. Reads within the same transaction observe staged writes **across every engine**: KV, document (schemaless and strict), columnar, timeseries, graph (single-hop, multi-hop, shortest path), vector search, spatial predicates, full-text search, and hybrid search fusion. Constraint violations (e.g., primary-key uniqueness) surface immediately at statement time, not at COMMIT.

docs/real-time.md

@@ -71,6 +71,13 @@ SHOW TRIGGERS;
 | `SYNC`            | Same transaction (ACID)   | Trigger time added | Yes                 |
 | `DEFERRED`        | Same transaction, batched | At COMMIT time     | Yes                 |
 
+**Graph mutations emit events.** Graph edge and node-label writes publish `WriteEvent`s like any other mutation, so triggers, change streams, and streaming MVs can react to graph changes:
+
+- **Edges** — published on the edge's collection with a stable `row_id` composed from the `(src, label, dst)` triple. Insert/put carries the edge properties in `new_value`; delete carries no payload.
+- **Node labels** — published tenant-wide on the dedicated stream `__graph_node_labels__`. `SET` labels → `Insert` with the added labels in `new_value` (`{ "labels": [...] }`); `REMOVE` → `Delete` with the removed labels in `old_value`. `row_id` is the node id.
+
+Implicit edges (a document insert carrying `_from`/`_to`) are not double-published — the underlying document write already emits its event. WAL replay reconstructs graph events byte-identically and dedups on LSN.
+
 **UPSERT and `ON CONFLICT` firing semantics.** The `WriteOp` tag emitted to the Event Plane is derived from storage prior-bytes, not from the surface SQL verb. An `UPSERT` or `INSERT ... ON CONFLICT (pk) DO UPDATE` that finds an existing row fires `AFTER UPDATE`; the same statement against a non-existent key fires `AFTER INSERT`. `ON CONFLICT DO NOTHING` on a conflict emits no event at all.
 
 ## CDC Change Streams

docs/vectors.md

@@ -78,6 +78,24 @@ LIMIT 10;
 
 Set `target_recall` and let the planner do the rest unless you need a hard latency ceiling.
 
+## Sparse Vectors
+
+For learned sparse representations (SPLADE, uniCOIL, BM25-style term weights), declare a dimensionless `SPARSEVECTOR` column. An inverted index over the non-zero dimensions is maintained automatically on every document write.
+
+```sql
+CREATE TABLE sparse_docs (id TEXT PRIMARY KEY, terms SPARSEVECTOR);
+
+-- Values are '{dimension: weight}' string literals
+INSERT INTO sparse_docs (id, terms) VALUES ('a', '{3: 1.0, 7: 1.0}');
+
+-- Dot-product top-k; LIMIT sets k
+SELECT id FROM sparse_docs
+ORDER BY sparse_score(terms, '{3: 1.0, 7: 0.5}') DESC
+LIMIT 3;
+```
+
+`SPARSEVECTOR` carries no fixed dimensionality — only non-zero `(dimension, weight)` pairs are stored and indexed. Documents sharing no dimension with the query are excluded by the inverted index rather than scored at zero. `sparse_score` ranks by descending dot product; it is a standalone surface, separate from the dense `vector_distance` path and the RRF hybrid fusion.
+
 ## Vector-Primary Collections
 
 By default, vectors are an _index_ attached to a column on a normal collection — the document store is the source of truth. For pure-vector workloads (RAG corpora, recommendation memory, embedding stores) flip a collection into vector-primary mode where the vector index is the primary access path and the document store is a metadata sidecar: