Evaluate retention and MemWAL enrollment gates

[aaltshuler]

Summary

• Revalidate the RFC-024 durable-head lookup candidate on Lance RC.1 and preserve the research no-go: indexed row/range work is flat, but complete physical authority lookup remains history-sensitive.
• Add RFC-025 Gate 0 evidence for checkpoint retention, including tag/cleanup semantics, physical-ref ABA guards, retention claims, and list/show/cleanup-root cost instruments. The current in-manifest BTREE registry is rejected because compacted scan bytes and operations grow with history.
• Add RFC-026 Gate E0 exact MemWAL enrollment classification using pinned N -> N+1 -> N+2 successor probes, strict shard-state validation, failed-attempt tracking, and local/RustFS negative matrices.
• Update the RFCs and write-path state-of-affairs documentation with the decisions and next steps.

Decision

RFC-024 and RFC-025 remain research-blocked on their current access shapes.

RFC-026 Gate E0 passes only for the bounded main-branch, unsharded, single-live-writer-process profile. This clears Phase A foundation work from the upstream release calendar, while an upstream receipt/seal API remains required for broader overlapping-process topology.

This PR is production-neutral: internal schema stays v6, and it adds no durable-head/checkpoint/stream rows, public APIs, stream acknowledgements, folds, or production writer path.

Validation

• cargo test --workspace --locked
• RFC-024 and RFC-025 local decision matrices
• RFC-026 Gate E0: 14 substantive local cells plus the explicit unconfigured-S3 cell
• Configured RustFS MemWAL positive/negative matrix
• Configured RustFS same-version ABA guard
• scripts/check-agents-md.sh
• git diff --check

Greptile Summary

This PR adds research gates for checkpoint retention and MemWAL enrollment. The main changes are:

• Exact MemWAL successor and shard-state classification tests.
• Checkpoint authority cost and native tag-retention tests.
• RustFS coverage for the MemWAL and physical-reference gates.
• Updated RFC decisions and write-path documentation.

Confidence Score: 4/5

The checkpoint-retention S3 gate needs CI wiring before merging.

• The MemWAL and physical-reference S3 tests reject skipped and unmatched runs.
• The new checkpoint-retention S3 test is not selected or invoked by the RustFS workflow.
• The remaining changes are production-neutral tests and documentation.

.github/workflows/ci.yml

Important Files Changed

Filename	Overview
.github/workflows/ci.yml	Adds non-vacuous RustFS checks for MemWAL and physical references but omits the checkpoint-retention S3 gate.
crates/omnigraph/tests/memwal_enrollment_gate.rs	Adds exact successor, enrollment-state, shard-state, failure, and S3 classification tests.
crates/omnigraph/tests/checkpoint_retention_cost.rs	Adds checkpoint authority cost curves and local/S3 decision matrices.
crates/omnigraph/tests/helpers/cost.rs	Adds object-store attempt tracking for exact-key probe measurements.
crates/omnigraph/tests/lance_surface_guards.rs	Expands guards for MemWAL APIs, current-HEAD witnesses, tags, branch deletion, and cleanup.
docs/rfcs/0025-checkpoint-retention.md	Records the checkpoint access-shape evidence and current no-go decision.
docs/rfcs/0026-memwal-streaming-ingest.md	Records the bounded MemWAL enrollment result and remaining activation requirements.

_{Reviews (1): Last reviewed commit: "Evaluate retention and MemWAL enrollment..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)

[greptile-apps]

Retention S3 Gate Never Runs

A pull request that changes only checkpoint_retention_cost.rs does not enable the RustFS job, and that job has no command for s3_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go even when another file enables it. The S3 test then skips outside a configured RustFS environment, so broken or vacuous retention evidence can merge without being exercised.

Context Used: AGENTS.md (source)