Skip to content

feat(mem-wal): add ShardWriter::put_no_wait#7362

Open
hamersaw wants to merge 2 commits into
lance-format:mainfrom
hamersaw:feature/wal-partial-column-update
Open

feat(mem-wal): add ShardWriter::put_no_wait#7362
hamersaw wants to merge 2 commits into
lance-format:mainfrom
hamersaw:feature/wal-partial-column-update

Conversation

@hamersaw

Copy link
Copy Markdown
Contributor

What

Adds ShardWriter::put_no_wait, a variant of put that performs the visible in-memory insert and triggers the durable WAL flush, then returns the BatchDurableWatcher without awaiting it. A thin wrapper restores put's behavior (await the watcher), so existing callers are unchanged.

put_memtable is split into:

  • put_memtable_no_wait — the in-memory critical section (insert under state_lock + track_batch_for_wal + flush triggers) followed by trigger_flush, returning (WriteResult, Option<BatchDurableWatcher>).
  • put_memtable — calls the above, then watcher.wait().

BatchDurableWatcher and WriteResult are re-exported from mem_wal.

Why

Lets an external caller hold its own serialization lock across only the in-memory read-merge-insert critical section and await durability after releasing it, so concurrent durable flushes still coalesce. The in-memory insert stays guarded by the writer's state_lock, so BatchStore's single-writer invariant holds regardless of the external lock — state_lock is intentionally not skipped.

This is the lance-side primitive for sophon's WAL partial-column-update path (read fresh tier → merge → insert under a per-bucket lock, durability awaited outside it).

Tests

test_put_no_wait_durable_visible_then_durable (row visible before durability, watcher resolves) and test_put_no_wait_non_durable_returns_no_watcher.

Split put_memtable into a no-wait core that performs the visible in-memory
insert and triggers the durable flush, returning the BatchDurableWatcher
un-awaited, plus a thin wrapper that awaits it (preserving put's behavior).
This lets an external caller serialize a read-merge-insert critical section
under its own lock and await durability after releasing it, so concurrent
durable flushes still coalesce. The in-memory insert stays guarded by the
writer's state_lock, so BatchStore's single-writer invariant holds without
relying on the external lock. Re-exports BatchDurableWatcher and WriteResult.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the enhancement New feature or request label Jun 18, 2026
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.67442% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/dataset/mem_wal/write.rs 97.67% 0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant