Skip to content

feat: add local SpillStore for index scratch#7369

Draft
RitwijParmar wants to merge 1 commit into
lance-format:mainfrom
RitwijParmar:codex/lance-spill-store
Draft

feat: add local SpillStore for index scratch#7369
RitwijParmar wants to merge 1 commit into
lance-format:mainfrom
RitwijParmar:codex/lance-spill-store

Conversation

@RitwijParmar

Copy link
Copy Markdown

Summary

This adds the generic local spill mechanism described in #7300 so future memory-budgeted index builders have a session-scoped place for reclaimable scratch files.

Changes include:

  • new SpillStore / SpillFile traits and LocalSpillStore implementation in lance-io
  • shared DiskQuota accounting wired into local writers for clean cap-exceeded errors
  • RAII cleanup for spill files, including quota release on drop
  • ScanScheduler::open_reader for already-open readers
  • Session wiring with a default local spill store and override hook

This intentionally does not migrate IVF consumers yet; that remains the follow-up slice from #7300.

Validation

  • cargo fmt --all
  • cargo test -p lance-io spill --lib
  • cargo check -p lance-io

I also tried cargo check -p lance, but the local environment is missing protoc, so lance-encoding build script stops before the top-level crate can finish.

Refs #7300

@github-actions github-actions Bot added the A-encoding Encoding, IO, file reader/writer label Jun 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@RitwijParmar RitwijParmar changed the title Add local SpillStore for index scratch feat: add local SpillStore for index scratch Jun 19, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant