Skip to content

feat: support core Lumina index build#347

Open
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:lumina-core-build
Open

feat: support core Lumina index build#347
QuakeWang wants to merge 1 commit into
apache:mainfrom
QuakeWang:lumina-core-build

Conversation

@QuakeWang
Copy link
Copy Markdown
Contributor

Purpose

Rust already supports reading Lumina global indexes, but it did not have a core path to build Lumina index files from table data and commit them into the index manifest. Index-only commits were also skipped, so the generated index metadata could not create a new snapshot.

Brief change log

  • Add LuminaIndexBuildBuilder on Table.
  • Plan Lumina shards by global row id and global-index.row-count-per-shard.
  • Extract ARRAY<FLOAT> vectors with strict _ROW_ID continuity validation.
  • Build Lumina index files under index/lumina-global-index-*.index.
  • Commit Lumina IndexFileMeta with canonical index_type = "lumina" and global row range metadata.
  • Support metadata-only index commits and reject overlapping global index ranges.
  • Fail fast for unsupported primary-key and deletion-vector tables.

Tests

API and Format

Documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant