Core: SCALAR Key Lookup Index — reference implementation of Secondary Index Spec - #17426
Draft
osscm wants to merge 5 commits into
Draft
Core: SCALAR Key Lookup Index — reference implementation of Secondary Index Spec#17426osscm wants to merge 5 commits into
osscm wants to merge 5 commits into
Conversation
Applies the index spec document from apache/iceberg PR apache#16961 authored by pvary. This defines the generic secondary index framework: SCALAR index type, HASH/IDENTITY transforms, Index Metadata, Tracking File, and Leaf File structure.
Implements IndexMetadata and IndexSnapshot POJOs matching the field definitions in format/index.md (PR apache#16961). Includes JSON parsers and a unit test suite covering round-trip serialization, snapshot lookup helpers, builder validation, and spec field name verification. 6/6 unit tests passing.
Adds Layer 1 of the SCALAR index implementation:
- IndexIdentifier: identifies an index by table + name
- IndexCatalog: interface with create/load/update/drop/list operations
with optimistic concurrency on updateIndex
- IndexMetadataIO: read/write index metadata JSON via FileIO, with
metadata file location naming (00001-{uuid}.metadata.json)
- InMemoryIndexCatalog: thread-safe in-memory implementation for tests,
includes CAS-based conflict detection on updateIndex
14/14 unit tests passing.
Implements the tracking file format for the secondary index spec: - TrackingFileEntry: value class holding leaf file location, bounds, record count, size, and optional key metadata (field IDs from spec) - TrackingFileWriter: writes entries as an Avro DataFile using Snappy compression, writing to Iceberg OutputFile (same pattern as manifests) - TrackingFileReader: readAll() and readMatching(min, max) for planning-time pruning — returns only leaf files whose transform-value range overlaps the query range 18/18 unit tests passing.
Implements the core build logic for the SCALAR index: - HashTransform: maps key values to hash buckets [0, numBuckets). Phase 1 uses Java hashCode; will align with Iceberg murmur3 bucket transform in a follow-up. - LeafFileMetadata: captures path, bounds, record count, and size of one written leaf file; converts to TrackingFileEntry for the tracking file writer. - ScalarIndexCommitter: given a list of leaf files from the Spark build job, writes the tracking file (Avro), creates an IndexSnapshot, writes the metadata JSON, and commits to the IndexCatalog with CAS semantics. Also adds scripts/build_scalar_index_taxi.scala — an end-to-end Spark script demonstrating the full build flow on NYC Yellow Taxi data using the medallion column as the SCALAR HASH index key. 25/25 unit tests passing.
github-actions
Bot
added
API
core
Specification
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference implementation of the SCALAR index type defined in apache/iceberg PR #16961 (Secondary Index Specification, Peter Vary).
PR #16961 defines the SCALAR index type, HASH and IDENTITY transforms, Index Metadata JSON, Tracking File (Avro), and Leaf File (Parquet) formats but does not yet include a working implementation. This PR provides the first end-to-end implementation.
Layers implemented:
25/25 unit tests passing.
Builds on: #16961