Add a portable SIMD block index for forward overlap queries - #27
Draft
sstadick wants to merge 22 commits into
Draft
Add a portable SIMD block index for forward overlap queries#27sstadick wants to merge 22 commits into
sstadick wants to merge 22 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Note On AI 🤖
I used agents extensively to benchmark different possible implementations and try out different indexing methods etc. The two key ideas were, surprisingly, my own, using the block-based index and inter-block simd comparisons.
I'm still conflicted about having agents do any of this, as this is not a library that has been touched by AI previously, and I don't know how dependents on this lib feel about agentically written code. If you have strong feelings and read this, open an issue and lets discuss it.
Summary
This replaces the original
max_len-bounded forward scan with an always-on32-interval block index while preserving rust-lapper's existing borrowed,
start-ordered iterator identity.
Blocks that are certainly before a query are skipped using exact maximum-end
summaries and forward next-greater links. Dense active prefixes are returned
directly, while mixed blocks are classified with an exact 32-bit overlap mask.
The mask backend uses AArch64 NEON, runtime-dispatched x86-64 AVX2, or an exact
scalar fallback.
Compatibility
kernels where supported.
PrimIntimplementations use the scalar mask.index.
The branch adds an
I: 'staticbound for safeTypeId-based primitivedispatch. Stable Rust cannot specialize primitive SIMD kernels while retaining
a blanket custom-
PrimIntfallback, so the bound is accepted and documented.It excludes lifetime-carrying coordinate types, not short-lived
Lappervalues.
Performance
Apple M3/AArch64 medians versus rust-lapper 1.3.0:
1-27-38-7All implementations returned identical overlap counts. Baseline and final
values came from separate controlled runs, so they are orientation numbers
rather than paired percentage estimates.
Validation
cargo fmt --all -- --checkcargo test --all-featurescargo clippy --all-targets --all-features -- -D warningsNative Intel/AMD AVX2 performance validation remains outstanding.
Production gates
The exact five-gate checklist is recorded in
plans/productionization.md:physical non-AVX2 host remain.
I: 'staticbound is accepted.workloads and was rejected.