bf_tree migration away from diskann-providers

[JordanMaples]

Migrate bf_tree provider to dedicated crate and modernize

Motivation

The long-term goal is to remove diskann-providers entirely. The bf_tree provider was one of several components keeping that crate alive — it depended on diskann-providers for generic delete infrastructure (DeletionCheck/AsDeletionCheck/RemoveDeletedIdsAndCopy) and PQ quantization types. This PR extracts bf_tree into a standalone diskann-bf_tree crate, cuts those dependencies, simplifies the generics, and switches from PQ to spherical quantization — all to reduce the surface area of diskann-providers and move closer to removing it.

What Changed

1. Crate Extraction (bf7d33fd)

Moved the bf_tree provider from diskann-providers/src/model/graph/provider/async_/bf_tree/ into a standalone diskann-bf_tree crate. Updated workspace Cargo.toml and CI configuration.

2. PQ → Spherical Quantization (74149bbc)

• QuantVectorProvider: Stores Arc<Poly<dyn Quantizer>> (spherical) instead of Arc<FixedChunkPQTable>.
• QuantQueryComputer: Newtype wrapper adapting spherical's Opaque-based API to PreprocessedDistanceFunction<&[u8], f32>.
• Serialization: Uses Quantizer::serialize / iface::try_deserialize (flatbuffers format).
• CreateQuantProvider: Now implemented for Poly<dyn Quantizer> instead of FixedChunkPQTable.

3. Remove D (Delete Provider) Parameter (14f504e8)

• BfTreeProvider<T, Q, D> → BfTreeProvider<T, Q>.
• Hard deletes via bf_tree's native delete(key) API replace bitmap-based soft deletes.
• delete() removes from all three trees (neighbors, full_vectors, quant_vectors).
• status_by_internal_id() checks vector presence (Ok = valid, Err = deleted).
• Removed delete_bitmap_serde.rs, NoDeletes/TableBasedDeletes from constructors.

4. Simplify to Core Strategies, Then Re-add (02d355fb → c598d71d)

First removed the old Hybrid/QuantAccessor/hybrid_computer code to start fresh, then re-added clean implementations:

• QuantAccessor (e1ac1011): New accessor that reads quantized vectors from the bf_tree and uses QuantQueryComputer for distance. Implements all Quantized strategy traits.
• Hybrid strategies with Rerank (c598d71d): Re-introduced Hybrid as a struct with max_fp_vecs_per_prune: Option<usize>. Rerank post-processor re-ranks quantized search results using full-precision vectors. Implements SearchPostProcess, SearchPreprocess, InsertPreprocess, and ExpandBeam for Hybrid.
• Removed criterion/iai benchmarks (benches/ directory).

5. Use CopyIds Instead of RemoveDeletedIdsAndCopy (8a3b6742)

Since bf_tree uses hard deletes, deleted IDs never appear in neighbor lists. The AsDeletionCheck / DeletionCheck / RemoveDeletedIdsAndCopy machinery is unnecessary. Replaced with glue::CopyIds and removed the AsDeletionCheck impl from FullAccessor.

6. Move DeletionCheck Traits Back to diskann-providers (38f1e614)

These traits were temporarily moved into diskann/src/graph/glue.rs during the extraction as a stop-gap. Now that bf_tree no longer depends on them, they are restored to their original location in diskann-providers/src/model/graph/provider/async_/postprocess.rs with pub(crate) visibility.

7. Scope diskann-providers API Back to pub(crate) (ee595e2c)

Reverted methods on TableDeleteProviderAsync (is_deleted, delete, undelete, clear) from pub back to pub(crate). Removed unused RemoveDeletedIdsAndCopy re-export.

8. Tests and Delete-Panic Fix (cbea923f)

• Bug fix: FullAccessor::on_elements_unordered and QuantAccessor::on_elements_unordered now silently skip deleted entries instead of propagating hard errors. bf_tree does hard deletes but graph edges still reference deleted nodes — when search traverses those edges, get_vector_into returns an ANNError (always-escalate), which panicked in expand_beam. The fix skips entries that fail to read.
• 7 new tests: test_quantized_index_search, test_quantized_index_multi_insert_search, test_quantized_delete_and_search, test_hybrid_index_search, test_hybrid_delete_and_search, test_full_precision_index_search, test_full_precision_delete_and_search.
• Added doc comments for OwnedOpaque and indexing strategy descriptions on BfTreeProvider.

⚠️ Breaking Changes

• Serialization format: Persisted indexes using the PQ format will not load. Full reindex required.
• Training API: Callers must provide Poly<dyn Quantizer> instead of FixedChunkPQTable.
• Constructor signatures: D parameter removed, no NoDeletes/TableBasedDeletes needed.
• Delete semantics: Deletes are immediate and permanent (hard delete from bf_tree).

Net Impact

• +230 net lines (+1461 / -1231) across the workspace
• diskann-bf_tree is now a self-contained crate with two generic parameters (T, Q) instead of three
• Three indexing strategies: FullPrecision, Quantized, Hybrid (with Rerank post-processor)
• No dependency on DeletionCheck / AsDeletionCheck / RemoveDeletedIdsAndCopy
• 7 new integration-style tests covering all three strategies (insert, search, delete)
• diskann-providers API surface reduced (methods scoped back to pub(crate))

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.52%. Comparing base (c7dfae6) to head (5877d59).

Additional details and impacted files

@@            Coverage Diff            @@
##             main    #1020     +/-   ##
=========================================
  Coverage   89.51%   89.52%             
=========================================
  Files         461      476     +15     
  Lines       85920    90193   +4273     
=========================================
+ Hits        76912    80746   +3834     
- Misses       9008     9447    +439     

Flag	Coverage Δ
miri	89.52% <100.00%> (+<0.01%)	⬆️
unittests	89.17% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-bf_tree/src/provider/bf_tree_provider.rs	91.51% <ø> (ø)
diskann-bf_tree/src/provider/mod.rs	0.00% <ø> (ø)
diskann-bf_tree/src/provider/neighbor_provider.rs	92.21% <ø> (ø)
...kann-bf_tree/src/provider/quant_vector_provider.rs	92.67% <100.00%> (ø)
diskann-bf_tree/src/provider/vector_provider.rs	93.44% <100.00%> (ø)
...del/graph/provider/async_/table_delete_provider.rs	97.41% <ø> (ø)

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Migrates the async bf_tree provider away from diskann-providers by extracting it into a dedicated diskann-bf_tree-provider crate, while also modernizing quantization (PQ → spherical) and simplifying deletion semantics (hard deletes).

Changes:

• Extracts bf_tree + caching into diskann-bf_tree-provider and updates workspace/CI accordingly.
• Replaces PQ-based quant vector store with spherical quantization (Poly<dyn Quantizer>) + updated serialization.
• Simplifies delete handling by removing the delete-provider generic and switching to hard deletes.

Reviewed changes

Copilot reviewed 18 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
diskann-providers/src/model/graph/provider/async_/table_delete_provider.rs	Makes delete-table operations public and removes bf_tree-specific bitmap (de)serialization helpers/tests.
diskann-providers/src/model/graph/provider/async_/mod.rs	Adjusts module exports, removes bf_tree/caching modules, re-exports postprocess utilities.
diskann-providers/src/model/graph/provider/async_/caching/utils.rs	Updates import path and minor formatting in tests.
diskann-providers/src/model/graph/provider/async_/caching/provider.rs	Refactors imports/bounds formatting; adds Future import.
diskann-providers/src/model/graph/provider/async_/caching/mod.rs	Removes caching module root from diskann-providers.
diskann-providers/src/model/graph/provider/async_/caching/example.rs	Updates test imports to use diskann_providers / local module paths.
diskann-providers/src/model/graph/provider/async_/caching/bf_cache.rs	Updates ConfigError import path and test formatting.
diskann-providers/src/model/graph/provider/async_/bf_tree/vector_provider.rs	Adds hard-delete helper and adjusts TestCallCount assertions.
diskann-providers/src/model/graph/provider/async_/bf_tree/quant_vector_provider.rs	Switches PQ → spherical quantization + new query/distance computer adapter + test updates.
diskann-providers/src/model/graph/provider/async_/bf_tree/provider.rs	Removes delete-provider generic, implements hard delete, switches postprocess to CopyIds, updates quantizer serialization.
diskann-providers/src/model/graph/provider/async_/bf_tree/neighbor_provider.rs	Updates TestCallCount import path.
diskann-providers/src/model/graph/provider/async_/bf_tree/mod.rs	Switches re-exports to new bf_tree_provider module and removes QuantAccessor re-export.
diskann-providers/Cargo.toml	Removes bf_tree feature/deps from diskann-providers.
diskann-bf_tree-provider/src/lib.rs	Introduces new crate entry point with provider + caching modules.
diskann-bf_tree-provider/src/caching/mod.rs	Adds caching module root (example tests disabled with TODO).
diskann-bf_tree-provider/Cargo.toml	Adds new crate manifest + dependencies for bf_tree provider and spherical quantization.
Cargo.toml	Adds diskann-bf_tree-provider to workspace members.
.github/workflows/ci.yml	Removes bf_tree from explicitly tested feature list.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[JordanMaples]

@copilot would you mind giving this a review

[Copilot]

@copilot would you mind giving this a review

Reviewed the current branch at cbea923: cargo test -p diskann-bf_tree and cargo clippy -p diskann-bf_tree --all-targets -- -D warnings both pass locally, and CI run 25761279810 shows no failed jobs. I don’t see additional blockers right now.

[hildebrandmw]

Thanks Jordan, we can do quite a bit of cleanup on the crate structure and item names which I think would really start to tighten up the crate.

One question I had for you is what the expected end state is: are we targeting a full clean up or an incremental, non-published state while we continue working? I ask because the bf-tree provider inherited a bit of cruft from earlier iterations of the code, and I'm wondering how aggressiver we want to be in this PR in particular, considering that we still need integration tests and benchmarks.

[hildebrandmw]

diskann-bf_tree is quite a name ... maybe diskann-bftree? At least something that doesn't mix hyphens and underscores :laugh:

Also - the formatting here is a little off.

[JordanMaples]

oh strange, i missed that. Also, yeah this is like the third iteration on a name. I'll just make it diskann-bftree.

[hildebrandmw]

Nit: shuffling diff?

[JordanMaples]

I'll move that back.

[hildebrandmw]

Don't need polonius-the-crab now that the caching provider is gone.

[hildebrandmw]

If the only contents here are a directory, we can move away from the madness of the current inmem and flatten the hierarchy even further. What about something like this:

diskann-bftree/
  src/
    lib.rs
    provider.rs
    vectors.rs
    neighbors.rs
    quant.rs

And the following renames:

• BfTreeProvider -> Provider: Users would import as diskann_bftree::Provider`, which IMO is completely unambiguous.
• NeighborProvider -> Neighbors.
• VectorProvider -> Vectors.
• QuantVectorProvider -> QuantVectors.

[JordanMaples]

sure, i can do that.

[hildebrandmw]

Maybe either duplicate TestCallCount or drop entirely? Alternatively, we can move this into the Context used by the bf-tree provider and always gather these metrics. It's generally pretty useful to have these things.

[hildebrandmw]

Would this second condition be sufficient? Otherwise, there's a transient situation where concurrent accesses could see a deleted vector, which IIRC we don't handle gracefully yet.

As an aside, we should change the NeighborAccessor traits to have ToRanked errors instead of ANNError.

[hildebrandmw]

Changing the semantics of deleted vector to being deleted in the underlying BFTree means we can no longer use the Panics error. Instead, we need a true ToRanked error with a transient when accessing deleted vectors. And this needs to get propagated all the way down to the inherent access methods on the various providers to avoid overheads associated with ANNError.

[hildebrandmw]

We really really don't want to use PassThrough here. While the prune algorithm caches the result internally to avoid refetching for every pairwise distance, Map will allow vectors to be reused across multiple prunes.

[hildebrandmw]

I think this can be Infallible instead?

[hildebrandmw]

We can completely ditch Hybrid as a strategy, just using FullPrecision and Quantized. Quantized search should still rerank, but that's generally expected.