test(vector/hnsw): stabilize incremental_append_recall_matches_fresh_build with best-of-3 (#886)#899
Merged
Merged
Conversation
…build with best-of-3 (#886) The incremental regime (base=200) runs a structurally different code path from the fresh build (incremental append via promoted_ep vs. one-shot construction), so their variance doesn't cancel like the rebuild regime (base=1) does. Rare unlucky parallel-insertion orderings under parallel_build could dip a single build's self-recall@10 into a tail below the fixed tolerance on low-core CI runners. Parametrize assert_append_matches_fresh with cold_trials and take the max self-recall@10 across independent seed_then_append builds, applied to the incremental regime only (cold_trials=3); the rebuild regime keeps cold_trials=1 (unchanged, not observed to be flaky). Verified the mitigation cannot mask a real regression: disabling the #872 promoted_ep fix under RAYON_NUM_THREADS=2 reproduces an identical failing self-recall@10 (~0.83) on every trial, since the pre-#872 defect is a deterministic structural funneling through old_ep, not a scheduling fluke. With the fix enabled, 20 consecutive release-mode runs under RAYON_NUM_THREADS=2 all pass. Closes #886 Refs #872, #885
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
incremental_append_recall_matches_fresh_build(the perf(vector/hnsw): incremental append onto a small base uses a low-level entry point, degrading concurrent-build quality (seed-then-bulk-load) #872 base>=ef_construction regression gate) occasionally failed on low-core CI runners because the incremental regime runs a structurally different code path from the fresh build (incremental append viapromoted_epvs. one-shot construction), so their variance doesn't cancel the way the rebuild regime (base=1) does.assert_append_matches_freshwithcold_trialsand take the max self-recall@10 across independentseed_then_appendbuilds, applied to the incremental regime only (cold_trials=3); the rebuild regime keepscold_trials=1(unchanged — not observed to be flaky).Why best-of-N, not median or a wider tolerance
Full investigation and design rationale posted to #886 as comments before this PR.
Verification
promoted_epfix (temporarily) underRAYON_NUM_THREADS=2(emulating CI'subuntu-latest2-core runner) — all 3 trials produced an identical failing self-recall@10 (~0.83 vs ~0.99 fresh), confirming the pre-perf(vector/hnsw): incremental append onto a small base uses a low-level entry point, degrading concurrent-build quality (seed-then-bulk-load) #872 defect is a deterministic structural funneling throughold_ep, not a scheduling fluke, so best-of-3 cannot mask it.writer.rsdiff vs. main is empty);incremental_append_recall_matches_fresh_buildrun 20 consecutive times in release mode underRAYON_NUM_THREADS=2— all pass (satisfies the issue's acceptance criterion).cargo fmt --checkclean;cargo clippy -p laurus --all-targets -- -D warningsclean on both 1.95 and 1.97 (CI) toolchains;cargo test -p laurus --lib(1265 passed) andcargo test -p laurus --tests(1467 passed) both green.Test plan
cargo test -p laurus --test hnsw_coldstart_recall_test(both regimes) passesRAYON_NUM_THREADS=2passCloses #886
Refs #872, #885