fix(python): added cleanup_frag_reuse_index#7221
Conversation
gstamatakis95
force-pushed
the
feat/python-cleanup-frag-reuse-index
branch
from
f2eaabb to
d90747b
Compare
Xuanwo
left a comment
Xuanwo
left a comment
There was a problem hiding this comment.
- The new Python cleanup entry point can panic when fragment-reuse metadata cannot be loaded. Corrupt or stale metadata can surface as a panic instead of a normal Python exception.
- The current test only exercises the already-caught-up index case. A cleanup implementation that drops all generations would still pass, so the regression can reappear.
…-frag-reuse-index # Conflicts: # python/python/tests/test_optimize.py
Thanks for the feedback @Xuanwo, addressed both issues. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Also moved to |
| Ok(true) => {} | ||
| Ok(false) => all_caught_up = false, | ||
| // An InvalidInput error means the reuse version is likely corrupted; drop it. | ||
| Err(Error::InvalidInput { .. }) => continue 'versions, |
There was a problem hiding this comment.
The cleanup path treats a partially indexed rewrite group as corrupt and drops the reuse generation, while the load path treats the same straddling state as recoverable. Calling the new Python cleanup API on an older dataset can remove the generation that keeps affected indexes safe.
| .await | ||
| .unwrap(); | ||
| // Surface corrupt/stale metadata as a normal error instead of panicking on unwrap. | ||
| let frag_reuse_details = load_frag_reuse_index_details(dataset, frag_reuse_index_meta).await?; |
There was a problem hiding this comment.
The cleanup path still reads external fragment-reuse details with unchecked offset/size arithmetic. Malformed metadata can still panic or produce an invalid object-store range instead of returning a Python exception.
2 participants
Closes #7220
Expose the existing Rust frag-reuse pruner to Python as
LanceDataset.cleanup_frag_reuse_index().Deferred-remap compaction appends a generation to the
__lance_frag_reuseindex on every commit, and without pruning these accumulate forever and are loaded on every vector index open. The race-safe pruner from #3836 already exists in Rust but had no Python binding and is never invoked automatically.The binding follows the
migrate_manifest_paths_v2wrapper pattern, the Python method is documented besidecleanup_old_versionsas a periodic maintenance call, and a new test verifies that__lance_frag_reusereports zero versions once all indexes have caught up and the cleanup runs.