feat: allow tuning miniblock value chunks to 32k#7356
Conversation
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
westonpace
left a comment
There was a problem hiding this comment.
I think bumping the max up to 32Ki is fine. I think there is some small chance we might end up seeing the "too much rep/def information and it doesn't all fit in the miniblock" error again but we can tackle that problem a different way when it surfaces.
This allows miniblock writers to use up to 32K logical values per chunk when explicitly configured via
LANCE_MINIBLOCK_MAX_VALUES, while keeping the default at 4096.The file format already stores
log_num_valuesin 4 bits, so the writer-side guard can allow values up to 15 without requiring the large-chunk metadata path. The compressed byte-size limits remain enforced.Fixes #7326.