Core: Coalesce nearby blob reads in PuffinReader.readAll - #17418
Open
vishnuprakaz wants to merge 1 commit into
Open
Core: Coalesce nearby blob reads in PuffinReader.readAll#17418vishnuprakaz wants to merge 1 commit into
vishnuprakaz wants to merge 1 commit into
Conversation
Resolve the TODO from apache#4537 by reading contiguous blobs in a single request, handed back as no-copy ByteBuffer views. A contiguous run is split into bounded reads (capped at MANIFEST_TARGET_SIZE, 8 MiB) so a large blob section never allocates one oversized buffer.
vishnuprakaz
force-pushed
the
puffin-readall-coalesce
branch
from
July 29, 2026 16:45
8f46a60 to
b663983
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PuffinReader.readAllissues one seek and read per blob. This resolves the TODO from #4537 to coalesce nearby reads: Puffin stats/index blobs are written contiguously, so reading them in fewer requests cuts round trips, which matters most on object stores.Contiguous and overlapping blobs are sorted by offset and read in a single request, returned as
ByteBufferviews over the region without copying. A run is split once it would exceed a max read size, capped at 8 MiB (theMANIFEST_TARGET_SIZEdefault), which bounds the extra memory held versus per-blob reads; a blob larger than the cap is still read on its own.RangeReadable.readVectoredwas considered, but its default implementation does not coalesce on S3/Hadoop and buffers every range eagerly.