Add zero-copy framed telemetry ingestion with buffer-pool backpressure #87
Merged
elizabetheonoja-art merged 5 commits intoJun 26, 2026
Conversation
Utility-Protocol#33) Replaces delimiter-scanning (`read_until('\n')`) on the binary telemetry stream -- which buffered the whole stream looking for a newline that never comes, growing to GBs before OOM -- with bounded, length-prefixed framing: - ingestion/buffer_pool: semaphore-gated recycling BufferPool (default 1024 x 64KB = 64MB). acquire() awaits when all buffers are in flight (backpressure instead of unbounded allocation); released buffers are recycled, keeping the steady-state allocation rate ~0 (tracked via allocation_count()). - ingestion/frame_parser: read_frame() reads the u32 BE length prefix, validates length <= MAX_FRAME_SIZE before reading, then read_exact()s exactly `length` bytes into a pooled buffer; CBOR is decoded zero-copy straight from the pooled slice (ciborium::de::from_reader, no per-frame Vec). - ingestion/stream_handler: bounded frame loop; an oversized length is logged as a security event and ends the connection. - benches/stream_throughput: criterion comparison of pooled vs per-frame-alloc reads over 10k frames. Adds ciborium (pure Rust CBOR; no protoc/native toolchain) -- bytes/serde already present. Self-contained (the issue's stream_handler.rs etc. did not exist). Tests cover pool recycling/backpressure, frame read+decode, oversized rejection before read, held-frame backpressure, and full-stream dispatch.
Contributor
|
@real-venus CI failed |
Contributor
Author
Now I am working on it. |
Contributor
Author
|
@elizabetheonoja-art |
elizabetheonoja-art
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Telemetry Stream Ingestion Memory Backpressure with Zero-Copy Binary Frames (#33)
Closes #33
What's added (
src/ingestion/)buffer_pool.rsBufferPool— aSemaphore-gated recycling pool (default 1024 × 64 KB = 64 MB).acquire()awaits when all buffers are in flight (backpressure to the reader instead of unbounded allocation); released buffers are recycled, so the steady-state allocation rate is ~0 (exposed viaallocation_count()).frame_parser.rsread_frame()reads theu32BE length prefix, validateslength ≤ MAX_FRAME_SIZEbefore reading, thenread_exact()s exactlylengthbytes into a pooled buffer. CBOR is decoded zero-copy straight from the pooled slice (ciborium::de::from_reader, no per-frameVec).stream_handler.rshandle_stream()— bounded frame loop dispatching each frame to a callback (buffer recycled on drop). An oversized length is logged as a security event and ends the connection.