Add priority backpressure filter with adaptive budget and spill-to-di…#89
Merged
elizabetheonoja-art merged 4 commits intoJun 26, 2026
Conversation
…sk (Utility-Protocol#4) Replaces the silent-drop fixed mpsc BackpressureFilter (kept for compat) with a PriorityBackpressureFilter in gateway/stream.rs: - Multi-level priority queue (Critical/High/Normal/Low): pop() always returns the highest priority first, so over-voltage alerts jump routine readings. - Adaptive memory budget: Critical/High bypass the budget (never dropped); Normal/Low spill once the byte budget is exceeded. Zero critical events dropped under saturation. - Spill-to-overflow via a SpillStore trait: InMemorySpillStore (bounded) and a disk-backed FileSpillStore (append log with read/write cursors, FIFO recovery) for codec-able events. drain_spill() replays spilled events back as memory frees; spawn_drain_task() runs it on an interval. - Serializable BackpressureStats (memory/utilization, per-priority counts, spill backlog, spilled/dropped totals) ready for the GET /api/v1/gateway/backpressure endpoint. Dependency-free / CI-runnable: spill-to-disk is std::fs (no mmap/libc), no new deps (serde/parking_lot already present). The existing BackpressureFilter and its tests are untouched. Tests cover strict priority ordering, critical budget bypass, zero-critical-drop under saturation, spill+drain full recovery, a mixed-load saturation integration test (all 50 criticals delivered, no loss), and file-spill FIFO roundtrip.
Contributor
Author
|
@elizabetheonoja-art |
elizabetheonoja-art
approved these changes
Jun 26, 2026
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
High-Throughput Tokenized Meter Event Streaming Backpressure Filter (#4)
Closes #4
What's added (
src/gateway/stream.rs)EventPriority(Critical/High/Normal/Low) +PriorityBackpressureFilterpop()always returns the highest priority first, so over-voltage alerts jump ahead of routine readings.Critical/Highbypass the byte budget (never spilled or dropped);Normal/Lowspill once the budget is exceeded.SpillStoretrait +InMemorySpillStore+FileSpillStoreSpillCodec).drain_spill()replays spilled events as memory frees;spawn_drain_task()runs it on an interval.BackpressureStats(Serialize) +stats()GET /api/v1/gateway/backpressure.