Issue 4: High-Throughput Tokenized Meter Event Streaming Backpressure Filter

[elizabetheonoja-art]

File: /src/gateway/stream.rs

Category: Gateway / Streaming

Description: The BackpressureFilter uses a fixed-capacity mpsc channel. When a utility infrastructure event causes a sudden spike (e.g., 500K meters reporting simultaneously after a brownout recovery), the channel saturates and begins dropping events silently. There is no adaptive backpressure mechanism, no priority queuing for critical events (over-voltage alerts vs routine readings), and no spill-to-disk strategy.

Parameters:

• Normal throughput: 10,000 events/sec
• Spike throughput: 500,000 events/sec
• Critical event priority: alerts delivered within 100ms regardless of queue depth
• Memory budget: 512MB max for event buffers

Codebase Navigation:

1. src/gateway/stream.rs:14 — BackpressureFilter::new — fixed buffer_capacity is the limitation
2. src/gateway/stream.rs:28 — push — silent drop on channel full

Resolution Blueprint:

1. Implement a multi-level priority queue (critical/high/normal/low) using a custom binary heap with priority ordering
2. Add adaptive buffer sizing: dynamically grow buffer up to 512MB during spikes, shrink during quiescence
3. Implement spill-to-disk using a memory-mapped ring buffer for overflow events when memory budget exceeded
4. Add a DrainTask that asynchronously flushes spilled events back into the processing pipeline
5. Expose GET /api/v1/gateway/backpressure returning current buffer utilization, spill count, and priority distribution

Acceptance Criteria:

• Zero critical events dropped under 500K/sec spike
• Adaptive buffer grows/shrinks within 512MB budget
• Spill-to-disk recovers all overflow events within 5 seconds of spike subsiding
• Integration test validates priority delivery during saturation

[real-venus]

Hey, I've read through the issue and I understand it.

Could you assign it to me? I can finish this within 24 hrs.

[clintjeff2]

Hello, I have 6 years of experience working on tasks like this and handling similar challenges across backend, frontend, and smart-contract development. I have carefully read and fully understood the task requirements, and I am confident I can deliver a clean, efficient, and well-tested solution promptly. I always ensure high-quality implementation, maintainability, and adherence to best practices.

Pull request in 5 hours.

[grantfox-oss]

🦊 GrantFox — @real-venus has been assigned to this issue as part of the Official Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #4)
2. Your PR will be reviewed by the Utility-Protocol maintainers

Good luck! Track your progress on GrantFox.