Commit d0d7c28
committed
fix(stream): proactively clean stream queue before each request to prevent stale done responses
This fixes an intermittent bug where streaming API responses would
return empty (done=True, no content), and subsequent requests would also
return empty responses until model switch/refresh. The root cause was
the STREAM_QUEUE holding leftover done signals from previous requests
due to a race condition in queue clearing.
- Clear stream queue at request start in _process_request_refactored
- Enhance use_stream_response to ignore and log empty stale done signals
- Add detailed logging for all stream/queue edge cases for diagnosis
- Reduce risk of state leakage between requests; safer under heavy load
No breaking changes. Existing APIs and non-streaming mode unaffected.1 parent 0dde544 commit d0d7c28
2 files changed
Lines changed: 325 additions & 271 deletions
0 commit comments