Repository navigation
A stream that is silent through a long prefill is dropped by Node clients and prefilled for nobody #567
Description
Before you start
- I have read the FAQ and my problem is not answered there.
- I have read the Roadmap and this is not already planned there.
- I have searched existing issues and found no duplicate.
- I am on the latest release, or on a freshly rebuilt
mainwhen building from source.
What happened
A streamed request sends its headers and one chunk, then nothing until the first token. A long prompt with nothing cached prefills for minutes first - 20 minutes for 985,621 tokens of Qwen3.8-Flash-Next on my card - and two things go wrong in that silence:
- Node clients drop the stream at 300 s. Node's
fetch(undici) aborts a response body that receives no bytes forbodyTimeout, 300 s by default, and the OpenAI SDK for Node streams throughfetch. Against an SSE endpoint that sends one chunk and then stays silent, Node 26.5.0 fails withTypeError: terminated, causeUND_ERR_BODY_TIMEOUT, at 307.1 s. The same endpoint sending: keep-aliveevery 20 s completes after 330 s. - A client that left is not noticed until the first token.
FrontendManager.stream_with_cancellationcallsrequest.is_disconnected()only when a chunk arrives, so the prefill of a dropped request runs to its end for nobody. With--max-running-requests 1the client's retry queues behind it.
FreeToken's own shell client waits longer (CHAT_READ_TIMEOUT = 900.0 in shell/client.py), which a 1M-token prefill also exceeds.
Expected: the stream stays open through a prefill of any length, a keep-alive line reaching the client within its idle timeout, and a client that leaves cancels its request's prefill.
Actual: the Node client fails at 307.1 s with UND_ERR_BODY_TIMEOUT (the log below), and the server prefills the request to its end for nobody.
How did you install FreeToken
Built from source
FreeToken version
0d652e7 (stream_with_cancellation is unchanged since v0.1.3, where I hit it)
OS
Windows 11
OS details
Windows 11 Pro 26200, Python 3.13.3. Nothing here is platform-specific: it is the HTTP layer.
GPU and driver
RTX 3090 Ti 24 GB, driver 616.92
CPU and system RAM
AMD Ryzen 9 9950X3D, 256 GB
Checkpoint
RadixArk/Qwen3.8-Flash-Next-NVFP4
Command
ft serve --model RadixArk/Qwen3.8-Flash-Next-NVFP4 --max-running-requests 1
# then stream any prompt whose prefill takes longer than 300 s from a Node client:
node -e "const r = await fetch('http://127.0.0.1:1919/v1/chat/completions', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({model: 'qwen3.8-flash-next', stream: true, messages: [{role: 'user', content: LONG_PROMPT}]})}); for await (const c of r.body) {}"Full log
0.0 s headers 200
0.0 s chunk "data: {\"first\":true}\n\n"
307.1 s FAILED TypeError: terminated cause=UND_ERR_BODY_TIMEOUT Body Timeout Error
Anything else
SSE comment lines keep the body alive and are invisible to clients: the OpenAI SDK's SSEDecoder drops lines that start with :, and FreeToken's own clients read only data: lines. I have a fix with tests and will open a PR linked here.
Activity
SUKUNA-AI commentedon Oct 19, 2026
A data point for point 2 (a client that left is not noticed until the first token), from a single RTX 5070 Ti 16 GB
running FreeToken 0.1.3 plus a fork, with prompts up to 250K tokens. We do not have anything for point 1 (keepalive on
/v1/chat/completions).Still present on main 4d2e2e2.
FrontendManager.stream_with_cancellation(server/api_server.py:364-374) stillcalls
request.is_disconnected()only insideasync for chunk in generator. During a cold prefill the generatoryields nothing after the role chunk, so the disconnect is seen only at the first token.
with_keepalive(
server/generation.py:482) is used by/v1/messagesand/v1/responses(anthropic_api.py:513,responses_api.py:605) but not by/v1/chat/completions.What we run (outside FreeToken,
serving/guard.pyin our repo): a pure ASGI middleware that replays the requestbody to the app and runs its own watcher task on
receive(). The watcher does not depend on chunks being produced:on
http.disconnectit dispatchesabort_user(uid)and then cancels the app task. The same middleware covers thenon-streaming case of #644.
Evidence (30-minute 256K stability run,
results/long256-30min/l256short-r1-analysis.json):the client disconnected 8, 30 or 55 s after sending, i.e. in the middle of a prefill whose TTFT at 250K is about 77 s.
Caveat: our long prefill is a fork-specific layer-major prefill that checks for an abort between its steps. Upstream's
chunked prefill would check between chunks instead. The detection part (a watcher on
receive()rather than a checkper chunk) is independent of that, and it is the part that matters for this issue.
Our implementation, for reference (fork of 0.1.3, Apache-2.0; paths in the fork):
serving/guard.pyresults/long256-30min/l256short-r1-analysis.jsonYevheniiKotyrlo commentedon Oct 27, 2026
#572 fixes point 2 as well as point 1, for every streaming endpoint:
stream_with_cancellationwaits for the next chunk at most 15 s at a time, sends: keep-alivewhile the stream is silent, and checksrequest.is_disconnected()on each wake, so a client that leaves mid-prefill aborts its request within 15 s instead of at the first token. The non-streaming case in #644 is outside it.