Summary
With request_body_mode: FULL_DUPLEX_STREAMED, Envoy can re-deliver the final RequestBody chunk (with end_of_stream=true) after the ext_proc server has already received and processed an end_of_stream=true chunk for the same request. This causes the ext_proc server to re-process an already-complete body, leading to parse failures.
Versions affected
| Istio version |
Envoy version |
Duplicate EoS observed |
| 1.26.2 |
1.34 |
No (3/3 pass) |
| 1.27.9 |
1.35.13 |
Yes (~2/3 fail) |
| 1.28.8 |
1.36.9 |
Yes (~4/5 fail) |
| 1.29.5 |
1.37.5 |
Yes (~7/8 fail) |
The failure rate increases with newer Envoy versions, suggesting a regression introduced in 1.35.
Reproduction
Setup: Single ext_proc filter with request_body_mode: FULL_DUPLEX_STREAMED, processing JSON inference requests.
Trigger: Send a large (~1MB+) request body. Multi-turn chat payloads consistently reproduce this.
# Generate a ~1.2MB JSON body
python3 -c "
import json
msgs = [{'role':'user','content':'analyze this: ' + 'x'*500}]
for i in range(20):
msgs.append({'role':'assistant','content':[{'type':'output_text','text':'y'*2000}]})
msgs.append({'role':'user','content':'check function ' + str(i)})
json.dump({'model':'gpt-5.5','input':msgs}, open('body.json','w'))
"
curl -d @body.json \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
https://<gateway>/v1/responsesExpected: One RequestBody message with end_of_stream=true, followed by no further RequestBody messages for this request.
Actual: Two RequestBody messages with end_of_stream=true for the same stream. Server logs:
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":false} x 7
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":true} <- first EoS, processed
{"msg":"Incoming request body chunk","x-request-id":"bef9abfc-...","EoS":true} <- duplicate EoS
Same payload without ext_proc: 100% success (50/50 at 2.5MB), confirming the issue is in ext_proc delivery, not upstream.
Impact
Any ext_proc server that accumulates body chunks and processes on end_of_stream=true will re-process the already-complete buffer with the duplicate chunk appended, causing parse failures. This affects real workloads -- multi-turn LLM conversations routinely exceed 1MB.
We have worked around this in llm-d-inference-payload-processor#225 by tracking body completion state and ignoring chunks after end_of_stream, but the duplicate delivery appears to violate the ext_proc contract.
Additional note
A second failure mode was observed under the same conditions (unexpected end of JSON input, suggesting the body was parsed before all chunks arrived), which may be a separate delivery anomaly. Kept out of scope here.
Summary
With
request_body_mode: FULL_DUPLEX_STREAMED, Envoy can re-deliver the finalRequestBodychunk (withend_of_stream=true) after the ext_proc server has already received and processed anend_of_stream=truechunk for the same request. This causes the ext_proc server to re-process an already-complete body, leading to parse failures.Versions affected
The failure rate increases with newer Envoy versions, suggesting a regression introduced in 1.35.
Reproduction
Setup: Single ext_proc filter with
request_body_mode: FULL_DUPLEX_STREAMED, processing JSON inference requests.Trigger: Send a large (~1MB+) request body. Multi-turn chat payloads consistently reproduce this.
Expected: One
RequestBodymessage withend_of_stream=true, followed by no furtherRequestBodymessages for this request.Actual: Two
RequestBodymessages withend_of_stream=truefor the same stream. Server logs:Same payload without ext_proc: 100% success (50/50 at 2.5MB), confirming the issue is in ext_proc delivery, not upstream.
Impact
Any ext_proc server that accumulates body chunks and processes on
end_of_stream=truewill re-process the already-complete buffer with the duplicate chunk appended, causing parse failures. This affects real workloads -- multi-turn LLM conversations routinely exceed 1MB.We have worked around this in llm-d-inference-payload-processor#225 by tracking body completion state and ignoring chunks after
end_of_stream, but the duplicate delivery appears to violate the ext_proc contract.Additional note
A second failure mode was observed under the same conditions (
unexpected end of JSON input, suggesting the body was parsed before all chunks arrived), which may be a separate delivery anomaly. Kept out of scope here.