Summary
Inbound WebSocket frame reassembly in httpd_ws_handler is quadratic in the
frame size, making large client→server frames slow on hardware. On an ESP32-S3,
a single 64 KB echo round-trip takes ~4 s, and under 4 concurrent connections
some 64 KB echoes exceed a 20 s client timeout. This is a throughput/latency
issue only — payloads are never truncated or corrupted, and it is unrelated to
the WS send path (which is fine).
Evidence
Measured with examples/httpd_debug/scripts/ws_stress.py against the
/ws echo endpoint on an ESP32-S3:
| Run |
Result |
gen (server→client) 4w × 8i × {8K,64K} |
64/64 perfect, 7 s |
echo (round-trip) 2w × 5i × 64K |
10/10 perfect, ~4.3 s each |
echo 4w × 8i × {8K,64K} |
62/64, 0 truncation / 0 corruption, 2 client timeouts; ~167 s |
Note the asymmetry: gen (large outbound frames) is fast; echo (which adds
large inbound frame reassembly) is slow. So the cost is on the receive side.
Root cause
For every TCP segment received, the httpd worker forwards the bytes to the WS
gen_server, which appends to a buffer and re-parses the entire accumulated
buffer from scratch:
src/httpd.erl:157-158 — each received packet is gen_server:cast to the WS
process (handle_web_socket_message/2), copying the segment across processes.
src/httpd_ws_handler.erl:108 — NewBuffer = <<Buffer/binary, Packet/binary>>
reconcatenates the growing buffer per segment.
src/httpd_ws_handler.erl:110 — parse_frame(NewBuffer) re-scans the full
buffer on every segment until the frame is complete.
A 64 KB frame arriving over lwIP (MSS ~1440 B) is ~45 segments, so the buffer is
reconcatenated and fully re-parsed ~45 times → O(n²) bytes touched, plus ~45
inter-process copies.
Proposed directions (not committing to one yet)
- Parse the frame header once to learn the total length, then accumulate into a
reverse iolist (or count remaining bytes) and only finalize/unmask when the
declared length is reached — avoids per-segment re-parse and re-concat.
- Avoid the per-segment cross-process hop (e.g. let the worker accumulate the
frame, or pass an iolist), reducing copies.
- Unmask in a single pass once the full payload is assembled (already O(n), but
ensure it isn't repeated on partial buffers).
Reproduce
cd examples/httpd_debug
./scripts/flash.sh
python3 scripts/ws_stress.py <device-ip> --mode echo -s 65536 -w 4 -i 8 -t 30
Context
Surfaced while validating the socket:send {error, eagain} backpressure
handling on hardware. The send-side fix (frames routed through
tcp_server:send/2) passed cleanly (0 truncation/corruption across all runs);
this issue tracks the separate inbound-reassembly performance work.
Summary
Inbound WebSocket frame reassembly in
httpd_ws_handleris quadratic in theframe size, making large client→server frames slow on hardware. On an ESP32-S3,
a single 64 KB echo round-trip takes ~4 s, and under 4 concurrent connections
some 64 KB echoes exceed a 20 s client timeout. This is a throughput/latency
issue only — payloads are never truncated or corrupted, and it is unrelated to
the WS send path (which is fine).
Evidence
Measured with
examples/httpd_debug/scripts/ws_stress.pyagainst the/wsecho endpoint on an ESP32-S3:gen(server→client) 4w × 8i × {8K,64K}echo(round-trip) 2w × 5i × 64Kecho4w × 8i × {8K,64K}Note the asymmetry:
gen(large outbound frames) is fast;echo(which addslarge inbound frame reassembly) is slow. So the cost is on the receive side.
Root cause
For every TCP segment received, the httpd worker forwards the bytes to the WS
gen_server, which appends to a buffer and re-parses the entire accumulated
buffer from scratch:
src/httpd.erl:157-158— each received packet isgen_server:castto the WSprocess (
handle_web_socket_message/2), copying the segment across processes.src/httpd_ws_handler.erl:108—NewBuffer = <<Buffer/binary, Packet/binary>>reconcatenates the growing buffer per segment.
src/httpd_ws_handler.erl:110—parse_frame(NewBuffer)re-scans the fullbuffer on every segment until the frame is complete.
A 64 KB frame arriving over lwIP (MSS ~1440 B) is ~45 segments, so the buffer is
reconcatenated and fully re-parsed ~45 times → O(n²) bytes touched, plus ~45
inter-process copies.
Proposed directions (not committing to one yet)
reverse iolist (or count remaining bytes) and only finalize/unmask when the
declared length is reached — avoids per-segment re-parse and re-concat.
frame, or pass an iolist), reducing copies.
ensure it isn't repeated on partial buffers).
Reproduce
Context
Surfaced while validating the
socket:send{error, eagain}backpressurehandling on hardware. The send-side fix (frames routed through
tcp_server:send/2) passed cleanly (0 truncation/corruption across all runs);this issue tracks the separate inbound-reassembly performance work.