Skip to content

WebSocket inbound frame reassembly is O(n²) — large frames slow on device #11

Description

@harmon25

Summary

Inbound WebSocket frame reassembly in httpd_ws_handler is quadratic in the
frame size, making large client→server frames slow on hardware. On an ESP32-S3,
a single 64 KB echo round-trip takes ~4 s, and under 4 concurrent connections
some 64 KB echoes exceed a 20 s client timeout. This is a throughput/latency
issue only — payloads are never truncated or corrupted, and it is unrelated to
the WS send path (which is fine).

Evidence

Measured with examples/httpd_debug/scripts/ws_stress.py against the
/ws echo endpoint on an ESP32-S3:

Run Result
gen (server→client) 4w × 8i × {8K,64K} 64/64 perfect, 7 s
echo (round-trip) 2w × 5i × 64K 10/10 perfect, ~4.3 s each
echo 4w × 8i × {8K,64K} 62/64, 0 truncation / 0 corruption, 2 client timeouts; ~167 s

Note the asymmetry: gen (large outbound frames) is fast; echo (which adds
large inbound frame reassembly) is slow. So the cost is on the receive side.

Root cause

For every TCP segment received, the httpd worker forwards the bytes to the WS
gen_server, which appends to a buffer and re-parses the entire accumulated
buffer from scratch:

  • src/httpd.erl:157-158 — each received packet is gen_server:cast to the WS
    process (handle_web_socket_message/2), copying the segment across processes.
  • src/httpd_ws_handler.erl:108NewBuffer = <<Buffer/binary, Packet/binary>>
    reconcatenates the growing buffer per segment.
  • src/httpd_ws_handler.erl:110parse_frame(NewBuffer) re-scans the full
    buffer on every segment until the frame is complete.

A 64 KB frame arriving over lwIP (MSS ~1440 B) is ~45 segments, so the buffer is
reconcatenated and fully re-parsed ~45 times → O(n²) bytes touched, plus ~45
inter-process copies.

Proposed directions (not committing to one yet)

  1. Parse the frame header once to learn the total length, then accumulate into a
    reverse iolist (or count remaining bytes) and only finalize/unmask when the
    declared length is reached — avoids per-segment re-parse and re-concat.
  2. Avoid the per-segment cross-process hop (e.g. let the worker accumulate the
    frame, or pass an iolist), reducing copies.
  3. Unmask in a single pass once the full payload is assembled (already O(n), but
    ensure it isn't repeated on partial buffers).

Reproduce

cd examples/httpd_debug
./scripts/flash.sh
python3 scripts/ws_stress.py <device-ip> --mode echo -s 65536 -w 4 -i 8 -t 30

Context

Surfaced while validating the socket:send {error, eagain} backpressure
handling on hardware. The send-side fix (frames routed through
tcp_server:send/2) passed cleanly (0 truncation/corruption across all runs);
this issue tracks the separate inbound-reassembly performance work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions