Fix macOS handoff flake: tolerate EINVAL when arming SO_RCVTIMEO on a closed peer

[jaredLunde]

The bug

On macOS, ~1–2 of 22 crash_matrix fault-injection scenarios failed per run, a different subset each time, all with the same supervisor-log signature:

handoff aborted: Some("ready read failed: io: Invalid argument (os error 22)")

EINVAL on the supervisor's timeout-bounded read of a control frame. Linux is unaffected.

Root cause (proven at the syscall level)

The supervisor arms a per-recv liveness timeout with SO_RCVTIMEO (UnixStream::set_read_timeout) before each control-socket read. macOS/BSD sosetoptlock rejects any setsockopt with EINVAL once a socket is fully shut down (SS_CANTRCVMORE | SS_CANTSENDMORE) — exactly the state a peer that closed its end leaves behind. Linux has no such check.

So when the successor (or incumbent) closed its socket end right as the supervisor reached set_read_timeout, the call returned EINVAL instead of the read returning a clean EOF. Flaky because it raced the peer's fork/exec/exit window — which is why a different subset failed each run.

Instrumentation at the failing setsockopt confirmed: fd was a valid connected SOCK_STREAM, SO_ERROR=0, timeval {10,0} (valid), yet both set_read_timeout and a hand-rolled setsockopt(SO_RCVTIMEO) returned EINVAL. Two states observed:

state	so_nread	peek	meaning
A	0	EOF (0)	peer closed, empty buffer → clean EOF
B	28	data (1)	peer closed with a complete Ready frame still buffered

State B is why the task's "no blind retry / no swallow" constraint matters: treating EINVAL as failure/EOF would drop a buffered Ready and abort a handoff the successor actually completed.

Fix

arm_recv_timeout() arms the timeout and, on EINVAL, confirms via a non-blocking MSG_PEEK that the read cannot block (peer gone → buffered frame is delivered, otherwise EOF) before reading without the timeout. A genuine EINVAL on a still-open empty socket is surfaced rather than risking an unbounded blocking read. Linux path unchanged; atomic SOCK_CLOEXEC socketpair creation preserved. Applied to all three arming sites (read_until/Ready, seal-wait, Hello).

Test fixes (pre-existing, macOS-timing-exposed)

• Several crash scenarios asserted O's asynchronously-written resume-called (and N's startup successor-pid) marker with an instant marker_exists right after the crashed process exited — racing the survivor's recovery. Converted to the bounded wait_marker(…, 3s) already used elsewhere in the file. A real "never resumes" bug still fails after 3s, so this absorbs scheduling jitter without masking regressions.
• count_open_fds read Linux-only /proc/self/fd; now uses /dev/fd on macOS/BSD so the FD-leak stress check runs (and passes) cross-platform — also confirming the fix leaks no fds.

Verification (macOS)

• crash_matrix green over 50 consecutive runs; the three task-named EINVAL scenarios over 40 more.
• Full cargo test --workspace --lib --tests green.
• cargo clippy --workspace --all-targets and cargo fmt --check clean.
• ARCHITECTURE.md updated (same commit) to document the macOS SO_RCVTIMEO-on-shutdown behavior.

🤖 Generated with Claude Code