Repository navigation
Conversation
…rocess-stream fan-out A subscriber whose client TCP connection died after a sandbox resume (its FIN arrived after the VM snapshot) leaves its consumer goroutine wedged inside stream.Send until the kernel's retransmit timeout fires — ~924s (~15 min) at the default tcp_retries2=15. MultiplexedChannel used unbuffered subscriber channels. The fan-out loop serialises over all subscribers; at the ghost both `case s.ch <- v` (nobody reading) and `case <-s.done` (never cancelled, because the consumer never returns to run its deferred cancel) block, stalling delivery to every other subscriber for the full timeout. This is e2b-dev#3292: stdout/stderr goes silent for exactly ~15 min, then resumes. Fix, scoped to multiplex.go: 1. Buffered subscriber channels (subscriberBufSize = 64) absorb normal bursts past a briefly-slow consumer. 2. A non-blocking overflow path (`default:`) treats a consumer that falls a full buffer behind as a ghost and drops it: cancel() so later iterations skip it, closeCh() so its own goroutine's range exits. Both are sync.Once-guarded, so double-close is impossible. chOnce is kept separate from once so cancel (done) and closeCh (ch) stay independent. Healthy subscribers are untouched — their sends are buffered or complete immediately. Adds multiplex_ghost_test.go with three regression tests for the case the existing Abandoned* tests do not cover — a ghost that never reads AND never cancels: - GhostNeverReadingDoesNotWedgeFanOut: flood subscriberBufSize+10 past a ghost; the healthy subscriber still receives every value in order. - GhostChannelClosedAfterOverflow: after overflow the ghost channel is closed, so a ranging consumer terminates instead of leaking. - MultipleHealthyUnaffectedByOneGhost: 3 healthy subscribers all receive the full stream alongside one ghost. All three fail (producer back-pressured to timeout) against the unbuffered implementation and pass under -race with the fix. Scoped to this one fix. The original e2b-dev#3344 also bundled a separate stream.Send write-deadline mechanism (writedeadline.go) and connect/start wiring; that is a distinct concern and is left out here so this change stays reviewable. Closes e2b-dev#3292. Supersedes e2b-dev#3344.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 23, 2026 08:10
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After a sandbox is paused and resumed, envd can keep a subscriber whose client TCP connection is already dead — the client's FIN arrived after the VM snapshot was taken. That subscriber's consumer goroutine is wedged inside
stream.Send, waiting for TCP's retransmit timeout: ~924 s (~15 min) at the kernel defaulttcp_retries2=15.MultiplexedChannelused unbuffered subscriber channels. The fan-out loop serialises over all subscribers; when it reaches the ghost, both arms of itsselectblock:case s.ch <- v:— the consumer is stuck instream.Send, so nobody is readings.ch.case <-s.done:— the consumer never returns, so its deferredcancel()never runs anddoneis never closed.The whole fan-out stalls for every subscriber for the full TCP timeout. This is exactly #3292: stdout/stderr goes silent for ~15 min, then suddenly resumes.
The existing
TestMultiplexedChannel_AbandonedConsumer*tests don't catch this: they model a consumer that stops reading but then callscancel(), which closesdoneand lets the fan-out unblock viacase <-s.done. A real ghost never returns fromstream.Send, socancel()never runs.Fix —
multiplex.goonlysubscriberBufSize = 64) absorb normal bursts past a briefly-slow consumer.default:): a consumer that falls a full buffer behind is treated as a ghost and dropped —cancel()closesdoneso later fan-out iterations skip it viaisCancelled();closeCh()closess.chso the ghost's ownfor rangeconsumer exits (no goroutine leak).sync.Once-guarded (oncefordone, a separatechOnceforch), so double-close is impossible and the two are independent.Healthy subscribers are untouched — their sends are buffered or complete immediately.
Tests —
multiplex_ghost_test.goThree regression tests for the uncovered case (ghost that never reads and never cancels):
GhostNeverReadingDoesNotWedgeFanOutsubscriberBufSize+10past a ghost; the healthy subscriber still receives every value, in order, within the timeout.GhostChannelClosedAfterOverflowfor rangeconsumer terminates instead of leaking.MultipleHealthyUnaffectedByOneGhostAll three fail (producer back-pressured to timeout on send 3) against the current unbuffered implementation, and pass under
-racewith the fix.go build ./...andgo vetare clean.Scope
Deliberately scoped to this one fix. The earlier #3344 also bundled a separate
stream.Sendwrite-deadline mechanism (writedeadline.go) plusconnect/start/servicewiring — a distinct concern, left out here so this change stays small and reviewable.Closes #3292. Supersedes #3344 (closed for inactivity / merge drift; maintainer invited a fresh PR against current main).