Repository navigation
Conversation
When a Firecracker VM exits, in-flight NBD commands and the reply socket
fail in predictable ways that are not backend faults:
- cmdRead/cmdWrite/cmdWriteZeroes select on the dispatch context, so an
in-flight backend call observes context.Canceled (or DeadlineExceeded)
the moment the sandbox is torn down.
- completeAsync's reply-socket write fails with "use of closed network
connection" (net.ErrClosed) or "broken pipe" (EPIPE) once the client
side is gone.
All four paths logged these at Error, so every normal teardown with
in-flight I/O produced a burst of ERROR lines that drown out the real
backend failures Error is meant to surface (e2b-dev#3274).
Introduce isExpectedTeardownError (context.Canceled, context.DeadlineExceeded,
net.ErrClosed, syscall.EPIPE) and route all four failure logs through a new
logCmdFailure helper that downgrades those to Warn and keeps everything else
at Error. Genuine backend errors (corrupt block, ECONNRESET, unexpected EOF)
still surface at Error.
Adds dispatch_teardown_test.go: a table-driven test over the classifier
covering wrapped context errors, net.ErrClosed/EPIPE inside *net.OpError (how
a closed-socket write actually surfaces), and the negatives that must stay
Error.
Scope note: this PR revives only the NBD-dispatch half of the closed e2b-dev#3277.
Its other half — the FC metrics flusher select-race guard (e2b-dev#3275) — is
already fixed on main by 50c64be ("stop flushing Firecracker metrics after
the VM exits"), so it is intentionally left out. e2b-dev#3275 is CLOSED COMPLETED;
e2b-dev#3274 is still open. Adapted to main's refactor (single completeAsync error
log, command.fields(), injected d.logger). Closes e2b-dev#3274. Supersedes e2b-dev#3277.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 23, 2026 08:20
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a Firecracker VM exits, in-flight NBD commands and the reply socket fail in predictable, non-fault ways — yet all of them were logged at ERROR, so every normal teardown with in-flight I/O produced a burst of ERROR lines that drown out the real backend failures ERROR is meant to surface (#3274):
cmdRead/cmdWrite/cmdWriteZeroesselect on the dispatch context, so an in-flight backend call observescontext.Canceled(orDeadlineExceeded) the instant the sandbox is torn down.completeAsync's reply-socket write fails withuse of closed network connection(net.ErrClosed) orbroken pipe(EPIPE) once the client side is gone.Fix —
dispatch.goisExpectedTeardownError(err)— true forcontext.Canceled,context.DeadlineExceeded,net.ErrClosed,syscall.EPIPE.logCmdFailure(ctx, msg, cmd, err)and route all four failure logs through it: expected teardown errors go to Warn, everything else stays at Error.Genuine backend errors (corrupt block,
ECONNRESET, unexpected EOF) still surface at Error.Before (every teardown with in-flight I/O):
After: the same lines at
level=warn; real backend faults unchanged atlevel=error.Tests —
dispatch_teardown_test.goTable-driven test over the classifier (the single predicate both log levels hang off): wrapped
context.Canceled,net.ErrClosed/EPIPEinside*net.OpError(how a closed-socket write actually surfaces), plus the negatives that must stay Error (ECONNRESET, unexpected EOF, a plain error, nil).go build ./...andgo vetclean; fullnbdpackage green.Scope
Revives only the NBD-dispatch half of the closed #3277. Its other half — the FC metrics flusher select-race guard (#3275) — is already fixed on main by
50c64bee8("stop flushing Firecracker metrics after the VM exits"), so it is intentionally dropped. #3275 is CLOSED COMPLETED; #3274 is still open.Adapted to main's refactor since the original PR (single
completeAsyncerror log,command.fields(), injectedd.logger). The fix is also broader-but-precise vs the original: the original only special-casedcontext.Canceledon the backend path and blanket-downgraded the fatal path; here one classifier covers both backend-cancel and closed-socket errors and leaves real failures at Error.Closes #3274. Supersedes #3277 (closed for inactivity / merge drift; maintainer invited a fresh PR against current main).