Repository navigation
Conversation
When an NFS client deletes a file that still has open file descriptors, it renames the file to .nfsXXXX on the server instead of removing it, and only sends the REMOVE once the last descriptor is closed. If the sandbox VM is killed before closing its descriptors, that REMOVE never arrives and the .nfsXXXX file - as large as the original, often several GiB - leaks on the NFS server permanently (observed in production: a 5 GiB workspace image lingering as .nfs000000019000ee98... with no process holding it). OnNetworkRelease runs after the VM is dead and all of its descriptors are gone, so any .nfs* orphan in the volume is safe to delete there. Add removeNFSOrphans, called per chroot before Close(): it scans the chroot host-side root (chrooted.Chrooted.ActualRoot) and removes top-level .nfs* files, skipping subdirectories, best-effort (failures logged, teardown never aborted). Rebased onto current main (OnNetworkRelease now iterates mounts by lifecycleID and takes the chroot from mounted.chroot). Adds a regression test: .nfs* removed, real/other dotfiles and a .nfs* subdirectory kept, and a missing or empty directory is a no-op. Fixes e2b-dev#3532. Supersedes e2b-dev#3533.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
October 23, 2026 03:21
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased follow-up to #3533, which @ryan-sana closed as inactive (76 days) and no longer merging cleanly. The leak is still present on current
main(no.nfs*cleanup anywhere in the tree), so this re-does the fix against main's current structure and adds a regression test.Problem
NFS silly-rename: when an NFS client deletes a file that still has open file descriptors, the client renames it to
.nfsXXXXon the server instead of removing it, and only sends theREMOVEonce the last descriptor is closed. If the sandbox VM is killed before closing its descriptors, thatREMOVEnever arrives and the.nfsXXXXfile — as large as the original, often several GiB — leaks on the NFS server permanently.Observed in production: a 5 GiB workspace image lingering as
.nfs000000019000ee98..., no process holding it (confirmed via/proc/*/fd), never reclaimed.Fix
OnNetworkReleaseruns after the VM is dead and all of its descriptors are gone, so any.nfs*orphan in the volume is safe to delete there. AddremoveNFSOrphans, called per chroot beforeClose():chrooted.Chrooted.ActualRoot, returned bychroot.Root()),.nfs*files, skips subdirectories (entry.IsDir()guard, so a directory named.nfs…is never touched),Adapted to current
main:OnNetworkReleasenow iteratesmountsbylifecycleIDand takes the chroot frommounted.chroot; the cleanup is inserted right beforechroot.Close().Test
TestRemoveNFSOrphans(no external deps):.nfs*files removed; real files, non-.nfsdotfiles, and a.nfs…subdirectory kept;go build ./...andgo vetpass.Note on reproduction
Silly-rename only occurs on a real NFS mount, so the mechanism itself isn't reproducible on a non-NFS dev filesystem; the test exercises the cleanup logic given
.nfs*orphans, and the mechanism/impact rests on the NFS protocol behavior plus the production observation above.Fixes #3532. Supersedes #3533.