bug(vm-driver): registry image prep expects bundle-style rootfs/ from `umoci raw unpack` — failure masked as success and broken disk cached forever

[s2cube]

Agent Diagnostic

• Investigated with a coding agent against a live macOS (Apple silicon) gateway running the VM driver, plus the repo source at current main.
• Tested OpenShell v0.0.86 (latest release at time of filing; main's crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh contains the same defect, so the bug reproduces on latest).
• Searched existing issues: no duplicate. bug: vm-driver BYOC does not build the container properly causing VM process to exit with 0 #2149 is related only in symptom — the exit-code masking described below collapses many distinct guest failures into the same vague "VM process exited with status 0".
• Investigation path: sandbox conditions → gateway logs → per-sandbox rootfs-console.log → prepared-disk inspection with debugfs/e2fsck → captured the (deleted-on-exit) image-prep VM console by snapshotting the staging dir during prep → finally booted the prep VM manually with openshell-driver-vm --internal-run-vm against a repaired copy of the poisoned disk, which surfaced the real guest error.
• The agent did root-cause and fix this locally (fix below); this issue is filed so the fix can land upstream.

Description

On a VM-driver gateway with no local container engine visible, creating a sandbox from any registry image other than the bootstrap/base image permanently fails with:

Ready=False reason=ProcessExited message="VM process exited with status 0"

and the sandbox's rootfs-console.log shows:

[0.074s] FATAL: prepared image disk missing /image-rootfs

Three stacked defects cause this:

1. Root cause — wrong expectation of umoci raw unpack output. In scripts/openshell-vm-sandbox-init.sh, prepare_guest_image_rootfs() runs:

   /opt/openshell/bin/umoci raw unpack --image "$payload_dir/oci:openshell" "$partial_root"
   if [ ! -d "$partial_root/rootfs" ]; then
       ts "FATAL: umoci unpack did not produce rootfs directory"
       exit 1
   fi
   mv "$partial_root/rootfs" "$image_root"

   umoci raw unpack extracts the image filesystem directly into the target directory — there is no bundle-style rootfs/ subdirectory (that's umoci unpack). The unpack actually succeeds every time; manually booting the prep VM shows:

   • unpacking rootfs ...
   • ... done
   • unpacked image rootfs: /overlay/image-rootfs.partial
   FATAL: umoci unpack did not produce rootfs directory
   

   and debugfs -R "ls /image-rootfs.partial" on the disk shows a complete rootfs (bin boot dev etc home lib ...) directly under image-rootfs.partial/. The script then FATALs and exits despite a successful unpack.

2. Masking bug A — guest init exit codes do not survive the libkrun boundary. When guest PID 1 exits (any status), the microVM powers off and the launcher's worker process exits 0. run_image_prep_vm in crates/openshell-driver-vm/src/driver.rs judges prep success purely by that process status, so the failed prep looks successful. (Same masking turns the sandbox-boot FATAL into the vague "VM process exited with status 0" — see also bug: vm-driver BYOC does not build the container properly causing VM process to exit with 0 #2149, a different bug with the identical reported symptom.)

3. Masking bug B — the prepared disk is cached without validation. build_prepared_image_disk renames the disk into the content-addressed cache without checking it contains /image-rootfs. The broken disk (payload only, no unpacked rootfs, and unclean-shutdown bitmap-checksum damage since init exits without sync/umount) then poisons every subsequent sandbox for that image digest until the cache directory is deleted by hand.

Why this is rarely seen: with Docker/Podman visible, prep uses the local-docker tar export path; the base image needs no prep at all (it is the bootstrap rootfs). Only the registry/umoci fallback path is affected.

Reproduction Steps

1. Run a gateway with the VM driver on a host where the gateway process cannot see a container engine (vm driver: no local container engine available, falling back to registry appears in the log).
2. openshell sandbox create --image ghcr.io/nvidia/openshell-community/sandboxes/pi:latest --name pi-test (any non-base registry image reproduces).
3. Sandbox reaches error with ProcessExited: VM process exited with status 0; the per-sandbox rootfs-console.log under the VM driver state dir shows FATAL: prepared image disk missing /image-rootfs.
4. Deleting and recreating never recovers (poisoned cache at <state-dir>/images/sandbox-prepared-rootfs-ext4-umoci-v3-*); deleting that directory rebuilds the disk, which breaks identically.
5. To see the true failure: copy the cached rootfs.ext4, e2fsck -fy it, then boot it manually with openshell-driver-vm --internal-run-vm --vm-root-disk <bootstrap rootfs.ext4> --vm-overlay-disk <copy> --vm-exec /srv/openshell-vm-sandbox-init.sh --vm-env OPENSHELL_VM_INIT_MODE=image-prep --vm-console-output <log> and read the console.

Environment

• OS: macOS 15 (Apple silicon), libkrun backend
• Container engine: none visible to the gateway process (registry/umoci prep path)
• OpenShell: v0.0.86 (Homebrew); defect also present in main's scripts/openshell-vm-sandbox-init.sh
• Latest release checked: yes (v0.0.86 is latest)
• Possible duplicates checked: yes — bug: vm-driver BYOC does not build the container properly causing VM process to exit with 0 #2149 shares the symptom string but is a different failure (BYOC read-only /image-cache mount); no issue matches this root cause

Logs

# gateway log (prep looks successful; no error between these lines)
WARN openshell_driver_vm::driver: vm driver: no local container engine available, falling back to registry image_ref=ghcr.io/nvidia/openshell-community/sandboxes/pi:latest
INFO openshell_driver_vm::driver: vm driver: sandbox root disk plan resolved ... image_identity=sandbox-prepared-rootfs-ext4-umoci-v3:openshell-0.0.86:sha256:00d0c5e9...

# sandbox rootfs-console.log (every boot from the poisoned cache)
[0.002s] setting up writable overlay root
[0.074s] FATAL: prepared image disk missing /image-rootfs

# prep VM console, captured via manual --internal-run-vm boot (the real error)
[0.003s] setting up writable overlay root
[0.055s] preparing sandbox image rootfs in guest (oci-layout)
   • unpacking rootfs ...
   • ... done
   • unpacked image rootfs: /overlay/image-rootfs.partial
[10.598s] FATAL: umoci unpack did not produce rootfs directory

# the "missing" rootfs, present all along (after e2fsck repairs the unclean state)
$ debugfs -R "ls /image-rootfs.partial" rootfs-copy.ext4
bin boot dev etc home lib media mnt opt proc root ...

Fix

Verified working locally (a pi-image sandbox reaches ready and runs /usr/bin/pi):

1. scripts/openshell-vm-sandbox-init.sh: after umoci raw unpack, move $partial_root itself to $image_root (keeping a defensive branch for a bundle-style rootfs/ layout).
2. driver.rs::build_prepared_image_disk: validate the produced disk contains /image-rootfs (read-only debugfs -R "stat /image-rootfs") before renaming it into the cache; on failure, do not cache and include the prep VM console tail in the error instead of reporting success.
3. driver.rs::prepared_image_disk_size_bytes: size for payload + unpacked rootfs coexisting (payload*4 + 1 GiB instead of payload*3 + 512 MiB; the file is sparse, so headroom is free) — with the layout bug fixed, the old floor is a near-miss for ~1.2 GB-compressed images.

Happy to open a PR with these changes.

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• I checked the latest OpenShell release and either reproduced the issue there or explained why I cannot upgrade/test it
• I searched existing issues for possible duplicates or explained why I could not
• My agent could not resolve this — the diagnostic above explains why (resolved locally; filed so the fix lands upstream)