Skip to content

feat(guest): supervise container init lifecycle#982

Open
BatmanByte wants to merge 3 commits into
boxlite-ai:mainfrom
BatmanByte:codex/init-stdio-drain
Open

feat(guest): supervise container init lifecycle#982
BatmanByte wants to merge 3 commits into
boxlite-ai:mainfrom
BatmanByte:codex/init-stdio-drain

Conversation

@BatmanByte

@BatmanByte BatmanByte commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Turn #982 into PR1 of the container-exit design: the guest now owns one init supervisor that observes the init process and publishes a terminal result only after stdout and stderr have drained. This is the guest-side foundation for #933; the shim's durable outcome file and host-state reconciliation remain follow-up work. Design context: Discussion #967.

Changes

  • Keep continuously draining init stdout/stderr, expose a completion signal for both readers, and retain the bounded diagnostic tail.
  • Build OCI init through the pre-thread zygote, then supervise its exit through the existing non-blocking zygote wait IPC.
  • Add generation-aware Container.Wait; same-generation not-ready requests are retryable, while stale generations fail precondition. Reconnected waiters return an already-published outcome immediately.
  • Persist and advance lifecycle_generation before each new VM run, then include it in Container.Init.

How to verify

  • make fmt:check:rust
  • make guest
  • make test:unit:rust FILTER=lifecycle_generation
  • make test:integration:rust FILTER=init_stdout_larger_than_a_pipe_does_not_block_entrypoint (requires a host where kern.hv_support is available)

Risks / rollout

Container.Wait is guest-only in this PR. The next PR adds the shim monitor, crash-durable outcome file, and host-side reconciliation before exposing natural init exit as a runtime status.

Summary by CodeRabbit

  • New Features
    • Added a Container “wait” capability that reports init termination per lifecycle run, using a lifecycle generation to prevent stale results.
  • Bug Fixes
    • Improved init stdout/stderr handling to avoid startup stalls from pipe backpressure.
    • Strengthened init output capture for exit diagnostics by retaining a bounded output tail and draining continuously until completion.
  • Tests
    • Added/updated integration coverage for large stdout+stderr writes during init and verified readiness within a timeout.
    • Updated draining assertions to ensure repeated reads work without requiring EOF.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c0644016-3217-4314-9382-6ca66586eb17

📥 Commits

Reviewing files that changed from the base of the PR and between 5a3d357 and b9f4787.

📒 Files selected for processing (2)
  • src/boxlite/tests/init_stdio.rs
  • src/guest/src/container/zygote.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/boxlite/tests/init_stdio.rs
  • src/guest/src/container/zygote.rs

📝 Walkthrough

Walkthrough

The change adds lifecycle-generation tracking and generation-scoped wait responses, introduces zygote-based init construction with supervised termination handling, continuously drains bounded init output tails, and adds tests for non-blocking large stdout and stderr writes.

Changes

Lifecycle-scoped init supervision

Layer / File(s) Summary
Lifecycle generation propagation
src/boxlite/src/litebox/..., src/shared/proto/boxlite/v1/service.proto
Box state persists and advances lifecycle generations, which are passed through initialization and used to scope container wait responses.
Zygote init construction and startup
src/guest/src/container/{lifecycle.rs,start.rs,zygote.rs}, src/guest/src/container/mod.rs
Startup builds init through zygote IPC, starts it, and returns its PID with output-completion tracking.
Continuous init output draining
src/guest/src/container/stdio.rs, src/boxlite/tests/init_stdio.rs
Background readers retain bounded stdout/stderr tails, report EOF completion, and are tested with large simultaneous output.
Supervision and wait handling
src/guest/src/container/supervisor.rs, src/guest/src/service/{container.rs,server.rs}
Supervisors combine process termination with output completion and map terminal outcomes to generation-aware wait responses.
Supporting executor adjustment
src/guest/src/service/exec/mod.rs
Container locking in an executor error path now uses an immutable binding.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BoxBuilder
  participant ContainerService
  participant Container
  participant Zygote
  participant InitSupervisor
  participant ContainerWaitClient
  BoxBuilder->>ContainerService: send lifecycle generation
  ContainerService->>Container: start init
  Container->>Zygote: build init with stdio fds
  Zygote-->>Container: return init build result
  Container->>InitSupervisor: supervise PID and output completion
  ContainerWaitClient->>ContainerService: Wait(container_id, lifecycle_generation)
  ContainerService->>InitSupervisor: await terminal outcome
  InitSupervisor-->>ContainerService: exit, signal, or failure
  ContainerService-->>ContainerWaitClient: ContainerWaitResponse
Loading

Possibly related PRs

  • boxlite-ai/boxlite#949: Both changes extend InitPipelineContext and its constructor in src/boxlite/src/litebox/init/types.rs.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main guest-side init supervision change.
Description check ✅ Passed The description follows the required template and covers summary, changes, verification, and rollout risks.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@BatmanByte
BatmanByte marked this pull request as ready for review July 13, 2026 07:39
@BatmanByte
BatmanByte requested a review from a team as a code owner July 13, 2026 07:39
@boxlite-agent

boxlite-agent Bot commented Jul 13, 2026

Copy link
Copy Markdown

📦 BoxLite review — 1 issues · b9f4787

Review evidence

  • git diff --numstat origin/main...HEAD && git diff origin/main...HEAD — reviewed full diff, 17 files
  • grep host-side callers of new Wait RPC/ContainerWaitRequest — no host-side caller found; plumbing unused
  • trace get_execution_plan (init/mod.rs) for reattach/restart generation reuse — fresh VM per non-Running start, no stale-supervisor race
  • cargo check -p boxlite-guest -p boxlite — cargo not installed in sandbox, no toolchain found

Risk notes

  • init failure detection — removed synchronous is_running()/diagnose_exit() check post-start; new InitSupervisor+Wait RPC added guest-side but never called from host, so immediate init crashes silently report as success
  • stdio drain rewrite — background-thread bounded-tail drain replacing non-blocking single-shot read fixes real pipe-full deadlock; well covered by new tests (large writer, no-block-on-open-writer, completion signaling)
  • lifecycle_generation plumbing — proto/host/guest generation threading is consistent; advance only on non-Running start, matches fresh-VM-per-generation model
  • zygote build_init unsafe fd handling — OwnedFd::from_raw_fd over SCM_RIGHTS matches existing build() pattern, ownership comment present
  • coverage sampled — only skimmed zygote.rs test additions and exec/mod.rs one-line change (Mutex lock mut->immut); no toolchain to compile/run guest or boxlite test suites in this sandbox
src/guest/src/service/container.rs
  ContainerService::init/wait  +129/-63  removed fail-fast check, added Wait RPC
src/guest/src/container/supervisor.rs
  InitSupervisor  +117/-0  new generation-gated watch-based supervisor
src/guest/src/container/stdio.rs
  ContainerStdio/drain_fd  +206/-67  background-thread bounded tail drain
src/guest/src/container/lifecycle.rs
  Container::start  +65/-24  async, zygote-built init, returns StartedContainer
src/guest/src/container/zygote.rs
  build_init/do_build_init  +93/-1  new IPC request to build init via zygote
src/boxlite/src/litebox/box_impl.rs
  BoxImpl start  +10/-1  advance+persist lifecycle_generation
src/boxlite/src/litebox/state.rs
  advance_lifecycle_generation  +29/-0  monotonic counter with overflow check
src/shared/proto/boxlite/v1/service.proto
  Container.Wait  +29/-0  new RPC, unused by host

reviewed b9f4787 in a BoxLite microVM · @boxlite-agent review to re-run · powered by BoxLite

Container init stdout and stderr were only read during diagnostics, so output beyond the pipe capacity could block the entrypoint indefinitely. Start dedicated bounded drain threads when the pipes are created and retain the latest 4 KiB for diagnostics.

The diagnostic accessor now returns a snapshot instead of consuming pipe readers, allowing repeated failure reports without waiting for inherited writers to close. Add a VM regression test that writes 1 MiB before proving the entrypoint continued.
@BatmanByte
BatmanByte force-pushed the codex/init-stdio-drain branch from e9de25a to bd36b9b Compare July 13, 2026 07:42

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/guest/src/container/stdio.rs`:
- Around line 41-42: Remove the top-level cfg(test) std::os::unix::io::AsRawFd
import in stdio.rs. Keep the existing AsRawFd import inside the tests module
unchanged, since it is the import used by the tests.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5ed1d573-a7d8-4f43-ac8c-aef9812dd071

📥 Commits

Reviewing files that changed from the base of the PR and between d029764 and e9de25a.

📒 Files selected for processing (4)
  • src/boxlite/tests/init_stdio.rs
  • src/guest/src/container/lifecycle.rs
  • src/guest/src/container/stdio.rs
  • src/guest/src/service/exec/mod.rs

Comment thread src/guest/src/container/stdio.rs Outdated
Create init through the pre-thread zygote and publish its terminal result through Container.Wait. The supervisor is the sole zygote waiter and only completes after stdout and stderr reach EOF, so a reconnecting host can retrieve an already-published result without losing the local output tail.

Persist a monotonic lifecycle generation before each VM run and include it in Container.Init. Wait rejects stale generations, preventing an old monitor from observing a later init run.
@BatmanByte BatmanByte changed the title fix(guest): continuously drain init stdio feat(guest): supervise container init lifecycle Jul 14, 2026

@boxlite-agent boxlite-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📦 BoxLite review — 1 issues

Comment on lines +126 to +149
let mut supervisors = self.init_supervisors.lock().await;
match supervisors.get(&container_id) {
Some(existing) if !existing.matches_generation(lifecycle_generation) => {
return Ok(Response::new(ContainerInitResponse {
result: Some(container_init_response::Result::Error(ContainerInitError {
reason:
"Container init already exists for another lifecycle generation"
.to_string(),
})),
}));
}
Some(_) => {
return Ok(Response::new(ContainerInitResponse {
result: Some(container_init_response::Result::Error(ContainerInitError {
reason: "Container init is already in progress".to_string(),
})),
}));
}
None => {
let supervisor = InitSupervisor::new(lifecycle_generation);
supervisors.insert(container_id.clone(), supervisor.clone());
supervisor
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 init_supervisors map has no eviction
Failed/terminal supervisor entries are never removed from the HashMap; low risk today since container_id is unique per VM boot, but no cleanup path exists if that assumption changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This registry is scoped to one guest VM, and each Box owns exactly one container. It is destroyed with the guest on stop/restart. The Container is retained while that VM lives for diagnostic output tails, so terminal eviction would either be dead code or prematurely discard diagnostics. No cleanup change is needed under the current one-container-per-VM model.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/guest/src/container/zygote.rs (1)

615-622: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a roundtrip test for InitBuildResult::Failed.

Only Built is tested; the analogous BuildResult type tests both Spawned and Failed variants (build_result_spawned_serde_roundtrip, build_result_failed_serde_roundtrip). Failed { error } carries opaque error text across the IPC boundary, so it's worth the same coverage.

♻️ Proposed addition
#[test]
fn init_build_result_failed_serde_roundtrip() {
    let result = InitBuildResult::Failed {
        error: "init build failed: container not found".to_string(),
    };
    let json = serde_json::to_vec(&result).unwrap();
    let decoded: InitBuildResult = serde_json::from_slice(&json).unwrap();
    assert_eq!(result, decoded);
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/guest/src/container/zygote.rs` around lines 615 - 622, Add a test
alongside init_build_result_serde_roundtrip named
init_build_result_failed_serde_roundtrip that constructs InitBuildResult::Failed
with representative error text, serializes it with serde_json, deserializes it,
and asserts the original and decoded values are equal.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/guest/src/service/container.rs`:
- Around line 124-150: The init supervisor registry in the Container.Init flow
must allow later lifecycle generations after a prior supervisor terminates.
Update the storage and lookup logic around init_supervisors and InitSupervisor
so supervisors are keyed by both container_id and lifecycle_generation, or
rotate terminal supervisors while preserving generation-specific results for
reconnecting waiters; retain rejection only for an active conflicting generation
or duplicate in-progress init.

---

Nitpick comments:
In `@src/guest/src/container/zygote.rs`:
- Around line 615-622: Add a test alongside init_build_result_serde_roundtrip
named init_build_result_failed_serde_roundtrip that constructs
InitBuildResult::Failed with representative error text, serializes it with
serde_json, deserializes it, and asserts the original and decoded values are
equal.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8c9337dc-20ca-4d46-8f21-d5a6e7603a81

📥 Commits

Reviewing files that changed from the base of the PR and between bd36b9b and 5a3d357.

📒 Files selected for processing (15)
  • src/boxlite/src/litebox/box_impl.rs
  • src/boxlite/src/litebox/init/mod.rs
  • src/boxlite/src/litebox/init/tasks/guest_init.rs
  • src/boxlite/src/litebox/init/types.rs
  • src/boxlite/src/litebox/state.rs
  • src/boxlite/src/portal/interfaces/container.rs
  • src/guest/src/container/lifecycle.rs
  • src/guest/src/container/mod.rs
  • src/guest/src/container/start.rs
  • src/guest/src/container/stdio.rs
  • src/guest/src/container/supervisor.rs
  • src/guest/src/container/zygote.rs
  • src/guest/src/service/container.rs
  • src/guest/src/service/server.rs
  • src/shared/proto/boxlite/v1/service.proto
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/guest/src/container/stdio.rs

Comment thread src/guest/src/service/container.rs
@DorianZheng DorianZheng added the e2e-local Triggers the local (in-process) E2E suite on the self-hosted runner label Jul 14, 2026
DorianZheng added a commit that referenced this pull request Jul 14, 2026
)

## Summary

[Run
29324915498](https://github.com/boxlite-ai/boxlite/actions/runs/29324915498/job/87061427258)
failed on fork PR #982: GitHub withholds secrets **and** repository
variables from fork `pull_request` runs (`Secret source: None`), so the
maintainer-applied `e2e-local` label sent the run into `start-runner`,
where `create-github-app-token` failed with "Input required and not
supplied: app-id". The label contract ("label ⇒ e2e runs") was
implemented on an event type that structurally cannot honor it for
forks.

Switch the PR trigger to label-gated `pull_request_target` (base
context: credentials available, workflow definition always from `main`),
hardened per the standard pattern (protobuf `test_runner.yml`, GitHub
security-hardening guidance):

- **Per-push fork approval** — a fork PR runs only on the `e2e-local`
labeling event, pinned to the labeled head SHA; pushes after approval
are skipped with a notice until a maintainer re-labels. Same-repo PRs
keep today's behavior (re-run on every push while labeled).
- **Checkout pins `head.sha`** — `pull_request_target` resolves to the
base branch by default, which would silently test `main`;
`persist-credentials: false` so no token remains on the persistent
runner when fork code runs.
- **Read-only default token** — top-level `permissions: contents: read`
(the `pull_request_target` default is write-capable); `should-run` gate
inputs move from inline `${{ }}` interpolation into `env:`.

Blast radius of an approved-then-malicious fork run stays contained:
secrets exist only in `start-runner`/`stop-runner` (GitHub-hosted, never
execute PR code), and the runner's instance profile allows only
self-termination (`scripts/ci/setup-ci-runner.sh`).

## Verification

- `actionlint`: identical 2 pre-existing findings as HEAD's version —
zero new findings
- Gate behavior: the `should-run` script extracted from the YAML and
executed across 9 event scenarios passes 9/9 (push/dispatch run;
same-repo labeled runs on label+synchronize; fork runs only on the
`e2e-local` labeling event; stale/unapproved fork events skip)
- Regression side: HEAD's gate script, given the failing run's exact
inputs (`pull_request`, fork, `e2e-local` present), returns `run=true` —
the path into the credential-less crash

After merge: remove and re-add `e2e-local` on #982 to run e2e against
its current head.

https://claude.ai/code/session_01HkVMvjNbCKPrbwxps95jTp

Co-authored-by: tester <t@t.test>
@DorianZheng DorianZheng added e2e-local Triggers the local (in-process) E2E suite on the self-hosted runner and removed e2e-local Triggers the local (in-process) E2E suite on the self-hosted runner labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

e2e-local Triggers the local (in-process) E2E suite on the self-hosted runner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants