nvx-rs: add the shared execution engine and its guest binding by esaurez · Pull Request #748 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-22T06:53:25Z

What and why

This adds nvx::exec, one framing-free engine that owns an execution's stream state, credit
arithmetic, ordering, and outcome, and nvx::guest::exec::binding, which binds the engine's
runner to the guest's Execution driver. Independently framed adapters, and later the control
protocol, drive the same engine, so in-flight memory stays bounded with no total output cap, and
timers, cancellation, and control keep progressing while any stream, sink, window, or transport
is blocked.

`nvx::exec` (standard library only, no feature)

Runner, the side that runs the workload, and Peer, the side that requested it: state
machines that never block. The caller passes the current Instant, retrieves bounded messages
and requests one at a time, and ticks each side at its reported deadline. Each side's credit
takes turns with its other messages, so that a caller that takes one message at a time starves
none of them, however fast credit frees up.
Types: Key (execution and generation), Stream, Pool, Window and Limits (one byte
window per stream, or frames charged against one pool that stdout and stderr share, with a
maximum chunk), Timers (timeout, grace, settlement), Cause, Status, LaunchFailure,
Settlement, Outcome, Receipt, and Violation.
Credit: whole-window initial grants, partial consumption, over-credit refusal, chunk and window
violations, stale and foreign messages told apart by generation, and a memory bound per side.
Completion: stdin EOF follows every accepted input byte, and an early close never leaves the
producer waiting; each stream keeps its order and ends after its last byte, and the other
stream's output never holds its end back; cancellation,
timeout, reset, drop, and disconnection progress whatever the streams do, drain for a bounded
grace, and mark discarded output as truncated; an exit that reaches the runner before a tick
fires its timeout stands as the workload's own; normal completion is lossless; the outcome is
reusable only after settlement, and a failed or late settlement fences it. An execution that
does not settle leaves its streams that are still open the settlement time and the grace after
the workload's end to close, then the runner gives up on them, as truncated, but for those that
the adapter reported with output_hung_up: no process holds them any more, so the runner waits
for the transport to take the rest. A launch result that has not arrived by then fences the
execution too, whether or not it settled, as a process may hold back what reports it: the
runner gives up on the result and on those streams, and sends the output and the outcome
without it, with the workload's status and no cause; a result that arrives after any give-up
counts for nothing, so no start follows the output. After a termination,
the runner fences an execution only if its cleanup is still incomplete once the grace and the
settlement time have passed; one whose cleanup completed waits only for the transport. Once the
peer has detached or given up, the runner's messages are late, so its own outcome stands, and a
start after the terminal outcome is out of order.

`nvx::guest::exec::binding` (Linux, feature `guest`)

Binding applies the runner's requests to G3's Execution: read allowances become grant,
discards discard, accepted input write_input and then close_input, or close_input as
soon as a write fails, which it reports as Event::InputFailed unless the program closed its
end, terminations cancel, disconnect, or terminate, and settlement a kill of the
execution's cgroup followed by pids.current checks until it is empty, after which the cgroup
is the caller's again: neither the execution nor the binding's drop kills it, so the caller may
reuse it at once. An execution without a cgroup is fenced, but after the program's end the
binding tells the runner which output streams hung up, polling them for POLLHUP, so that
none of their output is given up. A caller that reaps every child hands statuses over with
observe.
It never blocks: the caller polls descriptors(), input_writer(), and deadline(). Each
call first collects the program's status without blocking, unless the caller reaps the program,
and takes the launch report once it has arrived, and reports the program's end before the
runner's timers fire or it returns any message, even in a first call after the timeout, and
even before the launch report, which a process that another thread forked while the program was
spawning may hold back; the runner holds the start, the output, and the terminal outcome until
the report arrives, or until the settlement time and the grace have passed since the program's end,
when it gives up on the report and fences the execution. It
applies every request that its steps raised before it returns, such as closing the input after
its last byte, checks a settling cgroup whenever a check is due, however much output flows, and
takes a bounded number of steps, writes of the input among them, each of at most a chunk, so
that no execution holds the caller's loop. Once it returned a message, it steps the execution
before it returns another, and the writes leave a step for that, so that a caller that hands it
input before every call, which frees credit for each, never keeps it from reading output. G3's
driver gains crate-private steps that collect
the status and take the launch report without blocking, which next_event and the binding
share, and a crate-private release of a settled cgroup. nvx::guest::spawn::Command gains a
test-only spawn that hands a test a copy of the launch report's writer, to hold the report back.
G3's both_streams_take_turns_while_each_has_output now waits for the launch report as well
as the status before it reads, as the report can come after the status for the same reason.

Not in this PR

NVX's managed agent, host::instance, and the CLI are unchanged, and keep NVXC v1. There is no
wire format here. The test adapter's executions follow in a separate PR.

Part of #623
Refs #619, #535

Validation

Windows 11, from the repository root and nvx-rs with NVX_BUNDLE=skip:

python scripts\nvx.py verify
cargo fmt --all --check
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo clippy --locked --lib --no-default-features -- -D warnings
cargo build --locked --lib --no-default-features and cargo test --locked --lib --no-default-features
cargo test --locked --all-features, then cargo test --locked
cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings
cargo +1.89 check --locked --all-targets --all-features
tests/runtime-adapter: the protocol-only and host-only configurations, as the check action
runs them
aci_edge_sandboxes: the formatting, lint, test, documentation, and minimum Rust version steps
of its check action, with ACI_EDGE_SANDBOXES_AGENT_BUNDLE=skip

Linux 6.18, as an unprivileged user:

every Linux step of the nvx-rs and aci_edge_sandboxes check actions, with the repository's
pinned toolchain and the minimum Rust version: the commands above, the artifact-bundling test,
the macOS checks, python3 scripts/nvx.py check-guest-binaries, the guest binaries' build and
checks, and the agent-bundling test
tests/runtime-adapter: the protocol-only, host-only, and guest-only (musl) configurations

Two musl runs of check-guest-binaries failed in G3's
guest::exec::tests::both_streams_take_turns_while_each_has_output, whose launch report came
after the program's status because another test's forked child still held the report's pipe:
this PR now has that test wait for the report. One run of the musl unit tests failed only in
guest::sys::signal::tests::a_thread_that_blocks_sigpipe_keeps_only_a_signal_that_was_pending,
a race of the same kind in a test that this PR leaves unchanged.

Also on Linux 6.18, as root through sudo: the guest binding's tests, ten times, which then also
settle and reuse a real cgroup.

No real VM is needed for this change. Left to CI: the KVM, MSHV, and WHP backend matrix.

Copilot

🟡 Changes recommended

Unresolved fencing and settlement-progress defects affect execution safety and completion.

4 open issues

Late terminal overrides peer fence · New
Settlement expiry can wait indefinitely on open pipes · New
Output draining bypasses due settlement checks · New
Pipe capacity makes marker test timeout · New

What changed in this PR

Adds a shared, framing-free execution engine to nvx-rs and binds it to the Linux guest execution driver. This supports the streaming work in #623 without changing existing NVXC v1 callers.

Changes:

Introduces runner and peer state machines for credit-based streams, ordering, cancellation, and settlement.
Adds a nonblocking guest binding with cgroup cleanup and caller-owned status collection.
Adds deterministic engine tests, guest integration tests, documentation, and no-default-features CI checks.

File	Description
nvx-rs/src/lib.rs	Exports and documents the execution engine.
nvx-rs/src/guest/mod.rs	Documents the guest binding.
nvx-rs/src/guest/exec/binding/tests.rs	Tests real programs, backpressure, and settlement.
nvx-rs/src/guest/exec/binding.rs	Connects runner requests to the guest driver.
nvx-rs/src/guest/exec.rs	Exposes the binding module.
nvx-rs/src/exec/tests/truncation.rs	Tests draining, truncation, and cleanup deadlines.
nvx-rs/src/exec/tests/streams.rs	Tests stream ordering, stdin closure, and launch failure.
nvx-rs/src/exec/tests/races.rs	Tests competing termination events.
nvx-rs/src/exec/tests/mod.rs	Provides the deterministic execution harness.
nvx-rs/src/exec/tests/memory.rs	Tests bounded buffering across long streams.
nvx-rs/src/exec/tests/liveness.rs	Tests progress under blocked streams and transport.
nvx-rs/src/exec/tests/keys.rs	Tests stale, foreign, and late messages.
nvx-rs/src/exec/tests/credit.rs	Tests byte and frame credit accounting.
nvx-rs/src/exec/runner.rs	Implements the workload-side state machine.
nvx-rs/src/exec/peer.rs	Implements the requesting-side state machine.
nvx-rs/src/exec/mod.rs	Defines shared types and execution contracts.
nvx-rs/src/exec/credit.rs	Implements credit accounting and stream queues.
nvx-rs/README.md	Documents APIs, feature availability, and validation.
.github/actions/check-nvx-rs/action.yml	Adds feature-free library build and test checks.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Unresolved lifecycle and ordering issues can incorrectly fence settled executions, delay exit or stdin EOF handling, and accept a start after a terminal record.

4 open issues

4 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Queued output can delay default guest status collection, causing false timeouts and potentially discarding output from a normally completed workload.

1 open issue

Collect child status before servicing queued output and timers · New

4 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Unresolved lifecycle and timeout-ordering defects affect safe execution completion, and a regression fixture can stall on supported Linux pipe capacities.

3 open issues

1 resolved since last review

Collect child status before servicing queued output and timers

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

The binding can incorrectly classify an already-exited program as timed out when its first poll reaches the deadline.

1 open issue

Report cached exit before applying workload timeout · New

3 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The interacting streaming, termination, and cgroup-reuse state machines require final human review of their timing and cleanup guarantees.

1 open issue

Bound Binding::next() stdin writes to preserve cancellation checks · New

1 resolved since last review

Report cached exit before applying workload timeout

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The interacting streaming and cgroup-reuse lifecycles need final human review, and an unresolved stdin error-path defect remains.

1 open issue

Close execution input before marking runner input closed · New

1 resolved since last review

Bound Binding::next() stdin writes to preserve cancellation checks

_{Review effort: Balanced}

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

On input write failure (anything other than WouldBlock), the error detail is discarded. This makes… · New
Several binding integration tests spawn real programs that generate very large outputs (e.g., `seq… · New

1 resolved since last review

Close execution input before marking runner input closed

_{Review effort: Lite}

Copilot

🔵 Needs a closer look

The lifecycle and cgroup-reuse state machines warrant human review, and late launch-report ordering remains unresolved.

1 open issue

Ignore delayed launch reports after launch abandonment · New

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The interacting credit, termination, and cgroup-reuse state machines warrant final human review despite extensive regression coverage.

1 open issue

Ignore delayed launch reports after launch abandonment

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Interdependent credit, timer, stream-ordering, and cgroup-reuse guarantees warrant final human review despite extensive regression coverage.

No open issues 🎉

1 resolved since last review

Ignore delayed launch reports after launch abandonment

_{Review effort: Balanced}

Add nvx::exec, one framing-free engine that owns an execution's stream state, credit arithmetic, ordering, and outcome, for the side that runs the workload, the Runner, and the side that requested it, the Peer. Adapters with any framing drive the same state machines, so the bytes in flight stay bounded with no total output cap, and timers, cancellation, and control keep progressing while a stream, sink, window, or transport is blocked. The engine needs only the standard library and builds without features. Each side takes the current Instant from its caller, returns bounded messages and requests one at a time, and reports its next deadline. The adapter chooses one byte window per stream, or frames charged against one pool that standard output and error share, and the maximum chunk, and keys tell stale messages from foreign ones by generation. The input ends only after every byte that the runner accepted; each stream ends after its last byte; a cancellation, timeout, drop, reset, or disconnection progresses whatever the streams do, with a bounded grace and the truncation marked in the outcome; an exit that reaches the runner before a tick fires its timeout stands as the workload's own; normal completion is lossless; an execution is reusable only once it settled, and one that does not settle leaves the streams that a process may still hold a bounded time to close, while those that hung up wait for the transport. After a termination, the runner fences an execution whose cleanup is still incomplete once the grace and the settlement time have passed, while one whose cleanup completed waits only for the transport to take its last messages. Once the peer detached or gave up, the runner's messages are late, so the peer's own outcome stands whenever its caller retrieves it, and a start after the terminal outcome is out of order. Deterministic tests drive both sides through a simulated workload, transport, and sink on a manual clock. The check action now also builds and tests the library without default features. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

Add nvx::guest::exec::binding, whose Binding applies a Runner's requests to a guest Execution and feeds the execution's events back to it: read allowances become grants, discards discards, accepted input writes and then a close, or a close as soon as a write fails, which the binding reports unless the program closed its end, terminations a cancellation, disconnection, or termination, and settlement a kill of the execution's cgroup followed by occupancy checks until the cgroup is empty. From then on, the cgroup is the caller's again: the execution no longer kills its processes, not even when the binding drops, so the caller may reuse it at once. An execution without a cgroup never settles, so it is fenced; after the program's end, the binding still tells the runner which output streams hung up, as no process holds them any more, so that none of their output is given up. A caller that reaps every child hands statuses over with observe. The binding never blocks: the caller polls its descriptors, its input writer, and its deadline. Each call first collects the program's status without blocking, unless the caller reaps the program, and takes the launch report once it has arrived, and reports the program's end before its timers fire or it returns any message, even in a first call after the timeout; it applies every request that its steps raised, such as closing the input after its last byte, before it returns, checks a settling cgroup whenever a check is due, however much output flows, and takes a bounded number of steps, its writes of the input among them, each of at most a chunk, so that no execution holds the caller's loop. The driver's nonblocking collection of the status and of the launch report become steps of their own for that, the driver tells whether an output stream hung up, and it gives a settled cgroup back to its caller. Linux tests drive real programs through the binding with a peer in the same process: output and input beyond their windows, an input that the program closes early, one that closes with its last byte while output waits, and one whose write fails, which the binding reports, settlement before reuse with descendants, sessions, and inherited pipe writers, a cgroup that the caller reuses before it drops the binding, the fences of failed cleanups, statuses that the execution or the caller collects, which reach the runner ahead of queued output and of its timeout, even in a first call after it, the bounded wait for output that a descendant holds once the execution does not settle, and the unbounded one for the output that no process holds, due settlement checks while output drains slowly, the bounds on one call's steps and writes, and two executions in one loop, one with a stalled sink and one with a pipe writer that escaped its kill. Run as root, one test settles and reuses a real cgroup. The test loop waits only for what the bindings and peers wait on, so a step that nothing leads to fails a test rather than passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

Each side of nvx::exec returned credit before any other message, and grants credit as soon as any unit of it frees up. An adapter that takes one message at a time could free credit before every message: by handing the runner input that the workload takes between two messages, or by letting the peer's sink take output. The runner then returned only input credit, never the workload's start or its output, and the peer only one output pool's credit, never the input, its end, or the other pool's credit. The runner's input credit and its other messages now take turns: after credit, any other message that waits goes first. The peer's cancellation still goes first, and each output pool's credit and the input then take turns. Each side keeps its order otherwise: the start before output, each stream's data before its end, and the input's bytes before its end. Two deterministic tests take one message at a time while credit frees up before each, and check that every other message still goes out, in turns with the credit. Both fail without this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

The runner returned a stream's end only once neither stream had output to send. An adapter that takes one message at a time could hand the runner more of one stream's output before every message, while the peer's sink took it and granted its credit back: the runner then returned only that stream's output, and never the end of the other stream, though that stream had closed and its last byte had gone out. The runner now returns a stream's end before any output. A stream ends only once it closed and its queue is empty, so its data still goes before its end, and the end goes once, so it cannot hold back the other stream's output. A deterministic test closes one stream at once, and before every message hands the runner a chunk of the other stream with the credit for it. It checks, for each stream, that the closed stream's end goes out right after the start, while the other stream's output keeps flowing, and fails without this change. The test of the input credit's turns now expects standard error's end, which closed with no byte, before standard output's data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

Binding::next returned the runner's next message as soon as one waited, and took a step of the execution only when none did. A caller that takes one event at a time, and hands the binding more input before every call, which the program takes, frees input credit in every call: the binding then returned only credit, and never read the program's output, however long the input flowed. Once the binding returned a message, the execution's turn now comes before the next: it takes the execution's next step, if it has one, and starts over. The writes of a call leave a step for that turn while it is due, so a call still takes at most STEPS steps, and every request that a step raises is still applied before the call returns. A test of a real program that writes its output once the peer knows that it started, then reads its input to its end, hands the binding a chunk of input before every single call, within a page that any pipe takes whole, and checks that the output arrives before that input runs out. It fails without this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

Binding::next reported the program's end to the runner only once the launch report had reached it. A process that another thread forks while the program spawns holds a copy of the report's close-on-exec writer until it replaces itself with its program or ends, so the report can arrive after the program's end. A call in between fired the runner's timeout: the outcome named a timeout for a program that had ended on its own, and if the report came only after the grace, its output was discarded. The binding now reports the program's end once the execution has its status, whether or not the launch report has arrived. The runner already takes the end before the launch result: it disarms the timeout, requests the settlement, and holds the program's start, its output, and the terminal outcome until the result arrives. Runner::launched documents that order. A test-only Command::spawn_holding_report hands a test a copy of the launch report's writer. A test holds it while the program ends and the first calls come after the runner's timeout, whoever collects the status, then drops it, and checks that the outcome names no cause and the output arrived whole. It fails without this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

The binding's test of an execution that does not settle lets its program end once its output fits in the stalled sink's window, the runner's queue, and its pipe. `seq 1 8000` left 6,125 bytes for the pipe, more than one page, the least that a pipe holds, which a user beyond the kernel's limit on pipe pages may get: the program then never ended, and the test ran out of patience. `seq 1 7000` leaves 1,125 bytes for the pipe, as the other tests of the binding rely on no more than one page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

both_streams_take_turns_while_each_has_output drives the execution until the program's status is known, then expects only output, ends, and the outcome. The launch report can arrive after the status: a process that another test's thread forks while the program spawns holds the report's close-on-exec writer until it replaces itself with its program or ends. The launch event then came in the second loop, which failed the test. The first loop now waits for the launch report too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

The binding reports the program's end before its launch report, which a process that another thread forked while the program was spawning may hold back. The end disarms the runner's timeout, but nothing bounded the wait for the report after it: if that process held the report's writer for good, the runner never returned its terminal outcome, whether or not the execution settled, and its deadline was none once the settlement time and the grace had passed, unless the peer cancelled or timed out. The runner now leaves a launch result that has not arrived the settlement time and then the grace after the workload's end, the instant at which it already gives up on the open streams of an execution that does not settle. Then it gives up on the result, which fences the execution, as its cleanup is incomplete, and with it on the streams still open but for those that hung up. It sends the workload's output, without a start, and the terminal outcome, with the workload's status and no cause. A result that arrives before then still counts. Engine tests hold the launch result back for good, as the execution settles, its settlement fails, or never comes, with a timeout of the runner's own that the workload's end disarmed, and release it just before the runner would give up on it. A binding test holds a copy of the report's writer throughout, with and without a cgroup. The tests that hold it for good fail without this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

Runner::launched took the first launch result whenever it came, even once the runner had given up on it, after a termination or after the workload's end, and had sent output or a stream's end without it. A successful result that arrived then, before the terminal outcome, queued the workload's start after them, so the start followed the output. An adapter keeps polling what reports the launch, so a process that held the report back and lets it go while the runner drains the workload's output causes that. The runner now ignores a launch result once it gave up on it, as it ignores any result after the first: whatever was still unknown stays so. The engine test of a launch result that a process holds back after the workload's end now delivers it after the runner sent output and an end, before the terminal outcome, and a new test does so after a termination's give-up. Both fail without this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72a079e6-8bb1-45be-9aaf-ddd8dd701c43

Copilot

🔵 Needs a closer look

The interactions between credit flow, termination, and cgroup reuse warrant final human review despite no additional confirmed blocking findings.

No open issues 🎉

_{Review effort: Balanced}

Copilot AI balanced review requested due to automatic review settings October 22, 2026 06:53

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 22, 2026 06:54

View session