diff --git a/.github/actions/check-nvx-rs/action.yml b/.github/actions/check-nvx-rs/action.yml index cad79c383..a1b40e420 100644 --- a/.github/actions/check-nvx-rs/action.yml +++ b/.github/actions/check-nvx-rs/action.yml @@ -62,6 +62,9 @@ runs: set -euo pipefail cargo "+${{ steps.toolchains.outputs.toolchain }}" test --locked --all-features cargo "+${{ steps.toolchains.outputs.toolchain }}" test --locked + # Without features, the library still holds the protocol and the execution engine. + cargo "+${{ steps.toolchains.outputs.toolchain }}" build --locked --lib --no-default-features + cargo "+${{ steps.toolchains.outputs.toolchain }}" test --locked --lib --no-default-features # Stages a synthetic release in the current layout, with nvx-rs and its # provenance, which the bundled-artifacts test then requires to find. diff --git a/nvx-rs/README.md b/nvx-rs/README.md index 59bfe7693..ba11a2196 100644 --- a/nvx-rs/README.md +++ b/nvx-rs/README.md @@ -36,6 +36,8 @@ nvx-rs/ │ │ by the host and the guest │ ├── hypervisor.rs │ ├── error.rs nvx::Error and its ErrorKind +│ ├── exec/ the execution engine: the streams, credit, and outcome of an execution's two +│ │ sides, without I/O, threads, or features, for any framing and host │ ├── host/ feature `host` (default): artifacts, launch, network and egress policy files, │ │ filesystem, NVXS connection, session, process, platform, VM instances, │ │ sandbox stores, outcome reports, file seals, and the sandbox launch profile @@ -44,12 +46,12 @@ nvx-rs/ │ ├── testing/ feature `testing`: the fake OpenVMM │ ├── guest/ feature `guest`: the programs that run in Linux guests, the guest's NVXS │ │ connection, the libraries through which a guest init or agent opens the -│ │ terminal on which it serves its session, starts a program, collects its -│ │ children's statuses, settles its executions' cgroups, mounts host mappings -│ │ beneath the descriptors of the roots that it names, assembles and tears down -│ │ a layered root, and reads a GPT disk's partition table through its block -│ │ device, and the typed calls through which it runs the time ABI's steps and -│ │ the reseed +│ │ terminal on which it serves its session, starts a program, drives its +│ │ execution, alone or through the execution engine, collects its children's +│ │ statuses, settles its executions' cgroups, mounts host mappings beneath the +│ │ descriptors of the roots that it names, assembles and tears down a layered +│ │ root, and reads a GPT disk's partition table through its block device, and +│ │ the typed calls through which it runs the time ABI's steps and the reseed │ └── bin/ nvx-rs, nvx-fake-openvmm, and the guest binaries ├── tests/ fake-OpenVMM, CLI, NVXS session, process-supervision, store, and guest-binary │ tests, the store's record fixtures, and the golden fixtures that Python @@ -70,6 +72,7 @@ nvx-rs/ | `nvx::gpt` | `Disk::read`: the protective MBR, primary header, and primary partition entry array of a GPT disk, read through a `ReadAt` source that the caller gives and checked strictly, as UEFI specifies them and as Linux requires them, without ever reading the backup: 128-byte entries, at most 1 MiB of them, used entries first and without holes, a first partition and 1 to 254 layers, extents in the usable blocks without overlap, and unique GUIDs, with checked arithmetic in the disk's logical blocks. `Expected` adds the disk GUID and the partitions' unique GUIDs and sizes that a caller knows, and `Disk::read_first_partition` returns the first partition's opaque bytes, up to a bound and without trailing NUL bytes and whitespace. GUIDs are `nvx::uuid::Uuid`s. It has no dependencies, so the guest and the host's checks of the images that it stages share it | | `nvx::uuid` | `Uuid`: a UUID, such as the GUID of a GPT disk or partition or the UUID of a filesystem, in the order of its canonical text, the one text that it displays and parses: 32 lowercase hexadecimal digits in groups of 8, 4, 4, 4, and 12, joined by hyphens. `from_bytes_le` and `to_bytes_le` convert the layout in which GPT and other EFI GUIDs store their first three groups little-endian. It has no dependencies, so the host and the guest share it | | `nvx::Hypervisor` | KVM, MSHV, or WHP, with its serde form behind the `serde` feature | +| `nvx::exec` | The execution engine: the stream state, credit, ordering, and outcome of one execution, for any framing that carries its messages, without I/O, threads, global state, or features, so that it builds with `--no-default-features`. A `Runner` is the side that runs the workload: it reads the workload's output under the peer's credit, holds the input that the peer sent within its window until the workload takes it, requests the terminations, discards, input close, and settlement that its adapter applies, and returns one terminal `Outcome`, which tells whether the execution is reusable or fenced. A `Peer` is the other side: it takes output into windows that its sink empties, returns credit as the sink takes bytes, sends input within the runner's window, and ends the execution on a cancellation, a timeout, a drop, a reset, or a disconnection. Both take the current `Instant` from the caller, report their next deadline, and never wait, so a blocked stream, sink, window, or transport stops no timer, cancellation, or other execution, and both bound their memory. The adapter chooses one byte window per stream, or frames charged against one pool that stdout and stderr share, and the maximum chunk, in `Limits`; a `Key`'s generation tells stale messages from foreign ones | | `nvx::host::artifacts` | `Artifacts` (also `nvx::Artifacts`): the OpenVMM executable, guest kernel, and Alpine initramfs, from a release, a repository checkout, the environment, a directory next to the executable, or the bundle. Artifacts located in a release or checkout keep its `ArtifactDir`, and `Artifacts::runtime_identity` reads their `RuntimeIdentity` from its install metadata; artifacts named by path have none. `ArtifactDir` and `Guest`: every artifact of the guests that `nvx.py` runs, the OpenVMM executable, the kernel, the Alpine, Ubuntu, and Azure Linux initramfs images and their package manifests, and the Ubuntu distro layer and its manifest, in a checkout's `build/` and `openvmm/target/release/` or a release's `guest/` and `bin/`, checked in the order and with the messages of `nvx.py`. `ArtifactPaths` lists them all and serializes as JSON. `ArtifactDir::runtime_identity` reads the `RuntimeIdentity` of the OpenVMM executable, kernel, and Alpine initramfs from the install metadata, a release's `SHA256SUMS` or a checkout's build provenance, without hashing an artifact | | `nvx::host::launch` | `LaunchSpec` and OpenVMM's command line: the managed lifecycle, the `WorkloadIdentity`, the 1024-byte kernel command line, its reserved parameters, the 32,767-character limit of a Windows command line, and `LaunchSpec::check`, which every command line applies to the memory size, the non-root identity, the hostname, the guest network, and the kernel parameters | | `nvx::host::network` | `NetworkPolicy` and `openvmm_arguments`, the one lowering of a network posture into OpenVMM options, for the CLI and external orchestrators: the egress default and rules, ingress, `HostLoopback` access, denied, `Outbound` from the guest alone, or both ways, and its forwards. A proxy-only policy, `NetworkPolicy::proxy_only`, has one way out, a `NetworkProxy` that listens on a port of host loopback, which the guest reaches at the same port of its gateway, and whose URL as the guest sees it `NetworkProxy::url` gives; the lowering refuses a proxy beside any other way out. Egress rules are lowered exactly into OpenVMM's IPv4 and IPv6 rules by the lowering of egress policy files, with at most 256 rules for each action | @@ -93,6 +96,7 @@ nvx-rs/ | `nvx::guest::process` | `Child`: the status of one child that the caller names, by its pidfd or its process ID. It waits for that child alone, so another child's status stays for the caller; a caller that collects every status itself, such as a PID 1 that adopts orphans, hands each one to its `Child` with `observe`. Statuses are `std::process::ExitStatus`, real-time signals included | | `nvx::guest::cgroup` | `Layout`: the agent's cgroup and the workload cgroup, which the caller names, and which may not hold the agent's. `ExecutionCgroup`: the cgroup of one execution, the workload cgroup itself or a child of it, which it prepares with the caller's resource values, places processes in, checks for occupancy through `pids.current`, kills with `cgroup.kill`, and settles, over the execution's cgroup or the whole workload subtree, as the caller's `Scope` chooses. It holds its cgroups' directories open, and the process's removals and preparations of cgroups take turns, so that it never reaches another cgroup that takes one of their paths. Settling calls the caller's status collector between its checks, and collects no status itself | | `nvx::guest::exec` | `Execution`: drives one program that the caller spawned with the program's ends of its `Pipes`, without blocking, from the caller's own loop or reactor, and reports each step as an `Event`: the launch report, the output of each stream, kill failures, and the `Outcome`. It reads a stream only as far as the caller's `grant` allows, so a stalled sink stalls its own execution alone, writes the program's standard input without `SIGPIPE`, and ends the program on a cancellation, a timeout, or a disconnection, with `SIGKILL` through the program's pidfd to the program and, if the caller asks, its process group, while no collector has taken its status, and, through the opened directory of a cgroup that the caller names, to that cgroup. It closes the program's input once the program has ended. It waits for its program alone, by a pidfd, or takes the status from the caller's collector with `observe`; with `Options::caller_reaps`, that collector is the program's only one. NVX's managed agent is one caller | +| `nvx::guest::exec::binding` | `Binding`: an `nvx::exec::Runner` bound to an `Execution`, the glue that a guest adapter would otherwise write. The runner's read allowances become grants, its discards discards, the input that it accepted writes and then a close, its terminations a cancellation, a disconnection, or a termination, and its settlement a kill of the execution's cgroup and checks until no process occupies it; without a cgroup, nothing proves that the program's descendants are gone, so the execution is fenced. It feeds the execution's events back to the runner, statuses that a caller who reaps hands over with `observe` included, and never blocks: the caller polls its descriptors and its input writer until its deadline, and each call takes a bounded number of steps, so that no execution holds the caller's loop | | `nvx::guest::time` | x86_64: `TimeAbi`, the steps of NVX time ABI v1 as typed calls in the caller's process, for any guest init or agent: the cold-boot time step and, separately, the start of the time daemon in a child process that the caller supervises, the VM generation ID, a `CaptureRequest` that repairs a `Restore`, and a `RestoreFinish`. A call takes the permission for OpenVMM's ports on the calling thread for each port transaction alone, and fails with `TimeError::PortRelease` if the thread may still hold some of it, which a process that it started would inherit, and from which `nvx::guest::spawn` starts none. No call waits for a child that it did not start. `nvx-time` runs its steps through them | | `nvx::guest::reseed` | `Seed::read` and `reseed`: a restore's fresh entropy, which starts with its `GenerationId`, and the reseed of the kernel's random number generator with it, in the caller's process. `nvx-reseed` runs them | | `nvx::guest::root` | The operations of a layered root, for a guest init or agent, whether or not it is PID 1. `bootstrap`: the agent's and the workload's sibling cgroups of a `Layout`, with every process of their parent moved into the agent's, the agent's `memory.low`, and the workload cgroup prepared through `ExecutionCgroup::prepare`, with its `memory.max` and `pids.max`, and user namespaces kept unavailable. `assemble`: the `Assembly`'s block devices, found by their MMIO windows, checked against their expected sizes and flushed, and the `Root` that it mounts, at a runtime directory, a canonical path that no symbolic link leads through, that it holds exclusively until its teardown, where nothing is mounted already: a read-only EROFS image, whose expected `nvx::uuid::Uuid` it checks in its superblock, under an overlay with `metacopy=on,xino=on`, mounted `nosuid,nodev`, whose upper and work directories are on an ext4 scratch mounted `nosuid,nodev` too. `Root::mount_share` and `Root::mount_aggregate`: virtio-fs live shares at targets in the root that it checks, against the targets that `protocol::share` reserves among others, with one aggregate per root, which it attaches through its `mappings::Mapper` beneath the overlay, with NVX's container root's reserved targets: a share's device mounted out of the root, then moved onto its target, and a copy of each child's entry of the aggregate. `Root::map`: a caller's host mappings, through the same mapper. A share, a child, or a mapping that fails latches the root. `Root::teardown`: the shares and mappings in reverse order, then an aggregate's own mount, the overlay, the image, and the scratch, with every failed unmount reported. Each failure is a typed `Error`. The operations collect no child's status, change neither the caller's root nor its credentials, start no async runtime, and never power the VM off | @@ -585,9 +589,9 @@ runtime identity. | `serde` | The serde form of `Hypervisor` | | `guest` | The `guest` module and the guest binaries; see [Guest binaries](#guest-binaries) | -Without default features, the crate has only `protocol`, `contract`, `net`, -`gpt`, `uuid`, `Hypervisor`, and `Error`, and depends on nothing but `serde` -when `serde` is enabled and `nix` and `getrandom` when `guest` is; +Without default features, the crate has only `protocol`, `exec`, `contract`, +`net`, `gpt`, `uuid`, `Hypervisor`, and `Error`, and depends on nothing but +`serde` when `serde` is enabled and `nix` and `getrandom` when `guest` is; `aci_edge_sandboxes_model` uses it that way. ## Example @@ -773,7 +777,11 @@ The process-supervision test crashes orchestrator processes at each step of a launch through the supervisor, reattaches to the VMM stand-ins that they leave behind and stops them, and checks that a VMM inherits only its standard streams and the descriptors or handles that it is allowed while another thread spawns -processes. +processes. The tests of `nvx::exec` drive a runner and a peer through a +simulated workload, transport, and sink on a clock of their own, and, on Linux, +the tests of `nvx::guest::exec::binding` drive real programs through them, two +at once in one loop, one of them with a stalled sink; run as root, one of them +also settles a real cgroup. On Linux, a loopback test drives the host `ControlSession` through a broker shim into the real managed agent's session, dispatch, and execution engine, and `nvx.py` also lints and tests the @@ -788,11 +796,12 @@ python3 ../scripts/nvx.py test-guest-binaries \ The [`check-nvx-rs`](../.github/actions/check-nvx-rs/action.yml) action runs rustfmt, Clippy with warnings denied for every feature and for the library -without default features, the tests, a synthetic-release `bundled` test, -rustdoc with warnings denied, the minimum supported Rust version (1.89), and a -macOS check. On Linux, it also lints, builds, and tests the guest binaries -through `nvx.py`. `python3 scripts/nvx.py test-aci-edge-sandboxes --backend kvm` -exercises the crate on a real hypervisor through `aci_edge_sandboxes`. +without default features, the tests, a build and the tests of the library +without default features, a synthetic-release `bundled` test, rustdoc with +warnings denied, the minimum supported Rust version (1.89), and a macOS check. +On Linux, it also lints, builds, and tests the guest binaries through `nvx.py`. +`python3 scripts/nvx.py test-aci-edge-sandboxes --backend kvm` exercises the +crate on a real hypervisor through `aci_edge_sandboxes`. `tests/managed_lifecycle.rs` is an ignored test of the managed lifecycle of a guest whose init hands the boot to the managed agent, on a real hypervisor: diff --git a/nvx-rs/src/exec/credit.rs b/nvx-rs/src/exec/credit.rs new file mode 100644 index 000000000..08eb662a6 --- /dev/null +++ b/nvx-rs/src/exec/credit.rs @@ -0,0 +1,206 @@ +//! The credit arithmetic of an execution's pools, and the bounded queues of its streams. +//! +//! A sender holds [`Credit`]: the units of a pool that the receiver granted it and that it has +//! not used, which it spends on each data message it sends. A receiver holds [`Granted`]: the +//! units that it granted and that the sender has not used yet, which bound each data message it +//! accepts, and the units that its consumer freed, which it grants again. Both start from the +//! pool's window: the receiver's first grant opens the whole window, and a grant can never open +//! more than the window, so the units in flight never exceed it. + +use std::collections::VecDeque; + +use super::{Unit, Violation, Window}; + +/// The credit that a receiver granted the sender of one pool. +#[derive(Debug)] +pub(super) struct Credit { + /// The pool's window. + window: Window, + /// The units that the sender may still send. + available: u32, +} + +impl Credit { + /// Returns the credit of a pool with `window`, before the receiver's first grant. + pub(super) fn new(window: Window) -> Self { + Self { + window, + available: 0, + } + } + + /// Adds a grant of `units`. A grant of no units, or of more than the units in flight, which + /// would leave the sender more than the window, is refused and changes nothing. + pub(super) fn grant(&mut self, units: u32) -> Result<(), Violation> { + if units == 0 + || u64::from(self.available) + u64::from(units) > u64::from(self.window.size()) + { + return Err(Violation::Credit); + } + self.available += units; + Ok(()) + } + + /// Returns how many of the first `length` queued bytes one data message may carry now: as + /// many as the bytes of credit, or all of them while a frame of credit is left. + pub(super) fn sendable(&self, length: usize) -> usize { + match self.window.unit() { + Unit::Bytes => length.min(self.available as usize), + Unit::Frames if self.available > 0 => length, + Unit::Frames => 0, + } + } + + /// Spends the credit of one data message of `length` bytes, which + /// [`sendable`](Self::sendable) allowed. + pub(super) fn spend(&mut self, length: usize) { + self.available -= self.window.cost(length); + } +} + +/// What the receiver of one pool granted, and what its consumer freed. +#[derive(Debug)] +pub(super) struct Granted { + /// The pool's window. + window: Window, + /// The units that the sender may still send: granted, and not used yet. + open: u32, + /// The units that the consumer freed, or that the receiver never granted, which it grants + /// next. + returnable: u32, +} + +impl Granted { + /// Returns the window of a pool before its first grant, which opens the whole window. + pub(super) fn new(window: Window) -> Self { + Self { + window, + open: 0, + returnable: window.size(), + } + } + + /// Charges a data message of `length` bytes, which the caller checked against the maximum + /// chunk. A message beyond the units that remain open is refused and changes nothing. + pub(super) fn charge(&mut self, length: usize) -> Result<(), Violation> { + let cost = self.window.cost(length); + if cost > self.open { + return Err(Violation::Window); + } + self.open -= cost; + Ok(()) + } + + /// Frees `units` that the consumer took, to grant them again. + pub(super) fn free(&mut self, units: u32) { + self.returnable += units; + } + + /// Returns the units to grant now, if any, and opens them. + pub(super) fn take_grant(&mut self) -> Option { + let units = std::mem::take(&mut self.returnable); + self.open += units; + (units > 0).then_some(units) + } +} + +/// The bytes that one stream's receiver holds for its consumer, in order, with the bounds of the +/// frames that they arrived in when the pool counts frames. +#[derive(Debug)] +pub(super) struct Inbox { + /// What the pool counts. + unit: Unit, + /// The bytes, in order. + bytes: VecDeque, + /// The bytes left of each frame that the inbox holds, oldest first, when the pool counts + /// frames. + frames: VecDeque, +} + +impl Inbox { + /// Returns an empty inbox of a pool that counts `unit`. + pub(super) fn new(unit: Unit) -> Self { + Self { + unit, + bytes: VecDeque::new(), + frames: VecDeque::new(), + } + } + + /// Returns the number of bytes held. + pub(super) fn len(&self) -> usize { + self.bytes.len() + } + + /// Returns whether the inbox holds nothing. + pub(super) fn is_empty(&self) -> bool { + self.bytes.is_empty() + } + + /// Appends one data message, which its pool charged. + pub(super) fn push(&mut self, bytes: &[u8]) { + self.bytes.extend(bytes); + if self.unit == Unit::Frames { + self.frames.push_back(bytes.len()); + } + } + + /// Returns the bytes at the front, as many as are contiguous: a part of what the inbox holds + /// while it holds anything. + pub(super) fn front(&self) -> &[u8] { + self.bytes.as_slices().0 + } + + /// Removes the first `count` bytes, at most as many as the inbox holds, and returns the units + /// that this frees: each byte, or each frame whose last byte it removes. + pub(super) fn pop(&mut self, count: usize) -> u32 { + let count = count.min(self.bytes.len()); + self.bytes.drain(..count); + match self.unit { + Unit::Bytes => units(count), + Unit::Frames => { + let mut left = count; + let mut freed = 0; + while left > 0 { + let Some(frame) = self.frames.front_mut() else { + break; + }; + let taken = left.min(*frame); + *frame -= taken; + left -= taken; + if *frame == 0 { + self.frames.pop_front(); + freed += 1; + } + } + freed + } + } + } + + /// Removes everything, releases the storage, and returns how many bytes it removed and the + /// units that this frees. + pub(super) fn clear(&mut self) -> (usize, u32) { + let length = self.bytes.len(); + let freed = self.pop(length); + self.bytes = VecDeque::new(); + (length, freed) + } +} + +/// Returns `count` bytes as units of a byte pool, which a chunk's bound keeps within `u32`. +pub(super) fn units(count: usize) -> u32 { + u32::try_from(count).unwrap_or(u32::MAX) +} + +/// Moves the first bytes of `queue` into `buffer`, at most `length` of them, and returns how +/// many it moved. +pub(super) fn take(queue: &mut VecDeque, buffer: &mut [u8], length: usize) -> usize { + let length = length.min(buffer.len()).min(queue.len()); + let (first, second) = queue.as_slices(); + let head = length.min(first.len()); + buffer[..head].copy_from_slice(&first[..head]); + buffer[head..length].copy_from_slice(&second[..length - head]); + queue.drain(..length); + length +} diff --git a/nvx-rs/src/exec/mod.rs b/nvx-rs/src/exec/mod.rs new file mode 100644 index 000000000..90992d6c1 --- /dev/null +++ b/nvx-rs/src/exec/mod.rs @@ -0,0 +1,715 @@ +//! The credit and stdio engine of an execution, which every adapter shares, whatever its framing. +//! +//! An execution runs a workload on one side, the [`Runner`], for another side, the [`Peer`], +//! which sends its standard input and takes its standard output and error. The engine owns what +//! both sides must agree on: the state of each stream, the credit arithmetic that bounds the +//! bytes in flight, the order of each stream's data and end, and the execution's outcome. The +//! adapter owns everything else: its framing and transport, which messages it puts in one frame, +//! its admission of executions, and the processes, which a guest's adapter drives through +//! `guest::exec::binding` (Linux, feature `guest`). +//! +//! The engine needs only the standard library: no feature enables it, and it has no wire format, +//! async runtime, threads, global state, admission, or reaper. Each side is a state machine that +//! never blocks and never waits: the adapter hands it each input as it arrives, with the current +//! [`Instant`] where time matters, retrieves what it must do, one bounded item at a time, and +//! [ticks](Runner::tick) it once its [deadline](Runner::deadline) passes. So a stream, a sink, a +//! window, or a transport that blocks holds back only the bytes that wait on it, never a timer, a +//! cancellation, a reset, or another execution, and tests drive both sides deterministically. +//! +//! # Messages +//! +//! The two sides exchange these messages, each of which the adapter frames and tags with the +//! execution's [`Key`]: +//! +//! | From | Message | Meaning | +//! | --- | --- | --- | +//! | Peer | [credit](peer::Message::Credit) for output | the peer's consumer freed output units | +//! | Peer | [input](peer::Message::Input) | bytes of the workload's standard input | +//! | Peer | [end of input](peer::Message::InputEnd) | the input ends after its last byte | +//! | Peer | [cancel](peer::Message::Cancel) | end the workload | +//! | Runner | [credit](runner::Message::Credit) for input | the workload took input units | +//! | Runner | [started](runner::Message::Started) | the workload's program runs | +//! | Runner | [output](runner::Message::Output) | bytes of standard output or error | +//! | Runner | [end](runner::Message::End) | the stream ends after its last byte | +//! | Runner | [input closed](runner::Message::InputClosed) | the workload's input closed early | +//! | Runner | [terminal](runner::Message::Terminal) | the execution's [`Outcome`] | +//! +//! A message for an earlier generation of the execution's key is [stale](Receipt::Stale), and one +//! that the receiving side no longer needs, such as a credit after the execution ended, or any +//! message of the runner's once the peer detached from it or gave up waiting for it, is +//! [late](Receipt::Late): neither changes anything. A message that breaks the rules, such as data +//! beyond the window, is a [`Violation`], which changes nothing either, and after which the +//! adapter decides what to do, such as reset its session. +//! +//! # Credit +//! +//! Each data message carries at most [`Limits::chunk`] bytes, and costs credit in its pool: as +//! many units as its bytes, or one unit whatever its length, as the pool's [`Window`] counts. The +//! adapter chooses the pools: the input's, and either one window for each output stream, or one +//! pool that standard output and error [share](Limits::shared_output). Each receiver grants its +//! whole window first, then each unit again once its consumer took the unit's bytes: the runner +//! once the workload's input took them, the peer once its sink took them. A receiver refuses data +//! beyond what it granted, and a sender refuses a grant that would leave it more than the window, +//! so at most a window of units is ever in flight, and each side's buffers stay bounded, with no +//! limit on a stream's total length: see [`Runner::memory_bound`] and [`Peer::memory_bound`]. +//! +//! A sender queues at most one chunk of each stream ahead of the credit, which the runner reads +//! from the workload as its [read allowance](Runner::read_allowance) lets it, and the peer takes +//! from its source as its [write allowance](Peer::write_allowance) does. Data leaves the queue as +//! the adapter retrieves it, as one message of whatever the queue holds, up to a chunk: an +//! adapter whose transport is busy retrieves less often, and so sends fewer, fuller messages. +//! +//! # Completion +//! +//! - The workload's input ends only after every byte that the runner accepted: the runner +//! requests to [close it](runner::Request::CloseInput) once the peer ended it and the workload +//! took the rest. A workload that closes its input early, or ends, makes the runner tell the +//! peer that the input [closed](runner::Message::InputClosed), and drop what the peer sends +//! after, so that the peer's source never waits. Closing the input never closes the other +//! direction. +//! - Each stream keeps its order, and its end follows its last byte. A stream's end goes ahead +//! of any later output of the other stream, which never holds it back; otherwise, the streams' +//! messages have no order between them. The runner sends the terminal outcome after both +//! streams' ends. +//! - A cancellation, a timeout, a drop, a reset, or a disconnection proceeds whatever the streams +//! do. The runner requests to [end](runner::Request::Terminate) the workload, and pending output +//! still flows for [`Timers`]'s grace, then the rest is discarded, and the terminal outcome +//! marks each stream that lost bytes as [truncated](Outcome::is_truncated). A reset, a drop, or +//! a disconnection of the runner leaves no peer to deliver to, so the runner discards at once. +//! A workload that ends on its own, with no such cause, loses no byte, however slowly the peer +//! takes them, if its execution settles, or if no process holds its streams any more, as the +//! adapter tells the runner once they [hung up](Runner::output_hung_up). +//! - The logical outcome, the delivery of the output, and the settlement of the execution are +//! separate. Once the workload's status is known, the runner requests to +//! [settle](runner::Request::Settle) its processes, whose result the adapter reports once its +//! own check shows that the workload's descendants, and every process that holds one of its +//! pipes, are gone; it may stop checking once the runner no longer +//! [awaits](Runner::awaits_settlement) the result. The workload's exit, a signal to its process +//! group, or a peer's acknowledgement never counts. The terminal outcome follows the status, +//! the delivery, and the settlement, and says whether the execution is +//! [reusable](Outcome::is_reusable) or fenced: a failed or late settlement fences it. +//! - Each side bounds every wait: a termination gives the workload its grace and then +//! [`Timers`]'s settlement time to end, close its streams, and settle, after which the runner +//! gives up on an execution whose workload has not, and fences it, while one whose cleanup is +//! complete waits only for the transport to take its last messages; a workload that ended on +//! its own, but whose execution did not settle, leaves its streams the settlement time and then +//! the grace after its end to close or hang up, after which the runner gives up on those still +//! open that a process may hold, sends what it read of them, and marks them truncated; a launch +//! result that has not arrived by then fences the execution, whether or not it settled, as a +//! process may hold what reports it too, and the runner gives up on the result, and on those +//! streams, and sends the workload's output and outcome without it; a peer that ends the +//! execution itself waits as long for the terminal outcome, after which it gives up and fences +//! it too; and a peer that receives a terminal outcome with a cause, without having ended the +//! execution itself, gives its sink the grace from then on to take the output that it holds, +//! and discards the rest. +//! +//! # Example +//! +//! A peer and a runner, connected directly, as an adapter's framing and transport would connect +//! them: +//! +//! ``` +//! use std::time::{Duration, Instant}; +//! +//! use nvx::exec::{ +//! peer, runner, Key, Limits, Peer, Runner, Settlement, Status, Stream, Timers, Window, +//! }; +//! +//! let now = Instant::now(); +//! let key = Key::new(1, 0); +//! let window = Window::bytes(64 * 1024); +//! let limits = Limits::new(16 * 1024, window, window).unwrap(); +//! let timers = Timers::new(Duration::from_secs(1), Duration::from_secs(5)); +//! let mut peer = Peer::new(key, limits, timers, now); +//! let mut runner = Runner::new(key, limits, timers, now); +//! let mut buffer = vec![0u8; limits.chunk()]; +//! +//! // The workload runs, writes to its standard output, and exits; its processes settle. +//! runner.launched(Ok(())); +//! assert_eq!(runner.output(Stream::Stdout, b"hello"), 5); +//! runner.output_closed(Stream::Stdout); +//! runner.output_closed(Stream::Stderr); +//! runner.exited(now, Some(Status::Exited(0))); +//! assert_eq!(runner.next_request(), Some(runner::Request::Settle)); +//! runner.settled(Settlement::Settled); +//! +//! // The adapter carries each side's messages to the other until the peer finishes. +//! let mut stdout = Vec::new(); +//! let outcome = loop { +//! while let Some(message) = peer.next_message(&mut buffer) { +//! match message { +//! peer::Message::Credit { pool, units } => { +//! runner.credit(key, pool, units).unwrap(); +//! } +//! message => unreachable!("{message:?}"), +//! } +//! } +//! while let Some(message) = runner.next_message(&mut buffer) { +//! match message { +//! runner::Message::Credit { units } => { +//! peer.credit(key, units).unwrap(); +//! } +//! runner::Message::Started => { +//! peer.started(key).unwrap(); +//! } +//! runner::Message::Output { stream, length } => { +//! peer.output(key, stream, &buffer[..length]).unwrap(); +//! } +//! runner::Message::End(stream) => { +//! peer.end_output(key, stream).unwrap(); +//! } +//! runner::Message::Terminal(outcome) => { +//! peer.terminal(now, key, outcome).unwrap(); +//! } +//! runner::Message::InputClosed => { +//! peer.input_closed(key).unwrap(); +//! } +//! message => unreachable!("{message:?}"), +//! } +//! } +//! // The peer's sink takes what arrived. +//! let pending = peer.pending_output(Stream::Stdout).to_vec(); +//! stdout.extend_from_slice(&pending); +//! peer.consumed(Stream::Stdout, pending.len()); +//! let last = std::iter::from_fn(|| peer.next_event()).last(); +//! if let Some(peer::Event::Finished(outcome)) = last { +//! break outcome; +//! } +//! }; +//! assert_eq!(stdout, b"hello"); +//! assert_eq!(outcome.status(), Some(Status::Exited(0))); +//! assert!(outcome.is_reusable() && !outcome.is_truncated(Stream::Stdout)); +//! ``` + +use std::fmt; +use std::time::{Duration, Instant}; + +mod credit; +pub mod peer; +pub mod runner; + +#[cfg(test)] +mod tests; + +pub use peer::Peer; +pub use runner::Runner; + +/// The identity of one execution: an identifier that the adapter assigns, such as a request ID or +/// a stream slot, and the generation of that identifier, which tells a later execution with the +/// same identifier, or the same execution in a later session, from an earlier one. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub struct Key { + /// The execution's identifier. + id: u64, + /// The identifier's generation. + generation: u64, +} + +impl Key { + /// Returns the key of execution `id` in `generation`. + pub const fn new(id: u64, generation: u64) -> Self { + Self { id, generation } + } + + /// Returns the execution's identifier. + pub const fn id(&self) -> u64 { + self.id + } + + /// Returns the identifier's generation. + pub const fn generation(&self) -> u64 { + self.generation + } + + /// Checks the key of a message for this execution: `Ok(true)` for its own generation, + /// `Ok(false)` for an earlier one, whose messages are stale, and [`Violation::Key`] for + /// another identifier or a later generation. + fn admits(self, message: Key) -> Result { + if message.id != self.id { + return Err(Violation::Key); + } + match message.generation.cmp(&self.generation) { + std::cmp::Ordering::Less => Ok(false), + std::cmp::Ordering::Equal => Ok(true), + std::cmp::Ordering::Greater => Err(Violation::Key), + } + } +} + +/// One of the workload's output streams. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Stream { + /// The workload's standard output. + Stdout, + /// The workload's standard error. + Stderr, +} + +impl Stream { + /// Both streams, in the order in which the sides first serve them. + pub const ALL: [Self; 2] = [Self::Stdout, Self::Stderr]; + + /// Returns the stream's position in [`ALL`](Self::ALL). + const fn index(self) -> usize { + match self { + Self::Stdout => 0, + Self::Stderr => 1, + } + } +} + +/// A pool of credit, which a [credit message](peer::Message::Credit) names. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Pool { + /// The workload's standard input. + Input, + /// The workload's standard output, which has a pool of its own unless the output streams + /// [share](Limits::shared_output) one. + Stdout, + /// The workload's standard error, which has a pool of its own unless the output streams share + /// one. + Stderr, + /// The one pool that standard output and error [share](Limits::shared_output). + Output, +} + +/// What a pool's credit counts. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Unit { + /// Data bytes: a data message costs as many units as its bytes. + Bytes, + /// Data messages: a data message costs one unit, whatever its length up to a chunk. + Frames, +} + +/// The size of a pool's window: the most units that may be in flight at once. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Window { + /// What the window counts. + unit: Unit, + /// The units of the window. + size: u32, +} + +impl Window { + /// Returns a window of `size` bytes. + pub const fn bytes(size: u32) -> Self { + Self { + unit: Unit::Bytes, + size, + } + } + + /// Returns a window of `size` data messages. + pub const fn frames(size: u32) -> Self { + Self { + unit: Unit::Frames, + size, + } + } + + /// Returns what the window counts. + pub const fn unit(&self) -> Unit { + self.unit + } + + /// Returns the units of the window. + pub const fn size(&self) -> u32 { + self.size + } + + /// Returns the units that a data message of `length` bytes costs. + fn cost(&self, length: usize) -> u32 { + match self.unit { + Unit::Bytes => credit::units(length), + Unit::Frames => 1, + } + } + + /// Returns the most data bytes that the window holds with chunks of `chunk` bytes. + fn bytes_with(&self, chunk: usize) -> usize { + match self.unit { + Unit::Bytes => self.size as usize, + Unit::Frames => (self.size as usize).saturating_mul(chunk), + } + } +} + +/// The limits on which both sides of an execution agree: the maximum chunk, the input's window, +/// and the output's windows, one for each stream unless the streams share one pool. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Limits { + /// The most data bytes that one data message carries. + chunk: u32, + /// The window of the workload's standard input. + input: Window, + /// The window of each output pool. + output: Window, + /// Whether standard output and error share one pool. + shared: bool, +} + +impl Limits { + /// Returns limits with data messages of at most `chunk` bytes, the input's `input` window, + /// and one `output` window for each output stream, or `None` if the chunk or a window is + /// empty. + pub const fn new(chunk: u32, input: Window, output: Window) -> Option { + if chunk == 0 || input.size == 0 || output.size == 0 { + return None; + } + Some(Self { + chunk, + input, + output, + shared: false, + }) + } + + /// Makes standard output and error share one pool with the output window, which both + /// streams' data messages charge. + pub const fn shared_output(mut self) -> Self { + self.shared = true; + self + } + + /// Returns the most data bytes that one data message carries. + pub const fn chunk(&self) -> usize { + self.chunk as usize + } + + /// Returns the window of the workload's standard input. + pub const fn input(&self) -> Window { + self.input + } + + /// Returns the window of each output pool. + pub const fn output(&self) -> Window { + self.output + } + + /// Returns whether standard output and error share one pool. + pub const fn is_shared(&self) -> bool { + self.shared + } + + /// Returns the output pools, in the order of their positions. + fn output_pools(&self) -> &'static [Pool] { + if self.shared { + &[Pool::Output] + } else { + &[Pool::Stdout, Pool::Stderr] + } + } + + /// Returns the position of `stream`'s output pool. + fn pool_of(&self, stream: Stream) -> usize { + if self.shared { 0 } else { stream.index() } + } + + /// Returns the position of the output pool `pool`, or [`Violation::Pool`] if the limits have + /// no such output pool. + fn output_pool(&self, pool: Pool) -> Result { + match (pool, self.shared) { + (Pool::Output, true) => Ok(0), + (Pool::Stdout, false) => Ok(0), + (Pool::Stderr, false) => Ok(1), + _ => Err(Violation::Pool), + } + } + + /// Returns the checked length of a data message of `length` bytes: [`Violation::Chunk`] if + /// it is empty or longer than a chunk. + fn check_chunk(&self, length: usize) -> Result<(), Violation> { + if length == 0 || length > self.chunk() { + return Err(Violation::Chunk); + } + Ok(()) + } + + /// Returns the most data bytes that the input's window holds. + fn input_bytes(&self) -> usize { + self.input.bytes_with(self.chunk()) + } + + /// Returns the most data bytes that the output's windows hold together. + fn output_bytes(&self) -> usize { + let pools = if self.shared { 1 } else { 2 }; + self.output.bytes_with(self.chunk()).saturating_mul(pools) + } +} + +/// The local timers of one side of an execution. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Timers { + /// How long the execution may run before the side ends it, if it ever does. + timeout: Option, + /// How long pending output still flows once a termination begins, and how long the output + /// streams of an execution that did not settle may stay open after the settlement time. + grace: Duration, + /// How long the execution may then take to settle. + settlement: Duration, +} + +impl Timers { + /// Returns timers with no timeout, a `grace` for pending output once a termination begins, + /// and a `settlement` time after it, or after the workload's end, within which the runner + /// expects the execution to settle, and the peer expects the terminal outcome of an execution + /// that it ended. The output streams of a workload that ended on its own, but whose + /// execution did not settle, may stay open for the grace after the settlement time. + pub const fn new(grace: Duration, settlement: Duration) -> Self { + Self { + timeout: None, + grace, + settlement, + } + } + + /// Ends the execution for a [timeout](Cause::TimedOut) once `timeout` has passed since the + /// side was created: the runner, unless the workload's end reaches it before a + /// [tick](Runner::tick) fires the timeout, and the peer, if the execution has not finished. + pub const fn timeout(mut self, timeout: Duration) -> Self { + self.timeout = Some(timeout); + self + } + + /// Returns the timeout, if any. + pub const fn timeout_duration(&self) -> Option { + self.timeout + } + + /// Returns the grace of pending output. + pub const fn grace(&self) -> Duration { + self.grace + } + + /// Returns the settlement time. + pub const fn settlement(&self) -> Duration { + self.settlement + } +} + +/// Why a side ended an execution. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Cause { + /// The peer, or its caller, cancelled the execution. + Cancelled, + /// The execution's owner dropped it: the peer's caller dropped its handle, or the runner's + /// adapter abandoned the execution. + Dropped, + /// The execution's timeout passed. + TimedOut, + /// The session between the sides was reset. + Reset, + /// The transport between the sides was lost. + Disconnected, +} + +impl fmt::Display for Cause { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::Cancelled => "cancelled", + Self::Dropped => "dropped", + Self::TimedOut => "timed out", + Self::Reset => "reset", + Self::Disconnected => "disconnected", + }) + } +} + +/// The status with which a workload's program ended, as its collector reaped it. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Status { + /// The program exited with this code. + Exited(i32), + /// A signal of this number ended the program. + Signaled(i32), +} + +/// Why a workload's program did not start: the adapter's number of the launch stage that failed, +/// and the operating system's error number of the failure. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct LaunchFailure { + /// The adapter's number of the stage that failed. + stage: u32, + /// The error number of the failure. + error: i32, +} + +impl LaunchFailure { + /// Returns the failure of the stage that the adapter numbers `stage`, with the error number + /// `error`. + pub const fn new(stage: u32, error: i32) -> Self { + Self { stage, error } + } + + /// Returns the adapter's number of the stage that failed. + pub const fn stage(&self) -> u32 { + self.stage + } + + /// Returns the error number of the failure. + pub const fn error(&self) -> i32 { + self.error + } +} + +/// The result of settling an execution's processes, which the runner's adapter reports. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Settlement { + /// No process of the execution is left: the workload, its descendants, and every process + /// that held one of its pipes are gone. + Settled, + /// The adapter could not settle the execution, such as because a kill failed. + Failed, +} + +/// How an execution ended: its terminal outcome, which the runner sends the peer last. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Outcome { + /// The workload's status, if a collector reaped it. + status: Option, + /// Why a side ended the workload, if one did. + cause: Option, + /// Why the workload's program did not start, if it did not. + launch_failure: Option, + /// Whether each stream lost bytes. + truncated: [bool; 2], + /// Whether the execution settled, so that its resources may be reused. + reusable: bool, +} + +impl Outcome { + /// Returns the outcome of a workload that ended with `status`, if known, with no cause, no + /// launch failure, no truncation, and fenced: the form in which a peer's adapter decodes a + /// terminal record, before it applies the record's other fields. + pub const fn new(status: Option) -> Self { + Self { + status, + cause: None, + launch_failure: None, + truncated: [false, false], + reusable: false, + } + } + + /// Returns the outcome with `cause` as why a side ended the workload. + pub const fn with_cause(mut self, cause: Cause) -> Self { + self.cause = Some(cause); + self + } + + /// Returns the outcome with `failure` as why the workload's program did not start. + pub const fn with_launch_failure(mut self, failure: LaunchFailure) -> Self { + self.launch_failure = Some(failure); + self + } + + /// Returns the outcome with `stream` marked as truncated. + pub const fn with_truncated(mut self, stream: Stream) -> Self { + self.truncated[stream.index()] = true; + self + } + + /// Returns the outcome of an execution that settled, and so is reusable. + pub const fn with_reusable(mut self) -> Self { + self.reusable = true; + self + } + + /// Returns the workload's status, or `None` if no collector reaped it, as when its collector + /// lost it or a side gave up on the execution. + pub const fn status(&self) -> Option { + self.status + } + + /// Returns why a side ended the workload, or `None` if it ended on its own. A workload with a + /// cause may still have exited on its own before the side's end reached it, which its status + /// tells. + pub const fn cause(&self) -> Option { + self.cause + } + + /// Returns why the workload's program did not start, or `None` if it started, or if no + /// launch result arrived. + pub const fn launch_failure(&self) -> Option { + self.launch_failure + } + + /// Returns whether `stream` lost bytes, which only a termination, a failed read, or a side + /// that gave up causes. + pub const fn is_truncated(&self, stream: Stream) -> bool { + self.truncated[stream.index()] + } + + /// Returns whether the execution settled, so that its resources may be reused, or `false` if + /// it is fenced. + pub const fn is_reusable(&self) -> bool { + self.reusable + } +} + +/// What the side did with a message that it accepted. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Receipt { + /// The message changed the side's state. + Applied, + /// The message belongs to an earlier generation of the execution's key, and changed nothing. + Stale, + /// The side no longer needed the message, as for a credit after the execution ended, input + /// after the workload's input closed, or any message of the runner's once the peer detached + /// from it or gave up waiting for it: it changed nothing beyond the window it charged. + Late, +} + +/// A message that breaks the rules, which the side refuses without changing anything. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Violation { + /// The message's key names another execution, or a later generation of this one. + Key, + /// A credit names a pool that the execution's limits do not have. + Pool, + /// A credit grants no units, or more units than are in flight, which would leave its sender + /// more than the window. + Credit, + /// A data message is empty, or longer than a chunk. + Chunk, + /// A data message exceeds the credit that its receiver granted. + Window, + /// Data or an end follows the stream's end, or a second terminal outcome follows the first. + Ended, + /// A message comes out of order, such as a second start, or a terminal outcome before both + /// streams' ends. + Order, +} + +impl fmt::Display for Violation { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::Key => "the message names another execution or a later generation", + Self::Pool => "the credit names a pool that the execution does not have", + Self::Credit => "the credit grants no units or more units than are in flight", + Self::Chunk => "the data message is empty or longer than a chunk", + Self::Window => "the data message exceeds the granted credit", + Self::Ended => "the message follows the end of its stream or of the execution", + Self::Order => "the message comes out of order", + }) + } +} + +impl std::error::Error for Violation {} + +/// Returns the instant `duration` after `now`, or `None` if no instant that late exists. +fn after(now: Instant, duration: Duration) -> Option { + now.checked_add(duration) +} + +/// Returns the earlier of two optional instants. +fn earliest(first: Option, second: Option) -> Option { + match (first, second) { + (Some(first), Some(second)) => Some(first.min(second)), + (first, second) => first.or(second), + } +} diff --git a/nvx-rs/src/exec/peer.rs b/nvx-rs/src/exec/peer.rs new file mode 100644 index 000000000..97a4aefba --- /dev/null +++ b/nvx-rs/src/exec/peer.rs @@ -0,0 +1,684 @@ +//! The side of an execution that requested it: see the [module](super) documentation. + +use std::collections::VecDeque; +use std::time::Instant; + +use super::credit::{self, Credit, Granted, Inbox}; +use super::{ + Cause, Key, Limits, Outcome, Pool, Receipt, Stream, Timers, Violation, after, earliest, +}; + +/// A message from the peer to the runner, which the adapter frames and sends in order, as +/// [`Peer::next_message`] returns them. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Message { + /// End the workload for `cause`: [`Cause::Cancelled`], [`Cause::Dropped`], or + /// [`Cause::TimedOut`], as the peer's caller or timer ended the execution. + Cancel(Cause), + /// Credit of the output pool `pool`: `units` more of its window, which the runner may spend + /// on output. The first one of each pool opens its whole window. + Credit { + /// The output pool: [`Pool::Output`] if the streams share one, otherwise + /// [`Pool::Stdout`] or [`Pool::Stderr`]. + pool: Pool, + /// The units of the credit. + units: u32, + }, + /// Bytes of the workload's input, in order: the first `length` bytes of the buffer that the + /// adapter lent [`Peer::next_message`]. + Input { + /// The number of bytes, at least one and at most a chunk. + length: usize, + }, + /// The end of the workload's input, after its last byte. + InputEnd, +} + +/// What the peer tells its caller, as [`Peer::next_event`] returns them. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Event { + /// The workload's program runs: the runner confirmed its launch. + Started, + /// `stream` ended: its sink took its last byte, or the peer discarded the rest. + End(Stream), + /// The workload's input closed before its end was sent, or the execution ended: the caller + /// sends no more input, and the peer dropped what it held. + InputClosed, + /// The execution finished, with this outcome: the peer's last event. Its cause is the + /// runner's, or, if the runner reported none or never reported, the peer's own. + Finished(Outcome), +} + +/// One output stream, as the peer receives and delivers it. +#[derive(Debug)] +struct Incoming { + /// The bytes that the runner sent and the sink has not taken. + inbox: Inbox, + /// Whether the runner ended the stream. + ended: bool, + /// Whether the stream's end event was returned. + end_event: bool, + /// Whether the peer drops the stream's bytes, as no sink takes them any more. + discarding: bool, + /// Whether the peer dropped bytes of the stream. + truncated: bool, +} + +impl Incoming { + /// Drops what the inbox holds, notes whether that lost bytes, and returns the units that + /// this frees. + fn drop_inbox(&mut self) -> u32 { + let (length, freed) = self.inbox.clear(); + self.truncated |= length > 0; + freed + } +} + +/// The workload's standard input, as the peer takes it from its caller and sends it. +#[derive(Debug)] +struct Outgoing { + /// The bytes that the caller handed over and the peer has not sent: at most one chunk. + queue: VecDeque, + /// The credit that the runner granted. + credit: Credit, + /// Whether the caller ended the input. + ended: bool, + /// Whether the end of the input was sent. + end_sent: bool, + /// Whether the input closed: the runner closed it, or the execution ended. + closed: bool, + /// Whether the event that the input closed waits to be returned. + closed_event: bool, +} + +/// The side of an execution that requested it: see the [module](super) documentation. +/// +/// The adapter creates a peer as it asks the runner's side to run a workload, and hands it the +/// runner's messages and its caller's input. In return, it takes the peer's +/// [messages](Self::next_message), which it sends the runner, and its +/// [events](Self::next_event), which it tells its caller, until the execution +/// [finishes](Event::Finished). Its sink takes each stream's [pending output](Self::pending_output) +/// as it can, and acknowledges what it took with [`consumed`](Self::consumed); its source hands +/// over input as the [write allowance](Self::write_allowance) lets it. +#[derive(Debug)] +pub struct Peer { + /// The execution's key. + key: Key, + /// The limits that both sides agreed on. + limits: Limits, + /// The peer's timers. + timers: Timers, + /// Whether the runner confirmed the workload's launch. + started: bool, + /// Whether the event that the workload started waits to be returned. + started_event: bool, + /// The workload's standard input. + input: Outgoing, + /// The output streams. + outputs: [Incoming; 2], + /// The windows of the output pools: one per stream, or the first one shared. + windows: [Granted; 2], + /// The source whose message the peer returns first next: each output pool's credit, in the + /// order of the pools, then the input. They take turns, so that none starves another. + turn: usize, + /// The runner's terminal outcome, once it arrived. + terminal: Option, + /// Why the peer ended the execution, if it did. + cause: Option, + /// Whether the cancellation waits to be sent. + cancel_message: bool, + /// When the execution's timeout passes, while it runs. + timeout_at: Option, + /// When the peer ended the execution, once it did. + terminated_at: Option, + /// When the grace of pending output ends, while the peer waits for it. + grace_until: Option, + /// When the peer gives up waiting for the terminal outcome of an execution that it ended. + give_up_at: Option, + /// Whether the peer gave up waiting for the terminal outcome. + gave_up: bool, + /// Whether the peer is detached from the runner, from which nothing more arrives. + detached: bool, + /// Whether the outcome was returned. + finished: bool, + /// The most bytes that the peer held at once. + high_water: usize, +} + +impl Peer { + /// Returns the peer of the execution `key`, with the `limits` that both sides agreed on and + /// its own `timers`, created at `now`, from which its timeout runs. + pub fn new(key: Key, limits: Limits, timers: Timers, now: Instant) -> Self { + let incoming = || Incoming { + inbox: Inbox::new(limits.output.unit()), + ended: false, + end_event: false, + discarding: false, + truncated: false, + }; + Self { + key, + limits, + timers, + started: false, + started_event: false, + input: Outgoing { + queue: VecDeque::new(), + credit: Credit::new(limits.input), + ended: false, + end_sent: false, + closed: false, + closed_event: false, + }, + outputs: [incoming(), incoming()], + windows: [Granted::new(limits.output), Granted::new(limits.output)], + turn: 0, + terminal: None, + cause: None, + cancel_message: false, + timeout_at: timers.timeout.and_then(|timeout| after(now, timeout)), + terminated_at: None, + grace_until: None, + give_up_at: None, + gave_up: false, + detached: false, + finished: false, + high_water: 0, + } + } + + /// Returns the execution's key. + pub fn key(&self) -> Key { + self.key + } + + /// Returns the limits that both sides agreed on. + pub fn limits(&self) -> Limits { + self.limits + } + + /// Hands the peer the runner's credit of `units` for the workload's input. A grant of no + /// units, or of more than the units in flight, is a [`Violation::Credit`]. + pub fn credit(&mut self, key: Key, units: u32) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.ignores_runner() { + return Ok(Receipt::Late); + } + self.input.credit.grant(units)?; + if self.input.closed || self.input.end_sent { + return Ok(Receipt::Late); + } + Ok(Receipt::Applied) + } + + /// Hands the peer the runner's confirmation that the workload started. A second one, or one + /// after the terminal outcome, is a [`Violation::Order`]. + pub fn started(&mut self, key: Key) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.ignores_runner() { + return Ok(Receipt::Late); + } + if self.started || self.terminal.is_some() { + return Err(Violation::Order); + } + self.started = true; + self.started_event = true; + Ok(Receipt::Applied) + } + + /// Hands the peer bytes of `stream` from the runner, which the stream's pool must allow, and + /// which the peer holds until its sink takes them. Output after the stream's end is a + /// [`Violation::Ended`]. Once the peer discards the stream, it drops what arrives, as + /// [`Receipt::Late`], and grants its units again at once. + pub fn output(&mut self, key: Key, stream: Stream, bytes: &[u8]) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.ignores_runner() { + return Ok(Receipt::Late); + } + if self.outputs[stream.index()].ended { + return Err(Violation::Ended); + } + self.limits.check_chunk(bytes.len())?; + let window = &mut self.windows[self.limits.pool_of(stream)]; + window.charge(bytes.len())?; + let incoming = &mut self.outputs[stream.index()]; + if incoming.discarding { + incoming.truncated = true; + window.free(self.limits.output.cost(bytes.len())); + return Ok(Receipt::Late); + } + incoming.inbox.push(bytes); + self.note_buffered(); + Ok(Receipt::Applied) + } + + /// Hands the peer the end of `stream` from the runner, after its last byte. A second end is + /// a [`Violation::Ended`]. + pub fn end_output(&mut self, key: Key, stream: Stream) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.ignores_runner() { + return Ok(Receipt::Late); + } + let incoming = &mut self.outputs[stream.index()]; + if incoming.ended { + return Err(Violation::Ended); + } + incoming.ended = true; + Ok(Receipt::Applied) + } + + /// Hands the peer the runner's notice that the workload's input closed early. The peer drops + /// the input that it holds, takes no more, and tells its caller. + pub fn input_closed(&mut self, key: Key) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.ignores_runner() || self.input.closed || self.input.end_sent { + return Ok(Receipt::Late); + } + self.close_input(); + Ok(Receipt::Applied) + } + + /// Hands the peer the runner's terminal outcome at `now`, which follows both streams' ends: + /// a terminal outcome before them is a [`Violation::Order`], and a second one a + /// [`Violation::Ended`]. The peer finishes once its sink took the rest of the output, losing + /// nothing, unless a side ended the execution: then the sink takes the rest only for the + /// peer's grace, from when the peer ended the execution, or from `now` if the runner did. + /// Once the peer detached from the runner or gave up waiting for it, the terminal outcome is + /// [late](Receipt::Late), as every message of the runner's is then, and the peer's own + /// outcome stands, whenever its caller retrieves it. + pub fn terminal( + &mut self, + now: Instant, + key: Key, + outcome: Outcome, + ) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + self.tick(now); + if self.ignores_runner() { + return Ok(Receipt::Late); + } + if self.terminal.is_some() { + return Err(Violation::Ended); + } + if !self.outputs.iter().all(|incoming| incoming.ended) { + return Err(Violation::Order); + } + self.terminal = Some(outcome); + self.give_up_at = None; + self.timeout_at = None; + self.cancel_message = false; + self.close_input(); + if outcome.cause().is_some() && self.terminated_at.is_none() { + self.terminated_at = Some(now); + self.grace_until = after(now, self.timers.grace); + if self.grace_until.is_some_and(|at| now >= at) { + self.end_grace(); + } + } + Ok(Receipt::Applied) + } + + /// Returns how many more bytes of input the caller may hand over with + /// [`input`](Self::input): what the peer's queue has room for, at most one chunk ahead of the + /// runner's credit, and none once the input ended or closed, or the peer ended the + /// execution. + pub fn write_allowance(&self) -> usize { + if self.input.closed || self.input.ended || self.terminated_at.is_some() || self.finished { + return 0; + } + self.limits.chunk() - self.input.queue.len() + } + + /// Hands the peer bytes of the workload's input from its caller, in order, and returns how + /// many it took: as many as the [write allowance](Self::write_allowance) lets it. + pub fn input(&mut self, bytes: &[u8]) -> usize { + let length = bytes.len().min(self.write_allowance()); + self.input.queue.extend(&bytes[..length]); + self.note_buffered(); + length + } + + /// Ends the workload's input after the bytes that the caller handed over. + pub fn end_input(&mut self) { + if !self.input.closed { + self.input.ended = true; + } + } + + /// Returns the bytes of `stream` that the sink takes next, as many as are contiguous: a part + /// of what the peer holds, while it holds any. + pub fn pending_output(&self, stream: Stream) -> &[u8] { + self.outputs[stream.index()].inbox.front() + } + + /// Hands the peer the count of bytes of `stream`'s [pending output](Self::pending_output) + /// that the sink took, which the peer grants the runner again. + pub fn consumed(&mut self, stream: Stream, bytes: usize) { + let freed = self.outputs[stream.index()].inbox.pop(bytes); + self.windows[self.limits.pool_of(stream)].free(freed); + } + + /// Ends the execution at `now` for [`Cause::Cancelled`]: the peer sends the runner its + /// cancellation, takes no more input, delivers the pending output for its grace, and + /// waits for the terminal outcome for the settlement time after that. + pub fn cancel(&mut self, now: Instant) { + self.tick(now); + self.terminate(now, Cause::Cancelled); + } + + /// Ends the execution at `now` for [`Cause::Dropped`], because the caller dropped its + /// handle: as a [cancellation](Self::cancel), but the peer discards the output at once, as + /// no sink takes it any more. + pub fn dropped(&mut self, now: Instant) { + self.tick(now); + self.terminate(now, Cause::Dropped); + if !self.finished { + self.end_grace(); + } + } + + /// Detaches the peer from the runner at `now` for [`Cause::Reset`], because the session was + /// reset. The peer takes none of the runner's messages from then on. + pub fn reset(&mut self, now: Instant) { + self.detach(now, Cause::Reset); + } + + /// Detaches the peer from the runner at `now` for [`Cause::Disconnected`], because the + /// transport was lost. The peer takes none of the runner's messages from then on. + pub fn disconnect(&mut self, now: Instant) { + self.detach(now, Cause::Disconnected); + } + + /// Fires the timers whose instants `now` has reached: the execution's timeout, the end of + /// the grace of pending output, and the time at which the peer gives up waiting for the + /// terminal outcome of an execution that it ended. + pub fn tick(&mut self, now: Instant) { + if self.finished { + return; + } + if self.timeout_at.is_some_and(|at| now >= at) { + self.timeout_at = None; + self.terminate(now, Cause::TimedOut); + } + if self.grace_until.is_some_and(|at| now >= at) { + self.end_grace(); + } + if self.give_up_at.is_some_and(|at| now >= at) { + self.give_up_at = None; + self.gave_up = true; + self.end_grace(); + } + } + + /// Returns when the peer's next timer fires, for the adapter to [tick](Self::tick) it then, + /// or `None` while no timer runs. + pub fn deadline(&self) -> Option { + if self.finished { + return None; + } + [self.timeout_at, self.grace_until, self.give_up_at] + .into_iter() + .fold(None, earliest) + } + + /// Returns the next message to send the runner, or `None` if none waits: the cancellation + /// first, then credit of each output pool, the input, and its end once its bytes are sent. + /// Each pool's credit and the input take turns: after one of them, the others that wait go + /// first, so that none starves another, however much output the sink takes between two + /// calls. An input message carries the most bytes that the queue, the credit, a chunk, and + /// `buffer` allow, which it holds at the start of `buffer`; an empty buffer carries none. + pub fn next_message(&mut self, buffer: &mut [u8]) -> Option { + if self.finished || self.detached { + return None; + } + if std::mem::take(&mut self.cancel_message) { + return Some(Message::Cancel(self.cause.unwrap_or(Cause::Cancelled))); + } + let pools = self.limits.output_pools().len(); + let sources = pools + 1; + for offset in 0..sources { + let source = (self.turn + offset) % sources; + let message = if source < pools { + self.next_credit(source) + } else { + self.next_input(buffer) + }; + if message.is_some() { + self.turn = (source + 1) % sources; + return message; + } + } + None + } + + /// Returns credit of the output pool at `index` to send the runner, if a stream of the pool + /// is still open, no terminal outcome arrived, and the sink took bytes since the last grant. + fn next_credit(&mut self, index: usize) -> Option { + if self.terminal.is_some() { + return None; + } + let open = Stream::ALL.into_iter().any(|stream| { + self.limits.pool_of(stream) == index && !self.outputs[stream.index()].ended + }); + if !open { + return None; + } + let units = self.windows[index].take_grant()?; + let pool = self.limits.output_pools()[index]; + Some(Message::Credit { pool, units }) + } + + /// Returns the next input message to send the runner: the input, then its end once its bytes + /// are sent, unless the input closed. + fn next_input(&mut self, buffer: &mut [u8]) -> Option { + if self.input.closed { + return None; + } + let wanted = self + .input + .queue + .len() + .min(self.limits.chunk()) + .min(buffer.len()); + let length = self.input.credit.sendable(wanted); + if length > 0 { + credit::take(&mut self.input.queue, buffer, length); + self.input.credit.spend(length); + return Some(Message::Input { length }); + } + if self.input.ended && self.input.queue.is_empty() && !self.input.end_sent { + self.input.end_sent = true; + return Some(Message::InputEnd); + } + None + } + + /// Returns the next event for the caller, or `None` if none waits: that the workload + /// started, that its input closed, the end of each stream once its sink took its last byte, + /// and the outcome last, once the runner's terminal outcome arrived and the sink took the + /// rest of the output, or once the peer detached or gave up and delivered or discarded + /// what it held. + pub fn next_event(&mut self) -> Option { + if self.finished { + return None; + } + if std::mem::take(&mut self.started_event) { + return Some(Event::Started); + } + if std::mem::take(&mut self.input.closed_event) { + return Some(Event::InputClosed); + } + for stream in Stream::ALL { + let incoming = &mut self.outputs[stream.index()]; + if incoming.ended && incoming.inbox.is_empty() && !incoming.end_event { + incoming.end_event = true; + return Some(Event::End(stream)); + } + } + let outcome = self.outcome()?; + self.finished = true; + Some(Event::Finished(outcome)) + } + + /// Returns whether the peer has returned the execution's outcome. + pub fn is_finished(&self) -> bool { + self.finished + } + + /// Returns whether the runner confirmed the workload's launch. + pub fn has_started(&self) -> bool { + self.started + } + + /// Returns whether the peer is detached from the runner. + pub fn is_detached(&self) -> bool { + self.detached + } + + /// Returns the most bytes that the peer holds at once: one chunk of input, and the output's + /// windows. + pub fn memory_bound(&self) -> usize { + self.limits + .chunk() + .saturating_add(self.limits.output_bytes()) + } + + /// Returns the bytes that the peer holds now. + pub fn buffered(&self) -> usize { + self.input.queue.len() + + self + .outputs + .iter() + .map(|incoming| incoming.inbox.len()) + .sum::() + } + + /// Returns the most bytes that the peer has held at once. + pub fn high_water(&self) -> usize { + self.high_water + } + + /// Returns whether the peer takes no more of the runner's messages: it finished, detached + /// from the runner, or gave up waiting for it. So its outcome is fixed from then on, whenever + /// its caller retrieves it. + fn ignores_runner(&self) -> bool { + self.finished || self.detached || self.gave_up + } + + /// Ends the execution at `now` for `cause`, unless the peer ended it already: sends the + /// runner the cancellation, unless the runner's terminal outcome arrived, closes the input, + /// and begins the grace of pending output, after which the peer waits for the terminal + /// outcome for the settlement time. + fn terminate(&mut self, now: Instant, cause: Cause) { + if self.finished || self.terminated_at.is_some() { + return; + } + self.terminated_at = Some(now); + self.timeout_at = None; + self.cause = Some(cause); + self.close_input(); + self.grace_until = after(now, self.timers.grace); + if self.terminal.is_none() { + self.cancel_message = !self.detached; + self.give_up_at = self + .grace_until + .and_then(|grace| after(grace, self.timers.settlement)); + } + if self.grace_until.is_some_and(|at| now >= at) { + self.end_grace(); + } + } + + /// Detaches the peer from the runner at `now` for `cause`: ends the execution if the peer + /// has not, and stops waiting for the runner, from which nothing more arrives. + fn detach(&mut self, now: Instant, cause: Cause) { + self.tick(now); + if self.finished || self.detached { + return; + } + self.terminate(now, cause); + self.detached = true; + self.cancel_message = false; + self.give_up_at = None; + } + + /// Closes the input: drops what the peer holds of it, and tells the caller unless the caller + /// ended it. + fn close_input(&mut self) { + if self.input.closed { + return; + } + self.input.closed = true; + self.input.queue = VecDeque::new(); + self.input.closed_event = !self.input.end_sent; + } + + /// Ends the grace of pending output: drops what the inboxes hold, grants their units again, + /// and discards what still arrives. + fn end_grace(&mut self) { + self.grace_until = None; + for stream in Stream::ALL { + let incoming = &mut self.outputs[stream.index()]; + incoming.discarding = true; + let freed = incoming.drop_inbox(); + self.windows[self.limits.pool_of(stream)].free(freed); + } + } + + /// Returns the outcome once the peer can finish: the runner's terminal outcome, with the + /// streams that the peer truncated, once the sink took or the peer discarded the rest of the + /// output; or, once the peer detached or gave up, an outcome of its own, with no status, its + /// cause, each stream truncated that did not end whole, and fenced. + fn outcome(&self) -> Option { + if !self + .outputs + .iter() + .all(|incoming| incoming.inbox.is_empty()) + { + return None; + } + let mut outcome = match self.terminal { + Some(terminal) => terminal, + None if self.detached || self.gave_up => { + let mut outcome = Outcome::new(None); + for stream in Stream::ALL { + if !self.outputs[stream.index()].ended { + outcome = outcome.with_truncated(stream); + } + } + outcome + } + None => return None, + }; + if outcome.cause().is_none() + && let Some(cause) = self.cause + { + outcome = outcome.with_cause(cause); + } + for stream in Stream::ALL { + if self.outputs[stream.index()].truncated { + outcome = outcome.with_truncated(stream); + } + } + Some(outcome) + } + + /// Notes the bytes that the peer holds, for its high-water mark. + fn note_buffered(&mut self) { + self.high_water = self.high_water.max(self.buffered()); + } +} diff --git a/nvx-rs/src/exec/runner.rs b/nvx-rs/src/exec/runner.rs new file mode 100644 index 000000000..f43a2fd09 --- /dev/null +++ b/nvx-rs/src/exec/runner.rs @@ -0,0 +1,880 @@ +//! The side of an execution that runs its workload: see the [module](super) documentation. + +use std::collections::VecDeque; +use std::time::Instant; + +use super::credit::{self, Credit, Granted, Inbox}; +use super::{ + Cause, Key, LaunchFailure, Limits, Outcome, Pool, Receipt, Settlement, Status, Stream, Timers, + Violation, after, earliest, +}; + +/// A message from the runner to the peer, which the adapter frames and sends in order, as +/// [`Runner::next_message`] returns them. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Message { + /// Credit of the workload's input: `units` more of the input's window, which the peer may + /// spend on input. The first one opens the whole window. + Credit { + /// The units of the credit. + units: u32, + }, + /// The workload's program runs: its launch succeeded. + Started, + /// Bytes of `stream`, in order: the first `length` bytes of the buffer that the adapter lent + /// [`Runner::next_message`]. + Output { + /// The stream that the bytes belong to. + stream: Stream, + /// The number of bytes, at least one and at most a chunk. + length: usize, + }, + /// The end of `stream`, after its last byte. + End(Stream), + /// The workload's input closed before the peer ended it, because the workload closed it, + /// ended, or was ended: the peer sends no more input, and the runner drops what arrives. + InputClosed, + /// The execution's terminal outcome, after both streams' ends: the runner's last message. + /// Once the runner is detached from the peer, by a [reset](Runner::reset), a + /// [disconnection](Runner::disconnect), or a [drop](Runner::dropped), it is the only message + /// that the runner still returns, and the adapter keeps it rather than send it. + Terminal(Outcome), +} + +/// A request to the adapter, which applies it to the workload, as [`Runner::next_request`] +/// returns them. The runner makes each one once. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Request { + /// End the workload, and its processes, for `cause`, closing its input. + Terminate(Cause), + /// Read and drop the rest of `stream`, reporting the bytes with [`Runner::discarded`], until + /// it closes. + Discard(Stream), + /// Close the workload's input: the peer ended it, and the workload took every byte that the + /// runner accepted. + CloseInput, + /// Settle the execution's processes: end what is left of them, and report with + /// [`Runner::settled`] once the adapter's own check shows that none is left. + Settle, +} + +/// One output stream, as the runner reads and sends it. +#[derive(Debug, Default)] +struct Output { + /// The bytes read from the workload and not sent yet: at most one chunk. + queue: VecDeque, + /// Whether the workload's stream ended: every writer closed it, or reading it failed. + closed: bool, + /// Whether every writer closed the workload's stream, though the adapter has not read it to + /// its end: it ends once the adapter reads what it still holds. + hung_up: bool, + /// Whether the stream's end message was returned. + ended: bool, + /// Whether the runner drops the stream's bytes. + discarding: bool, + /// Whether the request to discard the stream waits to be returned. + discard_requested: bool, + /// Whether the stream lost bytes. + truncated: bool, +} + +impl Output { + /// Returns whether the stream is done: it closed, and its queue is empty. + fn is_done(&self) -> bool { + self.closed && self.queue.is_empty() + } + + /// Drops the queue, and notes whether that lost bytes. + fn drop_queue(&mut self) { + if !self.queue.is_empty() { + self.truncated = true; + } + self.queue = VecDeque::new(); + } +} + +/// The workload's standard input, as the runner receives and writes it. +#[derive(Debug)] +struct Input { + /// The bytes that the peer sent and the workload has not taken. + inbox: Inbox, + /// The input's window. + window: Granted, + /// Whether the peer ended the input. + peer_ended: bool, + /// Whether the workload's input is closed, or the runner requested to close it. + closed: bool, + /// Whether the request to close the input waits to be returned. + close_requested: bool, + /// Whether the message that the input closed early waits to be returned. + closed_message: bool, +} + +/// Where the settlement of the execution stands. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Settling { + /// The runner has not requested it. + Idle, + /// The runner requested it, and expects its result by this instant, if any. + Requested(Option), + /// The execution settled. + Settled, + /// The execution is fenced: its settlement failed, or came too late. + Fenced, +} + +/// The side of an execution that runs its workload: see the [module](super) documentation. +/// +/// The adapter creates a runner when it accepts the peer's request to run a workload, and hands +/// it the peer's messages and the workload's events. In return, it takes the runner's +/// [requests](Self::next_request), which it applies to the workload, and its +/// [messages](Self::next_message), which it sends the peer, until the +/// [terminal outcome](Message::Terminal). It reads each output stream of the workload only as far +/// as the [read allowance](Self::read_allowance) lets it, and writes the +/// [pending input](Self::pending_input) to the workload as the workload takes it. +#[derive(Debug)] +pub struct Runner { + /// The execution's key. + key: Key, + /// The limits that both sides agreed on. + limits: Limits, + /// The runner's timers. + timers: Timers, + /// The workload's launch result, once it arrived. + launch: Option>, + /// Whether the message that the workload started waits to be returned. + started_message: bool, + /// The workload's status, once its end is known: `Some(None)` if its status is lost. + exit: Option>, + /// Why the runner ended the workload, if it did. + cause: Option, + /// The request to end the workload, while it waits to be returned. + terminate_request: Option, + /// Whether the request to settle the execution waits to be returned. + settle_request: bool, + /// When the workload's timeout passes, while it runs. + timeout_at: Option, + /// When the termination began, once one did. + terminated_at: Option, + /// When the grace of pending output ends, while the runner waits for it. + grace_until: Option, + /// When the runner gives up on the execution after a termination, while it waits for it, + /// unless the execution's cleanup is complete by then. + give_up_at: Option, + /// When the runner gives up on what a process may still hold after the workload's end: the + /// launch result, if it has not arrived, and, if the execution does not settle, the output + /// streams that are still open. It is the settlement time and then the grace after the + /// workload's end. + abandon_at: Option, + /// Whether the runner gave up, and fenced the execution. + gave_up: bool, + /// Whether the runner gave up on the launch result after the workload's end, and fenced the + /// execution. + launch_abandoned: bool, + /// Whether the runner is detached from the peer, to which it sends nothing more. + detached: bool, + /// The output streams. + outputs: [Output; 2], + /// The credit of the output pools: one per stream, or the first one shared. + credits: [Credit; 2], + /// The stream whose output the runner sends first next. + turn: usize, + /// Whether the last message that the runner returned was credit of the input, so that any + /// other message that waits goes next: credit and the other messages take turns. + credited: bool, + /// The workload's standard input. + input: Input, + /// Where the settlement stands. + settling: Settling, + /// Whether the terminal outcome was returned. + finished: bool, + /// The most bytes that the runner held at once. + high_water: usize, +} + +impl Runner { + /// Returns the runner of the execution `key`, with the `limits` that both sides agreed on + /// and its own `timers`, created at `now`, from which its timeout runs. + pub fn new(key: Key, limits: Limits, timers: Timers, now: Instant) -> Self { + Self { + key, + limits, + timers, + launch: None, + started_message: false, + exit: None, + cause: None, + terminate_request: None, + settle_request: false, + timeout_at: timers.timeout.and_then(|timeout| after(now, timeout)), + terminated_at: None, + grace_until: None, + give_up_at: None, + abandon_at: None, + gave_up: false, + launch_abandoned: false, + detached: false, + outputs: Default::default(), + credits: [Credit::new(limits.output), Credit::new(limits.output)], + turn: 0, + credited: false, + input: Input { + inbox: Inbox::new(limits.input.unit()), + window: Granted::new(limits.input), + peer_ended: false, + closed: false, + close_requested: false, + closed_message: false, + }, + settling: Settling::Idle, + finished: false, + high_water: 0, + } + } + + /// Returns the execution's key. + pub fn key(&self) -> Key { + self.key + } + + /// Returns the limits that both sides agreed on. + pub fn limits(&self) -> Limits { + self.limits + } + + /// Hands the runner the peer's credit of `units` for the output pool `pool`, which the + /// execution's limits must have: [`Pool::Output`] if the streams share one pool, otherwise + /// [`Pool::Stdout`] or [`Pool::Stderr`]. A grant of no units, or of more than the units in + /// flight, is a [`Violation::Credit`]. + pub fn credit(&mut self, key: Key, pool: Pool, units: u32) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + let index = self.limits.output_pool(pool)?; + if self.finished { + return Ok(Receipt::Late); + } + self.credits[index].grant(units)?; + Ok(Receipt::Applied) + } + + /// Hands the runner bytes of the workload's input from the peer, which the input's window + /// must allow, and which the runner holds until the workload takes them. Input after the + /// peer ended it is a [`Violation::Ended`]. Once the workload's input closed, the runner + /// drops what arrives, as [`Receipt::Late`]. + pub fn input(&mut self, key: Key, bytes: &[u8]) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.finished { + return Ok(Receipt::Late); + } + if self.input.peer_ended { + return Err(Violation::Ended); + } + self.limits.check_chunk(bytes.len())?; + self.input.window.charge(bytes.len())?; + if self.input.closed { + return Ok(Receipt::Late); + } + self.input.inbox.push(bytes); + self.note_buffered(); + Ok(Receipt::Applied) + } + + /// Hands the runner the end of the workload's input from the peer. The runner requests to + /// [close the input](Request::CloseInput) once the workload took every byte before it. A + /// second end is a [`Violation::Ended`]. + pub fn end_input(&mut self, key: Key) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + if self.finished { + return Ok(Receipt::Late); + } + if self.input.peer_ended { + return Err(Violation::Ended); + } + self.input.peer_ended = true; + if self.input.closed { + return Ok(Receipt::Late); + } + self.close_input_when_taken(); + Ok(Receipt::Applied) + } + + /// Hands the runner the peer's cancellation at `now`, with the peer's `cause`, such as + /// [`Cause::Cancelled`], or [`Cause::TimedOut`] for a timeout of the peer's own. It ends the + /// workload for that cause if the workload still runs, and begins the grace of its pending + /// output. A cancellation once a termination began, or once the execution ended, is + /// [`Receipt::Late`]. + pub fn cancel(&mut self, now: Instant, key: Key, cause: Cause) -> Result { + if !self.key.admits(key)? { + return Ok(Receipt::Stale); + } + self.tick(now); + if self.finished || self.terminated_at.is_some() { + return Ok(Receipt::Late); + } + self.terminate(now, cause); + Ok(Receipt::Applied) + } + + /// Hands the runner the workload's launch result: `Ok` once its program runs, after which + /// the runner tells the peer that it [started](Message::Started), or why it did not start. + /// The workload's end, with [`exited`](Self::exited), may come before or after it: an adapter + /// learns of the launch late if another process holds what reports it. The runner sends the + /// workload's start, output, and terminal outcome only once it has the result, or gave up on + /// it: after a termination, or once the settlement time and the grace have passed since the + /// workload's end. Only the first result counts, and none once the runner gave up on it, as + /// it may have sent output, which no start may follow, or the terminal outcome. + pub fn launched(&mut self, result: Result<(), LaunchFailure>) { + if !self.awaits_launch() { + return; + } + self.launch = Some(result); + if result.is_ok() { + self.started_message = !self.detached; + } else { + self.close_input(); + } + } + + /// Returns how many more bytes of `stream` the adapter may read from the workload and hand + /// over with [`output`](Self::output): what the runner's queue of the stream has room for, at + /// most one chunk ahead of the peer's credit, and none once the stream closed or the runner + /// discards it. + pub fn read_allowance(&self, stream: Stream) -> usize { + let output = &self.outputs[stream.index()]; + if output.closed || output.discarding || self.gave_up { + return 0; + } + self.limits.chunk() - output.queue.len() + } + + /// Hands the runner bytes that the adapter read from `stream`, in order, and returns how + /// many it took: all of them, unless they exceed the [read allowance](Self::read_allowance), + /// beyond which it takes none. Once the runner discards the stream, it takes and drops them, + /// which truncates the stream. + pub fn output(&mut self, stream: Stream, bytes: &[u8]) -> usize { + let chunk = self.limits.chunk(); + let output = &mut self.outputs[stream.index()]; + if output.closed || self.gave_up { + return 0; + } + if output.discarding { + output.truncated |= !bytes.is_empty(); + return bytes.len(); + } + let length = bytes.len().min(chunk - output.queue.len()); + output.queue.extend(&bytes[..length]); + self.note_buffered(); + length + } + + /// Hands the runner the count of bytes of `stream` that the adapter read and dropped, as a + /// [discard](Request::Discard) asks, which truncates the stream unless it is zero. + pub fn discarded(&mut self, stream: Stream, bytes: usize) { + self.outputs[stream.index()].truncated |= bytes > 0; + } + + /// Hands the runner the end of `stream`: every writer of the workload's stream closed it. + pub fn output_closed(&mut self, stream: Stream) { + self.outputs[stream.index()].closed = true; + } + + /// Hands the runner a failure to read `stream`, which ends the stream as its close does, and + /// truncates it. + pub fn output_failed(&mut self, stream: Stream) { + let output = &mut self.outputs[stream.index()]; + output.closed = true; + output.truncated = true; + } + + /// Hands the runner the news that every writer of `stream` closed it, though the adapter has + /// not read it to its end: the stream ends once the adapter reads what it still holds. No + /// process can hold the stream open any more, so the runner never gives up on it after the + /// workload's end, even if the execution does not settle. + pub fn output_hung_up(&mut self, stream: Stream) { + self.outputs[stream.index()].hung_up = true; + } + + /// Returns the bytes of input that the adapter writes to the workload next, as many as are + /// contiguous: a part of what the runner holds, while it holds any and the input is open. + pub fn pending_input(&self) -> &[u8] { + if self.input.closed { + return &[]; + } + self.input.inbox.front() + } + + /// Hands the runner the count of bytes of the [pending input](Self::pending_input) that the + /// workload's input took, which the runner returns to the peer as credit. + pub fn input_written(&mut self, bytes: usize) { + if self.input.closed { + return; + } + let freed = self.input.inbox.pop(bytes); + self.input.window.free(freed); + self.close_input_when_taken(); + } + + /// Hands the runner the close of the workload's input other than by its + /// [request](Request::CloseInput): the workload closed it, or the adapter did as the workload + /// ended. The runner drops the pending input, and tells the peer, unless the peer ended the + /// input. + pub fn input_closed(&mut self) { + self.close_input(); + } + + /// Hands the runner the workload's end at `now`, with its status as its collector reaped it, + /// or `None` if the status is lost. Only the first end counts. The workload's timeout fires + /// no more, even if `now` has reached it, as the workload ended before the runner learned it: + /// the end stands as the workload's own. The runner closes the input, and requests to + /// [settle](Request::Settle) the execution, whose result it expects within the settlement + /// time. If the execution does not settle, as its settlement failed or came too late, the + /// processes that hold its output streams may outlive it, so the runner gives up on the + /// streams that are still open, but for those that [hung up](Self::output_hung_up), once the + /// grace after the settlement time has passed too: it reads them no more, sends what it read, + /// and marks them truncated. A process may also hold back what reports the workload's + /// [launch](Self::launched), whether or not the execution settles, so a launch result that + /// has not arrived by then fences the execution as well: the runner gives up on the result, + /// and on those streams, and sends the workload's output and terminal outcome without it, + /// with the workload's status and no cause. + pub fn exited(&mut self, now: Instant, status: Option) { + self.timeout_at = None; + self.tick(now); + if self.exit.is_some() || self.gave_up { + return; + } + self.exit = Some(status); + self.close_input(); + self.settle_request = true; + let settlement = after(now, self.timers.settlement); + self.settling = Settling::Requested(settlement); + self.abandon_at = settlement.and_then(|at| after(at, self.timers.grace)); + } + + /// Hands the runner the result of the [settlement](Request::Settle) that it requested: the + /// execution is reusable once it settled, and fenced if it failed. A result that the runner + /// did not request, or that comes after the settlement time, changes nothing. + pub fn settled(&mut self, result: Settlement) { + if let Settling::Requested(_) = self.settling { + self.settling = match result { + Settlement::Settled => Settling::Settled, + Settlement::Failed => Settling::Fenced, + }; + } + } + + /// Ends the workload at `now` for [`Cause::Dropped`], because the adapter abandons the + /// execution, and detaches the runner from the peer. + pub fn dropped(&mut self, now: Instant) { + self.detach(now, Cause::Dropped); + } + + /// Ends the workload at `now` for [`Cause::Reset`], because the session with the peer was + /// reset, and detaches the runner from the peer. + pub fn reset(&mut self, now: Instant) { + self.detach(now, Cause::Reset); + } + + /// Ends the workload at `now` for [`Cause::Disconnected`], because the transport to the peer + /// was lost, and detaches the runner from the peer. + pub fn disconnect(&mut self, now: Instant) { + self.detach(now, Cause::Disconnected); + } + + /// Fires the timers whose instants `now` has reached: the workload's timeout, the end of the + /// grace of pending output, the settlement time, the time after a termination at which the + /// runner gives up on the execution unless its cleanup is complete, and the time after the + /// workload's end at which it gives up on a launch result that has not arrived, and on the + /// output streams of an execution that did not settle. + pub fn tick(&mut self, now: Instant) { + if self.finished { + return; + } + if self.timeout_at.is_some_and(|at| now >= at) { + self.timeout_at = None; + self.terminate(now, Cause::TimedOut); + } + if self.grace_until.is_some_and(|at| now >= at) { + self.end_grace(); + } + if let Settling::Requested(Some(by)) = self.settling + && now >= by + { + self.settling = Settling::Fenced; + } + if self.give_up_at.is_some_and(|at| now >= at) { + if self.is_cleaned_up() { + self.give_up_at = None; + } else { + self.give_up(); + } + } + if self.abandon_at.is_some_and(|at| now >= at) { + if self.awaits_launch() { + self.abandon_launch(); + } + if self.settling == Settling::Fenced { + self.abandon_open_streams(); + } + } + } + + /// Returns when the runner's next timer fires, for the adapter to [tick](Self::tick) it + /// then, or `None` while no timer runs. + pub fn deadline(&self) -> Option { + if self.finished { + return None; + } + let settlement = match self.settling { + Settling::Requested(by) => by, + _ => None, + }; + let give_up = self.give_up_at.filter(|_| !self.is_cleaned_up()); + let abandon = self.abandon_at.filter(|_| { + self.awaits_launch() + || self.settling == Settling::Fenced + && self + .outputs + .iter() + .any(|output| !output.closed && !output.hung_up) + }); + [ + self.timeout_at, + self.grace_until, + settlement, + give_up, + abandon, + ] + .into_iter() + .fold(None, earliest) + } + + /// Returns the next request to apply to the workload, or `None` if none waits: ending the + /// workload first, then discarding streams, closing the input, and settling the execution. + pub fn next_request(&mut self) -> Option { + if let Some(cause) = self.terminate_request.take() { + return Some(Request::Terminate(cause)); + } + for stream in Stream::ALL { + let output = &mut self.outputs[stream.index()]; + if std::mem::take(&mut output.discard_requested) { + return Some(Request::Discard(stream)); + } + } + if std::mem::take(&mut self.input.close_requested) { + return Some(Request::CloseInput); + } + if std::mem::take(&mut self.settle_request) { + return Some(Request::Settle); + } + None + } + + /// Returns the next message to send the peer, or `None` if none waits: credit of the input, + /// that the workload started, that its input closed early, the end of each stream once its + /// bytes are sent, the workload's output, which takes turns between the streams as both have + /// bytes and credit, and the terminal outcome last, once the workload's status, both streams' + /// ends, and the settlement are known. A stream's end goes before the other stream's output, + /// which would otherwise hold it back for as long as it flows. Credit of the input and the + /// other messages take turns: after credit, any other message that waits goes first, so that + /// neither starves the other, however much input the workload takes between two calls. An + /// output message carries the most bytes that the stream's queue, its credit, a chunk, and + /// `buffer` allow, which it holds at the start of `buffer`; an empty buffer carries none, so + /// that only other messages return. + pub fn next_message(&mut self, buffer: &mut [u8]) -> Option { + if self.finished { + return None; + } + if !self.detached { + if !self.credited + && let Some(message) = self.next_credit() + { + self.credited = true; + return Some(message); + } + if let Some(message) = self.next_other(buffer) { + self.credited = false; + return Some(message); + } + if let Some(message) = self.next_credit() { + self.credited = true; + return Some(message); + } + } + let outcome = self.outcome()?; + self.finished = true; + Some(Message::Terminal(outcome)) + } + + /// Returns whether the runner has returned its terminal outcome. + pub fn is_finished(&self) -> bool { + self.finished + } + + /// Returns whether the runner is detached from the peer. + pub fn is_detached(&self) -> bool { + self.detached + } + + /// Returns whether the runner awaits the result of the [settlement](Request::Settle) that it + /// requested: from its request until the result, the settlement time, or the runner giving + /// up, after which the adapter may stop settling the execution. + pub fn awaits_settlement(&self) -> bool { + matches!(self.settling, Settling::Requested(_)) + } + + /// Returns the most bytes that the runner holds at once: one chunk of each output stream, + /// and the input's window. + pub fn memory_bound(&self) -> usize { + self.limits + .chunk() + .saturating_mul(2) + .saturating_add(self.limits.input_bytes()) + } + + /// Returns the bytes that the runner holds now. + pub fn buffered(&self) -> usize { + self.outputs + .iter() + .map(|output| output.queue.len()) + .sum::() + + self.input.inbox.len() + } + + /// Returns the most bytes that the runner has held at once. + pub fn high_water(&self) -> usize { + self.high_water + } + + /// Returns credit of the input to send the peer, if the runner still grants it and the + /// workload took input since the last grant. + fn next_credit(&mut self) -> Option { + if !self.input_accepts() { + return None; + } + let units = self.input.window.take_grant()?; + Some(Message::Credit { units }) + } + + /// Returns the next message other than credit of the input: that the workload started, that + /// its input closed early, the end of each stream once its bytes are sent, and the workload's + /// output. + fn next_other(&mut self, buffer: &mut [u8]) -> Option { + if std::mem::take(&mut self.started_message) { + return Some(Message::Started); + } + if std::mem::take(&mut self.input.closed_message) { + return Some(Message::InputClosed); + } + if !self.awaits_launch() { + for stream in Stream::ALL { + let output = &mut self.outputs[stream.index()]; + if output.is_done() && !output.ended { + output.ended = true; + return Some(Message::End(stream)); + } + } + return self.next_output(buffer); + } + None + } + + /// Returns the next output message, taking turns between the streams. + fn next_output(&mut self, buffer: &mut [u8]) -> Option { + for _ in Stream::ALL { + let stream = Stream::ALL[self.turn]; + self.turn = 1 - self.turn; + let credit = &mut self.credits[self.limits.pool_of(stream)]; + let output = &mut self.outputs[stream.index()]; + let wanted = output + .queue + .len() + .min(self.limits.chunk()) + .min(buffer.len()); + let length = credit.sendable(wanted); + if length > 0 { + credit::take(&mut output.queue, buffer, length); + credit.spend(length); + return Some(Message::Output { stream, length }); + } + } + None + } + + /// Returns whether the runner still grants input credit: the input is open, and the peer + /// has not ended it. + fn input_accepts(&self) -> bool { + !self.input.closed && !self.input.peer_ended + } + + /// Requests to close the input once the peer ended it and the workload took every byte. + fn close_input_when_taken(&mut self) { + if self.input.peer_ended && !self.input.closed && self.input.inbox.is_empty() { + self.input.closed = true; + self.input.close_requested = true; + } + } + + /// Notes that the workload's input closed, other than by the runner's request: drops the + /// pending input, and tells the peer unless the peer ended the input. + fn close_input(&mut self) { + if self.input.closed { + return; + } + self.input.closed = true; + self.input.inbox.clear(); + self.input.closed_message = !self.input.peer_ended && !self.detached; + } + + /// Ends the workload at `now` for `cause` if it still runs, and begins the grace of pending + /// output, unless a termination began already. + fn terminate(&mut self, now: Instant, cause: Cause) { + if self.terminated_at.is_some() { + return; + } + self.terminated_at = Some(now); + self.timeout_at = None; + if self.exit.is_none() { + self.cause = Some(cause); + self.terminate_request = Some(cause); + } + self.close_input(); + self.grace_until = after(now, self.timers.grace); + self.give_up_at = self + .grace_until + .and_then(|grace| after(grace, self.timers.settlement)); + if self.grace_until.is_some_and(|at| now >= at) { + self.end_grace(); + } + } + + /// Detaches the runner from the peer at `now` for `cause`: ends the workload if it still + /// runs, and discards its output at once, as no peer is left to take it. + fn detach(&mut self, now: Instant, cause: Cause) { + self.tick(now); + if self.finished || self.detached { + return; + } + self.terminate(now, cause); + self.detached = true; + self.started_message = false; + self.input.closed_message = false; + self.end_grace(); + let give_up = after(now, self.timers.settlement); + self.give_up_at = earliest(self.give_up_at, give_up); + } + + /// Ends the grace of pending output: drops what the queues hold, and requests to discard + /// each stream that is still open. + fn end_grace(&mut self) { + self.grace_until = None; + for output in &mut self.outputs { + output.drop_queue(); + if !output.closed && !output.discarding { + output.discarding = true; + output.discard_requested = true; + } + } + } + + /// Gives up on the execution after a termination: whatever is still unknown or open stays + /// so, each stream that did not close is truncated, and the execution is fenced. + fn give_up(&mut self) { + self.give_up_at = None; + self.grace_until = None; + self.gave_up = true; + for output in &mut self.outputs { + output.drop_queue(); + if !output.closed { + output.closed = true; + output.truncated = true; + } + } + if self.exit.is_none() { + self.exit = Some(None); + } + self.settling = Settling::Fenced; + self.settle_request = false; + } + + /// Gives up on the launch result, which has not arrived once the settlement time and the + /// grace have passed since the workload's end: a process that outlived the workload holds + /// what reports it. The execution's cleanup is incomplete, so the runner fences it, and then + /// gives up on its open streams too, and it sends the workload's output and terminal outcome + /// without the result. + fn abandon_launch(&mut self) { + self.launch_abandoned = true; + self.settling = Settling::Fenced; + } + + /// Gives up on the output streams that are still open after the workload's end, once the + /// execution did not settle, but for those that hung up: the processes that hold them + /// outlived the cleanup, so they may never close. The runner reads them no more and marks + /// them truncated, but still sends what it read, and loses nothing of the streams that closed + /// or hung up. + fn abandon_open_streams(&mut self) { + self.abandon_at = None; + for output in &mut self.outputs { + if !output.closed && !output.hung_up { + output.closed = true; + output.truncated = true; + } + } + } + + /// Returns whether the runner still waits for the workload's launch result: it has not + /// arrived, and the runner has not given up on it. + fn awaits_launch(&self) -> bool { + self.launch.is_none() && !self.gave_up && !self.launch_abandoned + } + + /// Returns whether the execution's cleanup is complete: the workload's launch result and + /// status are known, both of its streams closed, and the execution settled. Nothing is left + /// to give up on then but the delivery of the last messages, which the transport paces. + fn is_cleaned_up(&self) -> bool { + self.launch.is_some() + && self.exit.is_some() + && self.outputs.iter().all(|output| output.closed) + && self.settling == Settling::Settled + } + + /// Returns the terminal outcome once everything that it reports is known: the launch + /// result, the workload's status, both streams' ends, and the settlement. + fn outcome(&self) -> Option { + let delivered = self + .outputs + .iter() + .all(|output| output.is_done() && (output.ended || self.detached)); + let settled = matches!(self.settling, Settling::Settled | Settling::Fenced); + if self.awaits_launch() || self.exit.is_none() || !delivered || !settled { + return None; + } + let mut outcome = Outcome::new(self.exit.flatten()); + if let Some(cause) = self.cause { + outcome = outcome.with_cause(cause); + } + if let Some(Err(failure)) = self.launch { + outcome = outcome.with_launch_failure(failure); + } + for stream in Stream::ALL { + if self.outputs[stream.index()].truncated { + outcome = outcome.with_truncated(stream); + } + } + if self.settling == Settling::Settled { + outcome = outcome.with_reusable(); + } + Some(outcome) + } + + /// Notes the bytes that the runner holds, for its high-water mark. + fn note_buffered(&mut self) { + self.high_water = self.high_water.max(self.buffered()); + } +} diff --git a/nvx-rs/src/exec/tests/credit.rs b/nvx-rs/src/exec/tests/credit.rs new file mode 100644 index 000000000..4a91b8fb8 --- /dev/null +++ b/nvx-rs/src/exec/tests/credit.rs @@ -0,0 +1,452 @@ +//! The credit arithmetic: initial grants, window edges, tiny and maximal chunks and frames, a +//! shared pool, invalid grants and over-credit, and partial consumption. + +use super::*; + +/// Returns a peer of `limits`, created now, whose first grants the runner took. +fn granted_peer(limits: Limits) -> (Peer, Vec) { + let mut peer = Peer::new(KEY, limits, timers(), Instant::now()); + let grants = std::iter::from_fn(|| peer.next_message(&mut [])).collect(); + (peer, grants) +} + +/// Returns a runner of `limits`, created now, whose workload started. +fn started_runner(limits: Limits) -> Runner { + let mut runner = Runner::new(KEY, limits, timers(), Instant::now()); + runner.launched(Ok(())); + runner +} + +/// Returns every message that `runner` returns now, with an output buffer of `capacity` bytes, +/// and each output's bytes. +fn drain(runner: &mut Runner, capacity: usize) -> Vec { + let mut buffer = vec![0; capacity]; + std::iter::from_fn(|| { + runner + .next_message(&mut buffer) + .map(|message| to_peer(message, &buffer)) + }) + .collect() +} + +#[test] +fn each_receiver_first_grants_its_whole_window_once() { + let limits = bytes(4, 10, 8); + let (_, grants) = granted_peer(limits); + assert_eq!( + grants, + [ + peer::Message::Credit { + pool: Pool::Stdout, + units: 8 + }, + peer::Message::Credit { + pool: Pool::Stderr, + units: 8 + }, + ] + ); + let (_, grants) = granted_peer(limits.shared_output()); + assert_eq!( + grants, + [peer::Message::Credit { + pool: Pool::Output, + units: 8 + }] + ); + // The runner grants the input's window before the workload's launch, so that the peer may + // send input with its request. + let mut runner = Runner::new(KEY, limits, timers(), Instant::now()); + assert_eq!(drain(&mut runner, 0), [ToPeer::Credit(10)]); + runner.launched(Ok(())); + assert_eq!(drain(&mut runner, 0), [ToPeer::Started]); +} + +#[test] +fn a_byte_window_admits_data_up_to_its_edge_and_no_further() { + let limits = bytes(8, 8, 8); + let mut peer = Peer::new(KEY, limits, timers(), Instant::now()); + // Before its first grant, the window admits nothing. + assert_eq!( + peer.output(KEY, Stream::Stdout, b"a"), + Err(Violation::Window) + ); + while peer.next_message(&mut []).is_some() {} + assert_eq!( + peer.output(KEY, Stream::Stdout, b"12345"), + Ok(Receipt::Applied) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"6789"), + Err(Violation::Window) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"678"), + Ok(Receipt::Applied) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"9"), + Err(Violation::Window) + ); + // The other stream has a window of its own, and a refused message changed nothing. + assert_eq!( + peer.output(KEY, Stream::Stderr, b"abcdefgh"), + Ok(Receipt::Applied) + ); + assert_eq!(peer.pending_output(Stream::Stdout), b"12345678"); + // What the sink takes opens the window again, by as much. + peer.consumed(Stream::Stdout, 1); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Stdout, + units: 1 + }) + ); + assert_eq!(peer.output(KEY, Stream::Stdout, b"9"), Ok(Receipt::Applied)); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"0"), + Err(Violation::Window) + ); +} + +#[test] +fn data_messages_carry_at_least_a_byte_and_at_most_a_chunk() { + let limits = bytes(4, 64, 64); + let (mut peer, _) = granted_peer(limits); + assert_eq!(peer.output(KEY, Stream::Stdout, b""), Err(Violation::Chunk)); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"12345"), + Err(Violation::Chunk) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"1234"), + Ok(Receipt::Applied) + ); + let mut runner = Runner::new(KEY, limits, timers(), Instant::now()); + assert_eq!(runner.input(KEY, b""), Err(Violation::Chunk)); + while runner.next_message(&mut []).is_some() {} + assert_eq!(runner.input(KEY, b"12345"), Err(Violation::Chunk)); + assert_eq!(runner.input(KEY, b"1234"), Ok(Receipt::Applied)); +} + +#[test] +fn the_runner_sends_the_most_that_its_credit_a_chunk_and_the_buffer_allow() { + // A chunk of one byte sends each byte alone. + let mut runner = started_runner(bytes(1, 8, 8)); + assert_eq!(runner.read_allowance(Stream::Stdout), 1); + assert_eq!(runner.output(Stream::Stdout, b"ab"), 1); + runner.credit(KEY, Pool::Stdout, 8).unwrap(); + assert_eq!( + drain(&mut runner, 8), + [ + ToPeer::Credit(8), + ToPeer::Started, + ToPeer::Output(Stream::Stdout, b"a".to_vec()) + ] + ); + assert_eq!(runner.output(Stream::Stdout, b"b"), 1); + assert_eq!( + drain(&mut runner, 8), + [ToPeer::Output(Stream::Stdout, b"b".to_vec())] + ); + + // A larger chunk sends what the credit allows, then the rest once credit returns. + let mut runner = started_runner(bytes(6, 8, 8)); + assert_eq!(runner.output(Stream::Stdout, b"abcdef"), 6); + assert_eq!(runner.read_allowance(Stream::Stdout), 0); + runner.credit(KEY, Pool::Stdout, 4).unwrap(); + let messages = drain(&mut runner, 8); + assert_eq!( + messages[2..], + [ToPeer::Output(Stream::Stdout, b"abcd".to_vec())] + ); + // The queue has room again for what it sent, one chunk ahead of the credit. + assert_eq!(runner.read_allowance(Stream::Stdout), 4); + runner.credit(KEY, Pool::Stdout, 4).unwrap(); + // A small buffer bounds a message too. + let mut small = [0u8; 1]; + assert_eq!( + runner.next_message(&mut small), + Some(runner::Message::Output { + stream: Stream::Stdout, + length: 1 + }) + ); + assert_eq!(&small, b"e"); + assert_eq!( + drain(&mut runner, 8), + [ToPeer::Output(Stream::Stdout, b"f".to_vec())] + ); + // An empty buffer carries no output, but other messages still return. + assert_eq!(runner.output(Stream::Stdout, b"g"), 1); + runner.output_closed(Stream::Stderr); + assert_eq!(drain(&mut runner, 0), [ToPeer::End(Stream::Stderr)]); +} + +#[test] +fn a_frame_window_charges_one_unit_per_message_whatever_its_length() { + let limits = Limits::new(16, Window::frames(2), Window::frames(2)).unwrap(); + let (mut peer, grants) = granted_peer(limits); + assert_eq!( + grants[0], + peer::Message::Credit { + pool: Pool::Stdout, + units: 2 + } + ); + assert_eq!(peer.output(KEY, Stream::Stdout, b"x"), Ok(Receipt::Applied)); + assert_eq!( + peer.output(KEY, Stream::Stdout, &[7; 16]), + Ok(Receipt::Applied) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"y"), + Err(Violation::Window) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, &[7; 17]), + Err(Violation::Chunk) + ); + // A frame's unit returns only once the sink took its last byte. + peer.consumed(Stream::Stdout, 1); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Stdout, + units: 1 + }) + ); + peer.consumed(Stream::Stdout, 15); + assert_eq!(peer.next_message(&mut []), None); + peer.consumed(Stream::Stdout, 1); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Stdout, + units: 1 + }) + ); + + // The runner fills each frame with up to a chunk of what it read, whatever the reads. + let mut runner = started_runner(limits); + runner.credit(KEY, Pool::Stdout, 1).unwrap(); + for byte in b"abc" { + assert_eq!(runner.output(Stream::Stdout, &[*byte]), 1); + } + let messages = drain(&mut runner, 16); + assert_eq!( + messages[2..], + [ToPeer::Output(Stream::Stdout, b"abc".to_vec())] + ); + // With no frame left, it reads one chunk ahead and holds it. + assert_eq!(runner.output(Stream::Stdout, &[1; 20]), 16); + assert!(drain(&mut runner, 16).is_empty()); + runner.credit(KEY, Pool::Stdout, 2).unwrap(); + assert_eq!( + drain(&mut runner, 16), + [ToPeer::Output(Stream::Stdout, vec![1; 16])] + ); +} + +#[test] +fn a_shared_pool_charges_both_streams_and_answers_to_its_own_name() { + let limits = Limits::new(8, Window::bytes(8), Window::frames(3)) + .unwrap() + .shared_output(); + let (mut peer, _) = granted_peer(limits); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"o1"), + Ok(Receipt::Applied) + ); + assert_eq!( + peer.output(KEY, Stream::Stderr, b"e1"), + Ok(Receipt::Applied) + ); + assert_eq!( + peer.output(KEY, Stream::Stdout, b"o2"), + Ok(Receipt::Applied) + ); + assert_eq!( + peer.output(KEY, Stream::Stderr, b"e2"), + Err(Violation::Window) + ); + // A stalled standard output holds the pool that standard error shares. + peer.consumed(Stream::Stderr, 2); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Output, + units: 1 + }) + ); + assert_eq!( + peer.output(KEY, Stream::Stderr, b"e2"), + Ok(Receipt::Applied) + ); + + let mut runner = started_runner(limits); + assert_eq!(runner.credit(KEY, Pool::Stdout, 1), Err(Violation::Pool)); + assert_eq!(runner.credit(KEY, Pool::Input, 1), Err(Violation::Pool)); + runner.credit(KEY, Pool::Output, 2).unwrap(); + // Each stream reads a chunk ahead, and both draw on the pool's two frames, in turns. + assert_eq!(runner.output(Stream::Stdout, b"out"), 3); + assert_eq!(runner.output(Stream::Stderr, b"err"), 3); + let messages = drain(&mut runner, 8); + assert_eq!( + messages[2..], + [ + ToPeer::Output(Stream::Stdout, b"out".to_vec()), + ToPeer::Output(Stream::Stderr, b"err".to_vec()), + ] + ); + assert_eq!(runner.output(Stream::Stdout, b"more"), 4); + assert!(drain(&mut runner, 8).is_empty()); + // Per-stream pools answer to their own names only. + let mut runner = started_runner(bytes(8, 8, 8)); + assert_eq!(runner.credit(KEY, Pool::Output, 1), Err(Violation::Pool)); +} + +#[test] +fn invalid_grants_and_over_credit_are_refused_and_change_nothing() { + let mut runner = started_runner(bytes(4, 8, 8)); + assert_eq!(runner.credit(KEY, Pool::Stdout, 0), Err(Violation::Credit)); + assert_eq!(runner.credit(KEY, Pool::Stdout, 9), Err(Violation::Credit)); + runner.credit(KEY, Pool::Stdout, 5).unwrap(); + // More than the units in flight would leave the runner more than the window. + assert_eq!(runner.credit(KEY, Pool::Stdout, 4), Err(Violation::Credit)); + runner.credit(KEY, Pool::Stdout, 3).unwrap(); + assert_eq!(runner.credit(KEY, Pool::Stdout, 1), Err(Violation::Credit)); + // Eight bytes of credit send eight bytes, and no more, in chunks of four. + let mut sent = 0; + for _ in 0..4 { + sent += runner.output(Stream::Stdout, b"abcd"); + for message in drain(&mut runner, 4) { + if let ToPeer::Output(_, bytes) = message { + assert!(bytes.len() <= 4); + } + } + } + assert_eq!(sent, 12); + assert_eq!(runner.read_allowance(Stream::Stdout), 0); + // Once the eight bytes are in flight, the peer may return them, and no more. + assert_eq!(runner.credit(KEY, Pool::Stdout, 9), Err(Violation::Credit)); + runner.credit(KEY, Pool::Stdout, 8).unwrap(); + + let mut peer = Peer::new(KEY, bytes(4, 8, 8), timers(), Instant::now()); + assert_eq!(peer.credit(KEY, 0), Err(Violation::Credit)); + assert_eq!(peer.credit(KEY, 9), Err(Violation::Credit)); + peer.credit(KEY, 8).unwrap(); + assert_eq!(peer.credit(KEY, 1), Err(Violation::Credit)); +} + +#[test] +fn maximal_windows_and_chunks_neither_overflow_nor_admit_more_than_the_window() { + let limits = Limits::new(u32::MAX, Window::bytes(u32::MAX), Window::frames(u32::MAX)).unwrap(); + let mut runner = started_runner(limits); + runner.credit(KEY, Pool::Stdout, u32::MAX).unwrap(); + assert_eq!(runner.credit(KEY, Pool::Stdout, 1), Err(Violation::Credit)); + runner.credit(KEY, Pool::Stderr, u32::MAX - 1).unwrap(); + runner.credit(KEY, Pool::Stderr, 1).unwrap(); + // Two chunks of output and the input's window, and two windows of maximal frames. + assert_eq!(runner.memory_bound(), (u32::MAX as usize).saturating_mul(3)); + let peer = Peer::new(KEY, limits, timers(), Instant::now()); + assert_eq!(peer.memory_bound(), usize::MAX); + // Empty chunks and windows are refused. + assert!(Limits::new(0, Window::bytes(1), Window::bytes(1)).is_none()); + assert!(Limits::new(1, Window::bytes(0), Window::bytes(1)).is_none()); + assert!(Limits::new(1, Window::bytes(1), Window::frames(0)).is_none()); +} + +#[test] +fn partial_consumption_returns_only_what_the_sink_took() { + let (mut peer, _) = granted_peer(bytes(8, 8, 8)); + peer.output(KEY, Stream::Stdout, b"abcdef").unwrap(); + assert_eq!(peer.pending_output(Stream::Stdout), b"abcdef"); + peer.consumed(Stream::Stdout, 2); + assert_eq!(peer.pending_output(Stream::Stdout), b"cdef"); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Stdout, + units: 2 + }) + ); + // Credits coalesce until the adapter retrieves them. + peer.consumed(Stream::Stdout, 1); + peer.consumed(Stream::Stdout, 2); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Stdout, + units: 3 + }) + ); + // Taking more than is pending takes what is pending. + peer.consumed(Stream::Stdout, 100); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Credit { + pool: Pool::Stdout, + units: 1 + }) + ); + assert_eq!(peer.next_message(&mut []), None); + + // The runner returns input credit for what the workload took, as it takes it. + let mut runner = started_runner(bytes(8, 8, 8)); + drain(&mut runner, 0); + runner.input(KEY, b"12345678").unwrap(); + assert_eq!(runner.input(KEY, b"9"), Err(Violation::Window)); + assert_eq!(runner.pending_input(), b"12345678"); + runner.input_written(3); + assert_eq!(runner.pending_input(), b"45678"); + assert_eq!(drain(&mut runner, 0), [ToPeer::Credit(3)]); + runner.input(KEY, b"9ab").unwrap(); + assert_eq!(runner.input(KEY, b"c"), Err(Violation::Window)); +} + +#[test] +fn the_peer_sends_input_within_the_runner_s_credit() { + let limits = bytes(4, 6, 8); + let mut peer = Peer::new(KEY, limits, timers(), Instant::now()); + let mut buffer = [0u8; 4]; + // The source may hand over one chunk ahead of the credit. + assert_eq!(peer.write_allowance(), 4); + assert_eq!(peer.input(b"abcdefgh"), 4); + assert_eq!(peer.write_allowance(), 0); + let messages = std::iter::from_fn(|| peer.next_message(&mut buffer)).collect::>(); + assert!( + messages + .iter() + .all(|message| matches!(message, peer::Message::Credit { .. })), + "{messages:?}" + ); + peer.credit(KEY, 6).unwrap(); + assert_eq!( + peer.next_message(&mut buffer), + Some(peer::Message::Input { length: 4 }) + ); + assert_eq!(&buffer, b"abcd"); + assert_eq!(peer.input(b"efgh"), 4); + assert_eq!( + peer.next_message(&mut buffer), + Some(peer::Message::Input { length: 2 }) + ); + assert_eq!(&buffer[..2], b"ef"); + assert_eq!(peer.next_message(&mut buffer), None); + // The end follows the last byte. + peer.end_input(); + assert_eq!(peer.write_allowance(), 0); + assert_eq!(peer.next_message(&mut buffer), None); + peer.credit(KEY, 4).unwrap(); + assert_eq!( + peer.next_message(&mut buffer), + Some(peer::Message::Input { length: 2 }) + ); + assert_eq!( + peer.next_message(&mut buffer), + Some(peer::Message::InputEnd) + ); + assert_eq!(peer.next_message(&mut buffer), None); +} diff --git a/nvx-rs/src/exec/tests/keys.rs b/nvx-rs/src/exec/tests/keys.rs new file mode 100644 index 000000000..9748687a7 --- /dev/null +++ b/nvx-rs/src/exec/tests/keys.rs @@ -0,0 +1,136 @@ +//! Messages of an earlier generation, of another execution or a later generation, and messages +//! that arrive after the side no longer needs them. + +use super::*; + +/// The key of the same execution's earlier generation. +const EARLIER: Key = Key::new(KEY.id(), KEY.generation() - 1); +/// The key of the same execution's later generation. +const LATER: Key = Key::new(KEY.id(), KEY.generation() + 1); +/// The key of another execution. +const OTHER: Key = Key::new(KEY.id() + 1, KEY.generation()); + +#[test] +fn messages_of_an_earlier_generation_are_stale_and_change_nothing() { + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(8, 8, 8), timers(), now); + runner.launched(Ok(())); + while runner.next_message(&mut []).is_some() {} + assert_eq!(runner.credit(EARLIER, Pool::Stdout, 8), Ok(Receipt::Stale)); + assert_eq!(runner.input(EARLIER, b"stale"), Ok(Receipt::Stale)); + assert_eq!(runner.end_input(EARLIER), Ok(Receipt::Stale)); + assert_eq!( + runner.cancel(now, EARLIER, Cause::Cancelled), + Ok(Receipt::Stale) + ); + // No credit arrived, so the output waits; no input arrived; nothing ends the workload. + runner.output(Stream::Stdout, b"out"); + assert_eq!(runner.next_message(&mut [0; 8]), None); + assert_eq!(runner.pending_input(), b""); + assert_eq!(runner.next_request(), None); + assert_eq!(runner.input(KEY, b"12345678"), Ok(Receipt::Applied)); + + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + while peer.next_message(&mut []).is_some() {} + assert_eq!(peer.credit(EARLIER, 8), Ok(Receipt::Stale)); + assert_eq!(peer.started(EARLIER), Ok(Receipt::Stale)); + assert_eq!( + peer.output(EARLIER, Stream::Stdout, b"stale"), + Ok(Receipt::Stale) + ); + assert_eq!(peer.end_output(EARLIER, Stream::Stdout), Ok(Receipt::Stale)); + assert_eq!(peer.input_closed(EARLIER), Ok(Receipt::Stale)); + let outcome = Outcome::new(Some(Status::Exited(0))); + assert_eq!(peer.terminal(now, EARLIER, outcome), Ok(Receipt::Stale)); + assert_eq!(peer.pending_output(Stream::Stdout), b""); + assert_eq!(peer.next_event(), None); + assert!(!peer.has_started()); + assert_eq!(peer.input(b"in"), 2); + assert_eq!(peer.next_message(&mut [0; 8]), None); + // The window of the current generation is whole. + assert_eq!( + peer.output(KEY, Stream::Stdout, b"12345678"), + Ok(Receipt::Applied) + ); +} + +#[test] +fn messages_of_another_execution_or_a_later_generation_are_refused() { + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(8, 8, 8), timers(), now); + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + for key in [LATER, OTHER] { + assert_eq!(runner.credit(key, Pool::Stdout, 1), Err(Violation::Key)); + assert_eq!(runner.input(key, b"x"), Err(Violation::Key)); + assert_eq!(runner.end_input(key), Err(Violation::Key)); + assert_eq!( + runner.cancel(now, key, Cause::Cancelled), + Err(Violation::Key) + ); + assert_eq!(peer.credit(key, 1), Err(Violation::Key)); + assert_eq!(peer.started(key), Err(Violation::Key)); + assert_eq!(peer.output(key, Stream::Stdout, b"x"), Err(Violation::Key)); + assert_eq!(peer.end_output(key, Stream::Stdout), Err(Violation::Key)); + assert_eq!(peer.input_closed(key), Err(Violation::Key)); + assert_eq!( + peer.terminal(now, key, Outcome::new(None)), + Err(Violation::Key) + ); + } + assert_eq!(runner.next_request(), None); + assert!(!peer.has_started()); +} + +#[test] +fn messages_after_the_execution_ended_are_late() { + let mut pair = Pair::new( + bytes(64, 64, 64), + timers(), + Workload::new(100, 0), + Client::new(), + ); + pair.finish(); + let now = pair.now; + let runner = &mut pair.runner; + assert_eq!(runner.credit(KEY, Pool::Stdout, 1), Ok(Receipt::Late)); + assert_eq!(runner.input(KEY, b"late"), Ok(Receipt::Late)); + assert_eq!(runner.end_input(KEY), Ok(Receipt::Late)); + assert_eq!(runner.cancel(now, KEY, Cause::Cancelled), Ok(Receipt::Late)); + // A stale message stays stale, and a foreign one refused. + assert_eq!(runner.credit(EARLIER, Pool::Stdout, 1), Ok(Receipt::Stale)); + assert_eq!(runner.credit(OTHER, Pool::Stdout, 1), Err(Violation::Key)); + assert_eq!(runner.next_message(&mut [0; 64]), None); + assert_eq!(runner.next_request(), None); + assert_eq!(runner.deadline(), None); + + let peer = &mut pair.peer; + assert_eq!(peer.credit(KEY, 1), Ok(Receipt::Late)); + assert_eq!(peer.started(KEY), Ok(Receipt::Late)); + assert_eq!(peer.output(KEY, Stream::Stdout, b"late"), Ok(Receipt::Late)); + assert_eq!(peer.end_output(KEY, Stream::Stdout), Ok(Receipt::Late)); + assert_eq!(peer.input_closed(KEY), Ok(Receipt::Late)); + assert_eq!( + peer.terminal(now, KEY, Outcome::new(None)), + Ok(Receipt::Late) + ); + assert_eq!(peer.next_message(&mut [0; 64]), None); + assert_eq!(peer.next_event(), None); + assert_eq!(peer.deadline(), None); +} + +#[test] +fn a_cancellation_after_a_termination_began_is_late() { + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(8, 8, 8), timers(), now); + runner.launched(Ok(())); + assert_eq!( + runner.cancel(now, KEY, Cause::TimedOut), + Ok(Receipt::Applied) + ); + assert_eq!(runner.cancel(now, KEY, Cause::Cancelled), Ok(Receipt::Late)); + assert_eq!( + runner.next_request(), + Some(runner::Request::Terminate(Cause::TimedOut)) + ); + assert_eq!(runner.next_request(), None); +} diff --git a/nvx-rs/src/exec/tests/liveness.rs b/nvx-rs/src/exec/tests/liveness.rs new file mode 100644 index 000000000..e9365d358 --- /dev/null +++ b/nvx-rs/src/exec/tests/liveness.rs @@ -0,0 +1,400 @@ +//! Liveness: a stalled sink, an exhausted window, or a full transport holds back only the bytes +//! that wait on it. Cancellations, timeouts, resets, and drops still end the execution by their +//! deadlines, and another execution keeps streaming meanwhile. Credit that frees up before every +//! message takes turns with each side's other messages, which still go out, and a stream's end +//! goes out while the other stream's output keeps flowing. + +use super::*; + +/// The limits of these executions. +fn limits() -> Limits { + bytes(4096, 4096, 16 * 1024) +} + +/// Returns an execution whose workload writes a mebibyte to each stream to a sink that takes +/// nothing. +fn stalled() -> Pair { + let mut client = Client::new(); + client.rates = [0, 0]; + Pair::new(limits(), timers(), Workload::new(1 << 20, 1 << 20), client) +} + +/// Returns an execution whose workload streams `length` bytes of standard output to a sink that +/// takes 8 KiB in each step. +fn streaming(length: usize) -> Pair { + let mut client = Client::new(); + client.rates = [8 * 1024, 8 * 1024]; + Pair::new(limits(), timers(), Workload::new(length, 0), client) +} + +/// Steps `first` and `second` in turns, `count` times, as one adapter's loop drives both. +fn step_both(first: &mut Pair, second: &mut Pair, count: usize) { + for _ in 0..count { + first.step(); + second.step(); + } +} + +/// Moves both executions' clocks to `now`. +fn set_clock(first: &mut Pair, second: &mut Pair, now: Instant) { + first.now = now; + second.now = now; +} + +/// Asserts that `pair`'s execution is stalled: the runner reads nothing, the workload still +/// runs, the sink took nothing, and the peer holds no more than its windows. +fn assert_stalled(pair: &Pair) { + for stream in Stream::ALL { + assert_eq!(pair.runner.read_allowance(stream), 0); + } + assert!(pair.workload.status.is_none()); + assert!(pair.peer.buffered() <= 2 * 16 * 1024); + assert_eq!(pair.client.received, [0, 0]); +} + +#[test] +fn a_stalled_sink_with_an_exhausted_window_stalls_only_its_own_execution() { + let mut stalled = stalled(); + let mut other = streaming(8 << 20); + for _ in 0..STEPS { + if other.peer.is_finished() { + break; + } + step_both(&mut stalled, &mut other, 1); + } + let outcome = other + .client + .outcome() + .expect("the other execution finished"); + assert_eq!(other.client.received, [8 << 20, 0]); + assert!(!outcome.is_truncated(Stream::Stdout)); + // The stalled execution waits for its sink, with no timer that would end it. + assert_stalled(&stalled); + assert!(!stalled.runner.is_finished() && !stalled.peer.is_finished()); + assert_eq!(stalled.runner.deadline(), None); + assert_eq!(stalled.peer.deadline(), None); +} + +/// How a liveness check ends the stalled execution. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Ending { + /// The peer's caller cancels it. + Cancel, + /// The runner's timeout passes. + Timeout, + /// The session is reset. + Reset, + /// The peer's caller drops its handle. + Drop, +} + +#[test] +fn endings_finish_by_their_deadlines_while_a_sink_is_stalled_and_another_execution_streams() { + const TIMEOUT: Duration = Duration::from_secs(30); + for ending in [Ending::Cancel, Ending::Timeout, Ending::Reset, Ending::Drop] { + let mut stalled = stalled(); + if ending == Ending::Timeout { + stalled.runner = Runner::new(KEY, limits(), timers().timeout(TIMEOUT), stalled.now); + } + let mut other = streaming(8 << 20); + let start = stalled.now; + other.now = start; + step_both(&mut stalled, &mut other, 50); + assert_stalled(&stalled); + let ended = match ending { + Ending::Timeout => start + TIMEOUT, + _ => start + Duration::from_secs(1), + }; + set_clock(&mut stalled, &mut other, ended); + match ending { + Ending::Cancel => stalled.peer.cancel(ended), + Ending::Timeout => {} + Ending::Reset => { + stalled.runner.reset(ended); + stalled.peer.reset(ended); + } + Ending::Drop => stalled.peer.dropped(ended), + } + let streamed = other.client.received[0]; + step_both(&mut stalled, &mut other, 50); + // The workload ended at once, whatever its streams, and the other execution streamed on. + assert_eq!( + stalled.workload.status, + Some(Status::Signaled(9)), + "{ending:?}" + ); + assert!(other.client.received[0] > streamed, "{ending:?}"); + // A cancellation and a timeout leave the pending output its grace, as the deadline shows. + // A reset leaves no peer and a drop no sink to wait for, so their output goes at once. + if matches!(ending, Ending::Cancel | Ending::Timeout) { + assert!(!stalled.runner.is_finished(), "{ending:?}"); + assert_eq!(stalled.runner.deadline(), Some(ended + GRACE), "{ending:?}"); + } + set_clock(&mut stalled, &mut other, ended + GRACE); + let streamed = other.client.received[0]; + step_both(&mut stalled, &mut other, 10); + assert!(stalled.runner.is_finished(), "{ending:?}"); + // The peer learns of the runner's timeout only from the terminal outcome, which leaves + // its own sink a grace of its own. + if ending == Ending::Timeout { + assert!(!stalled.peer.is_finished()); + assert_eq!(stalled.peer.deadline(), Some(ended + GRACE + GRACE)); + set_clock(&mut stalled, &mut other, ended + GRACE + GRACE); + step_both(&mut stalled, &mut other, 10); + } + assert!(stalled.peer.is_finished(), "{ending:?}"); + assert!(other.client.received[0] > streamed, "{ending:?}"); + let cause = match ending { + Ending::Cancel => Cause::Cancelled, + Ending::Timeout => Cause::TimedOut, + Ending::Reset => Cause::Reset, + Ending::Drop => Cause::Dropped, + }; + let terminal = stalled.terminal(); + assert_eq!(terminal.cause(), Some(cause), "{ending:?}"); + assert_eq!(terminal.status(), Some(Status::Signaled(9)), "{ending:?}"); + assert!(terminal.is_reusable(), "{ending:?}"); + // A dropping peer takes and drops everything at once, so the runner discards nothing. + assert_eq!( + terminal.is_truncated(Stream::Stdout), + ending != Ending::Drop, + "{ending:?}" + ); + let outcome = stalled.client.outcome().unwrap(); + assert_eq!(outcome.cause(), Some(cause), "{ending:?}"); + assert!(outcome.is_truncated(Stream::Stdout), "{ending:?}"); + // The peer knows the runner's settlement unless the reset kept it from hearing. + assert_eq!(outcome.is_reusable(), ending != Ending::Reset, "{ending:?}"); + // The other execution still finishes whole. + other.finish(); + assert_eq!(other.client.received, [8 << 20, 0]); + assert!(!other.client.outcome().unwrap().is_truncated(Stream::Stdout)); + } +} + +#[test] +fn a_full_transport_holds_back_only_the_messages_that_wait_for_it() { + // The transport drains within the grace after the cancellation, or once the settlement time + // after the grace has passed too: the runner's outcome tells how the execution's cleanup + // went, however long the transport held it. + for held in [GRACE, GRACE + SETTLEMENT] { + let mut blocked = Pair::new(limits(), timers(), Workload::new(1 << 20, 0), Client::new()); + let mut other = streaming(8 << 20); + // The transport from the runner to the peer stops carrying messages, and fills before + // the credit runs out. + blocked.delivers_to_peer = false; + blocked.capacity = 3; + step_both(&mut blocked, &mut other, 50); + assert_eq!(blocked.to_peer.len(), 3); + assert_eq!(blocked.runner.read_allowance(Stream::Stdout), 0); + assert!(blocked.workload.status.is_none()); + let cancelled = blocked.now; + blocked.peer.cancel(cancelled); + let streamed = other.client.received[0]; + step_both(&mut blocked, &mut other, 50); + // The cancellation went the other way, and the runner ended the workload and settled + // it, though it could send nothing. + assert_eq!(blocked.workload.status, Some(Status::Signaled(9))); + assert!(blocked.workload.requests.contains(&runner::Request::Settle)); + assert!(other.client.received[0] > streamed); + set_clock(&mut blocked, &mut other, cancelled + GRACE); + step_both(&mut blocked, &mut other, 10); + assert!( + blocked + .workload + .requests + .contains(&runner::Request::Discard(Stream::Stdout)) + ); + // The execution's cleanup is complete, so the runner has no timer left: only the + // transport holds its last messages. + assert_eq!(blocked.workload.closed, [true, true]); + assert_eq!(blocked.runner.deadline(), None); + set_clock(&mut blocked, &mut other, cancelled + held); + step_both(&mut blocked, &mut other, 10); + // Once the transport drains, the runner sends no stale output: only both ends and the + // terminal outcome follow what the transport held. + let sent = blocked.sent_by_runner.len(); + blocked.delivers_to_peer = true; + blocked.capacity = usize::MAX; + blocked.finish(); + let after = &blocked.sent_by_runner[sent..]; + assert!( + after.iter().all(|message| matches!( + message, + runner::Message::End(_) | runner::Message::Terminal(_) + )), + "{after:?}" + ); + let terminal = blocked.terminal(); + assert_eq!(terminal.cause(), Some(Cause::Cancelled), "{held:?}"); + assert!(terminal.is_truncated(Stream::Stdout), "{held:?}"); + assert!(terminal.is_reusable(), "{held:?}"); + // The peer takes the terminal outcome while it waits for it; once it gave up on it, its + // own outcome stands, fenced. + let outcome = blocked.client.outcome().unwrap(); + assert_eq!(outcome.cause(), Some(Cause::Cancelled), "{held:?}"); + assert!(outcome.is_truncated(Stream::Stdout), "{held:?}"); + assert_eq!(outcome.is_reusable(), held == GRACE, "{held:?}"); + other.finish(); + assert_eq!(other.client.received, [8 << 20, 0]); + } +} + +#[test] +fn input_credit_takes_turns_with_the_runner_s_other_messages() { + // An adapter takes one message of the runner at a time, and before each, hands it the + // workload's output as far as it reads ahead, and all the input that the peer's credit + // allows, which the workload takes at once: credit of the input frees up before every + // message. + let mut runner = Runner::new(KEY, bytes(4, 4, 16), timers(), Instant::now()); + runner.launched(Ok(())); + runner + .credit(KEY, Pool::Stdout, 16) + .expect("the runner takes the peer's grant"); + runner.output_closed(Stream::Stderr); + let output = b"abcdefgh"; + let mut read = 0; + let mut buffer = [0; 4]; + let mut credit = 0; + let mut written = 0; + let mut others = Vec::new(); + for _ in 0..16 { + if read < output.len() { + let length = runner + .read_allowance(Stream::Stdout) + .min(output.len() - read); + read += runner.output(Stream::Stdout, &output[read..read + length]); + if read == output.len() { + runner.output_closed(Stream::Stdout); + } + } + if credit > 0 { + let input = vec![b'i'; credit]; + deliver_to_runner(&mut runner, Instant::now(), ToRunner::Input(input)); + runner.input_written(credit); + written += credit; + credit = 0; + } + match runner + .next_message(&mut buffer) + .map(|message| to_peer(message, &buffer)) + { + Some(ToPeer::Credit(units)) => credit += units as usize, + Some(message) => others.push(message), + None => break, + } + } + // The workload's start, the end of standard error, which closed with no byte, the output, + // and the end of standard output went out, in turns with the credit, which kept the input + // flowing. + assert_eq!( + others, + [ + ToPeer::Started, + ToPeer::End(Stream::Stderr), + ToPeer::Output(Stream::Stdout, b"abcd".to_vec()), + ToPeer::Output(Stream::Stdout, b"efgh".to_vec()), + ToPeer::End(Stream::Stdout), + ] + ); + assert!(written >= 4 * 4, "{written}"); +} + +#[test] +fn a_stream_s_end_goes_out_while_the_other_stream_s_output_keeps_flowing() { + // An adapter takes one message of the runner at a time, and before each, hands it as much of + // one stream's output as it reads ahead, while the peer's sink takes each output message at + // once and grants its credit back: that stream always has bytes and credit. The other stream + // closed at once, with no byte. + for flowing in Stream::ALL { + let (closed, pool) = match flowing { + Stream::Stdout => (Stream::Stderr, Pool::Stdout), + Stream::Stderr => (Stream::Stdout, Pool::Stderr), + }; + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(4, 4, 16), timers(), now); + runner.launched(Ok(())); + deliver_to_runner(&mut runner, now, ToRunner::Credit(pool, 16)); + runner.output_closed(closed); + let mut buffer = [0; 4]; + let mut others = Vec::new(); + for _ in 0..16 { + let length = runner.read_allowance(flowing); + assert_eq!(runner.output(flowing, &vec![b'x'; length]), length); + let message = runner.next_message(&mut buffer).expect("output waits"); + match to_peer(message, &buffer) { + ToPeer::Credit(_) => {} + message => { + if let ToPeer::Output(_, bytes) = &message { + let units = u32::try_from(bytes.len()).unwrap(); + deliver_to_runner(&mut runner, now, ToRunner::Credit(pool, units)); + } + others.push(message); + } + } + } + // The closed stream's end went out right after the start, and the other stream's output + // flowed on, a full chunk in each message. + assert_eq!( + others[..2], + [ToPeer::Started, ToPeer::End(closed)], + "{flowing:?}" + ); + assert!(others.len() > 10, "{flowing:?}: {others:?}"); + assert!( + others[2..] + .iter() + .all(|message| *message == ToPeer::Output(flowing, b"xxxx".to_vec())), + "{flowing:?}: {others:?}" + ); + } +} + +#[test] +fn output_credit_takes_turns_with_the_peer_s_input_and_its_other_pool() { + // An adapter takes one message of the peer at a time, and before each, its source hands over + // as much input as the peer takes, and the runner sends a byte of standard output, which the + // sink takes at once: that pool's credit frees up before every message. + let mut peer = Peer::new(KEY, bytes(4, 16, 4), timers(), Instant::now()); + let mut buffer = [0; 4]; + while peer.next_message(&mut buffer).is_some() {} + peer.credit(KEY, 16) + .expect("the peer takes the runner's grant"); + peer.output(KEY, Stream::Stderr, b"zz") + .expect("the peer takes standard error within its window"); + peer.consumed(Stream::Stderr, 2); + let input = b"0123456789abcdef"; + let mut handed = 0; + let mut sent = Vec::new(); + for _ in 0..16 { + if handed < input.len() { + handed += peer.input(&input[handed..]); + if handed == input.len() { + peer.end_input(); + } + } + peer.output(KEY, Stream::Stdout, b"x") + .expect("the peer's credit keeps the runner's window open"); + peer.consumed(Stream::Stdout, 1); + let message = peer.next_message(&mut buffer).expect("credit waits"); + sent.push(to_runner(message, &buffer)); + } + // The input, its end, and the credit of standard error went out, in turns with the credit + // of standard output. + let others = sent + .iter() + .filter(|message| !matches!(message, ToRunner::Credit(Pool::Stdout, _))) + .cloned() + .collect::>(); + assert_eq!( + others, + [ + ToRunner::Input(b"0123".to_vec()), + ToRunner::Credit(Pool::Stderr, 2), + ToRunner::Input(b"4567".to_vec()), + ToRunner::Input(b"89ab".to_vec()), + ToRunner::Input(b"cdef".to_vec()), + ToRunner::InputEnd, + ] + ); +} diff --git a/nvx-rs/src/exec/tests/memory.rs b/nvx-rs/src/exec/tests/memory.rs new file mode 100644 index 000000000..a896fe2cf --- /dev/null +++ b/nvx-rs/src/exec/tests/memory.rs @@ -0,0 +1,89 @@ +//! Bounded memory: streams far longer than their windows arrive whole while each side holds no +//! more than its bound, with byte windows and with a shared pool of frames. + +use super::*; + +/// Asserts that neither side of `pair` ever held more than its bound, and returns both +/// high-water marks. +fn assert_bounded(pair: &Pair) -> (usize, usize) { + let runner = (pair.runner.high_water(), pair.runner.memory_bound()); + let peer = (pair.peer.high_water(), pair.peer.memory_bound()); + assert!(runner.0 <= runner.1, "the runner held {runner:?}"); + assert!(peer.0 <= peer.1, "the peer held {peer:?}"); + (runner.0, peer.0) +} + +#[test] +fn a_stream_far_longer_than_its_window_arrives_whole_in_bounded_memory() { + const STDOUT: usize = 64 << 20; + const STDERR: usize = 1 << 20; + const WINDOW: u32 = 64 * 1024; + let limits = bytes(16 * 1024, WINDOW, WINDOW); + let mut client = Client::new(); + client.rates = [48 * 1024, 5 * 1024]; + let mut pair = Pair::new(limits, timers(), Workload::new(STDOUT, STDERR), client); + let outcome = pair.finish(); + // The client checked each byte against the workload's pattern as it took it. + assert_eq!(pair.client.received, [STDOUT, STDERR]); + assert!(!outcome.is_truncated(Stream::Stdout) && !outcome.is_truncated(Stream::Stderr)); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + let (runner, peer) = assert_bounded(&pair); + // The runner held at most a chunk of each stream, and the peer at most both windows. + assert!(runner <= 2 * limits.chunk(), "{runner}"); + assert!(peer <= 2 * WINDOW as usize, "{peer}"); + assert!(pair.in_flight_high_water <= 2 * WINDOW as usize); +} + +#[test] +fn a_shared_pool_of_frames_carries_both_streams_whole_in_bounded_memory() { + // Frames of up to 64 KiB less 256 bytes, in a pool of 64 frames that both streams share. + const CHUNK: u32 = 64 * 1024 - 256; + const LENGTH: usize = 16 << 20; + let limits = Limits::new(CHUNK, Window::frames(64), Window::frames(64)) + .unwrap() + .shared_output(); + let mut client = Client::new(); + client.rates = [100 * 1024, 30 * 1024]; + let mut pair = Pair::new(limits, timers(), Workload::new(LENGTH, LENGTH / 4), client); + let outcome = pair.finish(); + assert_eq!(pair.client.received, [LENGTH, LENGTH / 4]); + assert!(!outcome.is_truncated(Stream::Stdout) && !outcome.is_truncated(Stream::Stderr)); + let (runner, peer) = assert_bounded(&pair); + assert!(runner <= 2 * CHUNK as usize, "{runner}"); + assert!(peer <= 64 * CHUNK as usize, "{peer}"); + // Each output message filled at most one frame. + assert!(pair.sent_by_runner.iter().all(|message| match message { + runner::Message::Output { length, .. } => *length <= CHUNK as usize, + _ => true, + })); +} + +#[test] +fn input_longer_than_its_window_arrives_whole() { + for limits in [ + bytes(4096, 16 * 1024, 16 * 1024), + Limits::new(1000, Window::frames(4), Window::bytes(4096)).unwrap(), + ] { + let mut workload = Workload::new(0, 0); + workload.reads_to_end = true; + workload.input_rate = 5_000; + let mut client = Client::new(); + client.input = (0..1usize << 20) + .map(|index| (index * 7 + index / 251) as u8) + .collect(); + let mut pair = Pair::new(limits, timers(), workload, client); + let outcome = pair.finish(); + assert_eq!(pair.workload.input, pair.client.input, "{limits:?}"); + assert!(pair.workload.input_end); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable()); + let (runner, peer) = assert_bounded(&pair); + // The runner held at most the input's window, and the peer at most a chunk of input. + let window = match limits.input().unit() { + Unit::Bytes => limits.input().size() as usize, + Unit::Frames => limits.input().size() as usize * limits.chunk(), + }; + assert!(runner <= window, "{runner}"); + assert!(peer <= limits.chunk(), "{peer}"); + } +} diff --git a/nvx-rs/src/exec/tests/mod.rs b/nvx-rs/src/exec/tests/mod.rs new file mode 100644 index 000000000..52f2cbc7b --- /dev/null +++ b/nvx-rs/src/exec/tests/mod.rs @@ -0,0 +1,577 @@ +//! Deterministic checks of the engine, with no operating-system calls: the credit arithmetic of +//! each side, the order of each stream, the completion rules, terminal races, bounded memory, and +//! liveness, with the sides connected by bounded queues, and a simulated workload and client. + +use std::collections::VecDeque; +use std::time::{Duration, Instant}; + +use super::{ + Cause, Key, LaunchFailure, Limits, Outcome, Peer, Pool, Receipt, Runner, Settlement, Status, + Stream, Timers, Unit, Violation, Window, peer, runner, +}; + +mod credit; +mod keys; +mod liveness; +mod memory; +mod races; +mod streams; +mod truncation; + +/// The key of the executions in these tests. +const KEY: Key = Key::new(7, 3); +/// The grace of pending output in these tests. +const GRACE: Duration = Duration::from_secs(1); +/// The settlement time in these tests. +const SETTLEMENT: Duration = Duration::from_secs(5); +/// The capacity of each simulated pipe between the workload and the runner's adapter. +const PIPE: usize = 64 * 1024; +/// The most steps that a test drives a pair through before it fails. +const STEPS: usize = 1_000_000; + +/// Returns limits with chunks of `chunk` bytes, an input window of `input` bytes, and an output +/// window of `output` bytes for each stream. +fn bytes(chunk: u32, input: u32, output: u32) -> Limits { + Limits::new(chunk, Window::bytes(input), Window::bytes(output)).unwrap() +} + +/// Returns the timers of these tests: a grace of [`GRACE`], a settlement time of +/// [`SETTLEMENT`], and no timeout. +fn timers() -> Timers { + Timers::new(GRACE, SETTLEMENT) +} + +/// Returns the byte at `position` of `stream`'s simulated output: a pattern that differs between +/// the streams and between nearby positions, so that a lost, repeated, or reordered byte shows. +fn pattern(stream: Stream, position: usize) -> u8 { + let salt = match stream { + Stream::Stdout => 0x5a, + Stream::Stderr => 0xa5, + }; + (position as u8) + ^ ((position >> 8) as u8).rotate_left(3) + ^ ((position >> 16) as u8).rotate_left(5) + ^ ((position >> 24) as u8).rotate_left(7) + ^ salt +} + +/// Returns `length` bytes of `stream`'s pattern from `position`. +fn patterned(stream: Stream, position: usize, length: usize) -> Vec { + (position..position + length) + .map(|position| pattern(stream, position)) + .collect() +} + +/// A message in flight from the peer to the runner, with its data. +#[derive(Clone, Debug, PartialEq, Eq)] +enum ToRunner { + /// A cancellation. + Cancel(Cause), + /// Credit of an output pool. + Credit(Pool, u32), + /// Input. + Input(Vec), + /// The input's end. + InputEnd, +} + +/// A message in flight from the runner to the peer, with its data. +#[derive(Clone, Debug, PartialEq, Eq)] +enum ToPeer { + /// Credit of the input. + Credit(u32), + /// The workload started. + Started, + /// Output. + Output(Stream, Vec), + /// A stream's end. + End(Stream), + /// The input closed early. + InputClosed, + /// The terminal outcome. + Terminal(Outcome), +} + +/// Returns the message that `message` describes, with the data that `buffer` holds. +fn to_runner(message: peer::Message, buffer: &[u8]) -> ToRunner { + match message { + peer::Message::Cancel(cause) => ToRunner::Cancel(cause), + peer::Message::Credit { pool, units } => ToRunner::Credit(pool, units), + peer::Message::Input { length } => ToRunner::Input(buffer[..length].to_vec()), + peer::Message::InputEnd => ToRunner::InputEnd, + } +} + +/// Returns the message that `message` describes, with the data that `buffer` holds. +fn to_peer(message: runner::Message, buffer: &[u8]) -> ToPeer { + match message { + runner::Message::Credit { units } => ToPeer::Credit(units), + runner::Message::Started => ToPeer::Started, + runner::Message::Output { stream, length } => { + ToPeer::Output(stream, buffer[..length].to_vec()) + } + runner::Message::End(stream) => ToPeer::End(stream), + runner::Message::InputClosed => ToPeer::InputClosed, + runner::Message::Terminal(outcome) => ToPeer::Terminal(outcome), + } +} + +/// Hands `message` to `runner` at `now`, which must accept it. +fn deliver_to_runner(runner: &mut Runner, now: Instant, message: ToRunner) -> Receipt { + let key = runner.key(); + match message { + ToRunner::Cancel(cause) => runner.cancel(now, key, cause), + ToRunner::Credit(pool, units) => runner.credit(key, pool, units), + ToRunner::Input(bytes) => runner.input(key, &bytes), + ToRunner::InputEnd => runner.end_input(key), + } + .expect("the runner accepts the peer's message") +} + +/// Hands `message` to `peer` at `now`, which must accept it. +fn deliver_to_peer(peer: &mut Peer, now: Instant, message: ToPeer) -> Receipt { + let key = peer.key(); + match message { + ToPeer::Credit(units) => peer.credit(key, units), + ToPeer::Started => peer.started(key), + ToPeer::Output(stream, bytes) => peer.output(key, stream, &bytes), + ToPeer::End(stream) => peer.end_output(key, stream), + ToPeer::InputClosed => peer.input_closed(key), + ToPeer::Terminal(outcome) => peer.terminal(now, key, outcome), + } + .expect("the peer accepts the runner's message") +} + +/// A simulated workload, as the runner's adapter drives it: a program that writes its output +/// through pipes of [`PIPE`] bytes, takes its input at a rate, and exits once it wrote all of its +/// output, after its input's end if it reads to it. +#[derive(Debug)] +struct Workload { + /// Whether the adapter reports a successful launch at the first step. + launches: bool, + /// Whether a process holds back what reports the launch, so that the adapter reports it only + /// once this is false again. + launch_held: bool, + /// Whether the adapter reported the launch. + launched: bool, + /// The bytes of each stream that the program writes in all. + lengths: [usize; 2], + /// The bytes of each stream that the program wrote into its pipe. + written: [usize; 2], + /// The bytes of each stream that the adapter read from the pipe. + read: [usize; 2], + /// Whether the adapter reported each stream's close. + closed: [bool; 2], + /// Whether the adapter discards each stream. + discarding: [bool; 2], + /// The bytes of each stream that the adapter discarded. + discarded: [usize; 2], + /// The input that the program took. + input: Vec, + /// The most input bytes that the program takes in one step. + input_rate: usize, + /// The input bytes after which the program closes its input, if it does. + input_limit: Option, + /// Whether the program's input closed. + input_closed: bool, + /// Whether the program saw its input's end. + input_end: bool, + /// Whether the program exits only once its input closed: at its end, or because the + /// program closed it. + reads_to_end: bool, + /// Whether the program exits once it has done its work; otherwise it runs until it is + /// ended. + exits: bool, + /// The program's own exit code. + code: i32, + /// The program's status, once it ended. + status: Option, + /// Whether the adapter reported the status. + reported: bool, + /// Whether a termination fails to end the program. + unkillable: bool, + /// Whether a process that the program left behind holds each stream's pipe open once the + /// program ended, until a settlement that succeeds ends it. + held: [bool; 2], + /// Whether the adapter reported that each stream hung up: no process holds its pipe any more. + hung_up: [bool; 2], + /// The settlement that the adapter reports once the runner requests it, if any. + settlement: Option, + /// Whether the runner requested the settlement and the adapter has not reported it. + settling: bool, + /// The requests that the runner made, in order. + requests: Vec, +} + +impl Workload { + /// Returns a workload that writes `stdout` and `stderr` bytes, takes all of its input at + /// once, exits with 0 without waiting for its input's end, and settles. + fn new(stdout: usize, stderr: usize) -> Self { + Self { + launches: true, + launch_held: false, + launched: false, + lengths: [stdout, stderr], + written: [0, 0], + read: [0, 0], + closed: [false, false], + discarding: [false, false], + discarded: [0, 0], + input: Vec::new(), + input_rate: usize::MAX, + input_limit: None, + input_closed: false, + input_end: false, + reads_to_end: false, + exits: true, + code: 0, + status: None, + reported: false, + unkillable: false, + held: [false, false], + hung_up: [false, false], + settlement: Some(Settlement::Settled), + settling: false, + requests: Vec::new(), + } + } + + /// Takes one step of the workload and of its adapter at `now`: applies the runner's + /// requests, writes the program's output into its pipes, feeds it input, reads the pipes as + /// the runner allows, reports what happened, and applies the requests that this makes. + fn step(&mut self, runner: &mut Runner, now: Instant) { + runner.tick(now); + self.apply_requests(runner); + if !self.launched && !self.launch_held { + self.launched = true; + if self.launches { + runner.launched(Ok(())); + } else { + runner.launched(Err(LaunchFailure::new(5, 2))); + self.status = Some(Status::Exited(125)); + self.lengths = [0, 0]; + } + } + if self.status.is_none() { + for index in 0..2 { + let room = PIPE - (self.written[index] - self.read[index]); + self.written[index] += room.min(self.lengths[index] - self.written[index]); + } + self.take_input(runner); + if self.exits + && self.written == self.lengths + && (!self.reads_to_end || self.input_closed) + { + self.status = Some(Status::Exited(self.code)); + self.input_closed = true; + } + } + for stream in Stream::ALL { + let index = stream.index(); + let held = self.written[index] - self.read[index]; + if self.discarding[index] { + self.read[index] += held; + self.discarded[index] += held; + runner.discarded(stream, held); + } else { + let length = held.min(runner.read_allowance(stream)); + let bytes = patterned(stream, self.read[index], length); + assert_eq!(runner.output(stream, &bytes), length); + self.read[index] += length; + } + if !self.closed[index] + && !self.held[index] + && self.status.is_some() + && self.written[index] == self.read[index] + { + self.closed[index] = true; + runner.output_closed(stream); + } + } + if self.status.is_some() && !self.reported { + self.reported = true; + runner.exited(now, self.status); + } + // Once the program ended, a stream that no process holds hung up, whatever its pipe + // still holds. + for stream in Stream::ALL { + let index = stream.index(); + if self.status.is_some() && !self.held[index] && !self.hung_up[index] { + self.hung_up[index] = true; + runner.output_hung_up(stream); + } + } + self.apply_requests(runner); + if self.settling + && let Some(settlement) = self.settlement + { + self.settling = false; + if settlement == Settlement::Settled { + self.held = [false, false]; + } + runner.settled(settlement); + } + } + + /// Applies the requests that the runner makes. + fn apply_requests(&mut self, runner: &mut Runner) { + while let Some(request) = runner.next_request() { + self.requests.push(request); + match request { + runner::Request::Terminate(_) => { + if !self.unkillable && self.status.is_none() { + self.status = Some(Status::Signaled(9)); + self.input_closed = true; + } + } + runner::Request::Discard(stream) => self.discarding[stream.index()] = true, + runner::Request::CloseInput => { + self.input_end = true; + self.input_closed = true; + } + runner::Request::Settle => self.settling = true, + } + } + } + + /// Feeds the program the input that the runner holds, at the program's rate, until the + /// program closes its input. + fn take_input(&mut self, runner: &mut Runner) { + let mut budget = self.input_rate; + while budget > 0 && !self.input_closed { + if self.input_limit == Some(self.input.len()) { + self.input_closed = true; + runner.input_closed(); + break; + } + let pending = runner.pending_input(); + let mut length = pending.len().min(budget); + if let Some(limit) = self.input_limit { + length = length.min(limit - self.input.len()); + } + if length == 0 { + break; + } + self.input.extend_from_slice(&pending[..length]); + runner.input_written(length); + budget -= length; + } + } +} + +/// The peer's caller, as the peer's adapter drives it: a source of input and a sink for each +/// output stream, which takes at most a rate of bytes in each step, and checks each byte +/// against the workload's pattern. +#[derive(Debug)] +struct Client { + /// The input that the source hands over in all. + input: Vec, + /// The bytes of input that the source handed over. + sent: usize, + /// Whether the source ends the input once it handed over everything. + ends_input: bool, + /// The most bytes that each sink takes in one step: zero stalls it. + rates: [usize; 2], + /// The bytes that each sink took. + received: [usize; 2], + /// The events, in order. + events: Vec, +} + +impl Client { + /// Returns a client that sends no input, ends it, and takes each stream at once. + fn new() -> Self { + Self { + input: Vec::new(), + sent: 0, + ends_input: true, + rates: [usize::MAX, usize::MAX], + received: [0, 0], + events: Vec::new(), + } + } + + /// Takes one step of the client and of the peer's adapter at `now`: hands the peer input as + /// it allows, takes output at the sinks' rates, and collects the peer's events. + fn step(&mut self, peer: &mut Peer, now: Instant) { + peer.tick(now); + if self.sent < self.input.len() { + self.sent += peer.input(&self.input[self.sent..]); + } + if self.ends_input && self.sent == self.input.len() { + peer.end_input(); + } + for stream in Stream::ALL { + let index = stream.index(); + let mut budget = self.rates[index]; + while budget > 0 { + let pending = peer.pending_output(stream); + let length = pending.len().min(budget); + if length == 0 { + break; + } + for (offset, &byte) in pending[..length].iter().enumerate() { + let position = self.received[index] + offset; + assert_eq!( + byte, + pattern(stream, position), + "{stream:?} byte {position}" + ); + } + self.received[index] += length; + peer.consumed(stream, length); + budget -= length; + } + } + while let Some(event) = peer.next_event() { + assert!( + !matches!(self.events.last(), Some(peer::Event::Finished(_))), + "the outcome is the last event" + ); + self.events.push(event); + } + } + + /// Returns the execution's outcome, once the peer finished. + fn outcome(&self) -> Option { + self.events.iter().find_map(|event| match event { + peer::Event::Finished(outcome) => Some(*outcome), + _ => None, + }) + } +} + +/// A runner and a peer, with their workload and client, connected by a queue of messages in each +/// direction, as their adapters and a transport would connect them. +struct Pair { + /// The current instant of the simulated clock. + now: Instant, + /// The runner. + runner: Runner, + /// The peer. + peer: Peer, + /// The runner's workload. + workload: Workload, + /// The peer's caller. + client: Client, + /// The messages in flight from the peer to the runner. + to_runner: VecDeque, + /// The messages in flight from the runner to the peer. + to_peer: VecDeque, + /// The most messages that each queue holds, after which its sender's adapter retrieves no + /// more. + capacity: usize, + /// Whether the transport delivers the runner's messages to the peer. + delivers_to_peer: bool, + /// Every message that the runner returned, in order, without its data. + sent_by_runner: Vec, + /// The terminal outcomes that the runner returned. + terminals: Vec, + /// The most data bytes in flight at once from the runner to the peer. + in_flight_high_water: usize, + /// The adapters' buffer. + buffer: Vec, +} + +impl Pair { + /// Returns a pair of sides with `limits` and `timers`, created at `now`, around `workload` + /// and `client`. + fn new(limits: Limits, timers: Timers, workload: Workload, client: Client) -> Self { + let now = Instant::now(); + Self { + now, + runner: Runner::new(KEY, limits, timers, now), + peer: Peer::new(KEY, limits, timers, now), + workload, + client, + to_runner: VecDeque::new(), + to_peer: VecDeque::new(), + capacity: usize::MAX, + delivers_to_peer: true, + sent_by_runner: Vec::new(), + terminals: Vec::new(), + in_flight_high_water: 0, + buffer: vec![0; limits.chunk()], + } + } + + /// Takes one step: the workload, the client, both adapters' retrieval of messages while the + /// queues have room, and the transport's delivery. + fn step(&mut self) { + self.workload.step(&mut self.runner, self.now); + self.client.step(&mut self.peer, self.now); + while self.to_runner.len() < self.capacity { + let Some(message) = self.peer.next_message(&mut self.buffer) else { + break; + }; + self.to_runner.push_back(to_runner(message, &self.buffer)); + } + while self.to_peer.len() < self.capacity { + let Some(message) = self.runner.next_message(&mut self.buffer) else { + break; + }; + self.sent_by_runner.push(message); + if let runner::Message::Terminal(outcome) = message { + self.terminals.push(outcome); + if self.runner.is_detached() { + continue; + } + } + self.to_peer.push_back(to_peer(message, &self.buffer)); + } + let in_flight = self + .to_peer + .iter() + .map(|message| match message { + ToPeer::Output(_, bytes) => bytes.len(), + _ => 0, + }) + .sum::(); + self.in_flight_high_water = self.in_flight_high_water.max(in_flight); + while let Some(message) = self.to_runner.pop_front() { + deliver_to_runner(&mut self.runner, self.now, message); + } + while self.delivers_to_peer + && let Some(message) = self.to_peer.pop_front() + { + if !self.peer.is_detached() { + deliver_to_peer(&mut self.peer, self.now, message); + } + } + } + + /// Takes steps until `done` holds, and fails after [`STEPS`] steps. + fn run_until(&mut self, mut done: impl FnMut(&Self) -> bool) { + for _ in 0..STEPS { + if done(self) { + return; + } + self.step(); + } + panic!("the pair did not get there in time"); + } + + /// Takes steps until both sides finished, and returns the peer's outcome. + fn finish(&mut self) -> Outcome { + self.run_until(|pair| pair.runner.is_finished() && pair.peer.is_finished()); + self.client.outcome().unwrap() + } + + /// Takes `count` steps. + fn steps(&mut self, count: usize) { + for _ in 0..count { + self.step(); + } + } + + /// Moves the clock forward by `duration`. + fn advance(&mut self, duration: Duration) { + self.now += duration; + } + + /// Returns the runner's terminal outcome, which it returned exactly once. + fn terminal(&self) -> Outcome { + assert_eq!(self.terminals.len(), 1, "one terminal outcome"); + self.terminals[0] + } +} diff --git a/nvx-rs/src/exec/tests/races.rs b/nvx-rs/src/exec/tests/races.rs new file mode 100644 index 000000000..4944cb18c --- /dev/null +++ b/nvx-rs/src/exec/tests/races.rs @@ -0,0 +1,352 @@ +//! Terminal races: the workload's exit against a cancellation, a timeout, a reset, and a +//! disconnection, in either order, on both sides, each with exactly one terminal outcome, and an +//! exit that arrives once the timeout passed, but before a tick fired it. + +use super::*; + +/// How a race ends the execution, besides the workload's exit. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Ending { + /// The peer's cancellation. + Cancel, + /// The runner's timeout. + Timeout, + /// A reset of the session. + Reset, + /// A lost transport. + Disconnect, +} + +impl Ending { + /// Every ending. + const ALL: [Self; 4] = [Self::Cancel, Self::Timeout, Self::Reset, Self::Disconnect]; + + /// Returns the cause that the ending gives. + fn cause(self) -> Cause { + match self { + Self::Cancel => Cause::Cancelled, + Self::Timeout => Cause::TimedOut, + Self::Reset => Cause::Reset, + Self::Disconnect => Cause::Disconnected, + } + } + + /// Applies the ending to `runner` at `now`, which its timeout has reached. + fn apply(self, runner: &mut Runner, now: Instant) { + match self { + Self::Cancel => { + runner.cancel(now, KEY, Cause::Cancelled).unwrap(); + } + Self::Timeout => runner.tick(now), + Self::Reset => runner.reset(now), + Self::Disconnect => runner.disconnect(now), + } + } +} + +/// The timeout of the runners in these races. +const TIMEOUT: Duration = Duration::from_secs(10); + +/// Drives `runner` from `now` until it finishes, settling it when it asks, granting output credit +/// if `credit`, and moving the clock forward by a grace when it waits, and returns its messages +/// and requests. +fn drive( + runner: &mut Runner, + mut now: Instant, + credit: bool, +) -> (Vec, Vec) { + let mut buffer = [0u8; 64]; + let mut messages = Vec::new(); + let mut requests = Vec::new(); + for _ in 0..16 { + runner.tick(now); + while let Some(request) = runner.next_request() { + requests.push(request); + if request == runner::Request::Settle { + runner.settled(Settlement::Settled); + } + } + if credit && !runner.is_detached() && !runner.is_finished() { + let _ = runner.credit(KEY, Pool::Stdout, 64); + } + while let Some(message) = runner.next_message(&mut buffer) { + messages.push(message); + } + if runner.is_finished() { + break; + } + now += GRACE; + } + assert!(runner.is_finished(), "the runner finished"); + (messages, requests) +} + +#[test] +fn an_exit_racing_each_ending_gives_one_terminal_outcome() { + for ending in Ending::ALL { + for exit_first in [true, false] { + let start = Instant::now(); + // Only the timeout's race gives the runner a timeout, which the exit stops when it + // comes first. + let timers = if ending == Ending::Timeout { + timers().timeout(TIMEOUT) + } else { + timers() + }; + let mut runner = Runner::new(KEY, bytes(64, 64, 64), timers, start); + runner.launched(Ok(())); + // Output that waits for credit, which the ending's grace may discard. + assert_eq!(runner.output(Stream::Stdout, b"abc"), 3); + runner.output_closed(Stream::Stdout); + runner.output_closed(Stream::Stderr); + let now = start + TIMEOUT; + let status = if exit_first { + runner.exited(now - Duration::from_millis(1), Some(Status::Exited(3))); + ending.apply(&mut runner, now); + Status::Exited(3) + } else { + ending.apply(&mut runner, now); + runner.exited(now, Some(Status::Signaled(9))); + Status::Signaled(9) + }; + // A timeout after the exit changes nothing, so the output waits for credit. + let lossless = exit_first && ending == Ending::Timeout; + let (messages, requests) = drive(&mut runner, now, lossless); + let terminals = messages + .iter() + .filter_map(|message| match message { + runner::Message::Terminal(outcome) => Some(*outcome), + _ => None, + }) + .collect::>(); + assert_eq!(terminals.len(), 1, "{ending:?} {exit_first}"); + let outcome = terminals[0]; + assert_eq!(outcome.status(), Some(status), "{ending:?} {exit_first}"); + let cause = (!exit_first).then(|| ending.cause()); + assert_eq!(outcome.cause(), cause, "{ending:?} {exit_first}"); + assert_eq!( + requests + .iter() + .filter(|request| matches!(request, runner::Request::Terminate(_))) + .count(), + usize::from(!exit_first), + "{ending:?} {exit_first}" + ); + assert_eq!(outcome.is_truncated(Stream::Stdout), !lossless); + assert!(!outcome.is_truncated(Stream::Stderr)); + assert!(outcome.is_reusable()); + // A detached runner sends nothing but keeps its terminal outcome. + let detached = matches!(ending, Ending::Reset | Ending::Disconnect); + assert_eq!(messages.len() == 1, detached, "{messages:?}"); + // Nothing follows the terminal outcome. + assert_eq!(runner.next_message(&mut [0; 64]), None); + assert_eq!(runner.next_request(), None); + assert_eq!(runner.deadline(), None); + assert_eq!(runner.cancel(now, KEY, Cause::Cancelled), Ok(Receipt::Late)); + } + } +} + +#[test] +fn an_exit_that_arrives_once_the_timeout_passed_but_before_a_tick_stops_the_timeout() { + // The adapter learns of the exit only at the timeout's instant, or a grace after it, and + // hands it over before it ticks the runner: the end stands as the workload's own, and the + // output that waits for credit loses nothing. + for late in [Duration::ZERO, GRACE] { + let start = Instant::now(); + let mut runner = Runner::new(KEY, bytes(64, 64, 64), timers().timeout(TIMEOUT), start); + runner.launched(Ok(())); + assert_eq!(runner.output(Stream::Stdout, b"abc"), 3); + runner.output_closed(Stream::Stdout); + runner.output_closed(Stream::Stderr); + let now = start + TIMEOUT + late; + runner.exited(now, Some(Status::Exited(0))); + let (messages, requests) = drive(&mut runner, now, true); + assert!( + !requests + .iter() + .any(|request| matches!(request, runner::Request::Terminate(_))), + "{late:?}: {requests:?}" + ); + let Some(runner::Message::Terminal(outcome)) = messages.last() else { + panic!("{messages:?}"); + }; + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None, "{late:?}"); + assert!(!outcome.is_truncated(Stream::Stdout), "{late:?}"); + assert!(outcome.is_reusable()); + } +} + +#[test] +fn the_first_cause_wins() { + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(64, 64, 64), timers().timeout(TIMEOUT), now); + runner.launched(Ok(())); + runner.tick(now + TIMEOUT); + runner.cancel(now + TIMEOUT, KEY, Cause::Cancelled).unwrap(); + runner.disconnect(now + TIMEOUT); + assert_eq!( + runner.next_request(), + Some(runner::Request::Terminate(Cause::TimedOut)) + ); + runner.exited(now + TIMEOUT, Some(Status::Signaled(9))); + runner.output_closed(Stream::Stdout); + runner.output_closed(Stream::Stderr); + let (messages, _) = drive(&mut runner, now + TIMEOUT, false); + let Some(runner::Message::Terminal(outcome)) = messages.last() else { + panic!("{messages:?}"); + }; + assert_eq!(outcome.cause(), Some(Cause::TimedOut)); +} + +#[test] +fn the_peer_finishes_once_whether_its_cancellation_or_the_terminal_outcome_comes_first() { + let terminal = |cause: Option| { + let outcome = Outcome::new(Some(Status::Signaled(9))).with_reusable(); + match cause { + Some(cause) => outcome.with_cause(cause), + None => outcome, + } + }; + for cancel_first in [true, false] { + for runner_cause in [None, Some(Cause::Cancelled), Some(Cause::TimedOut)] { + let now = Instant::now(); + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + while peer.next_message(&mut []).is_some() {} + peer.output(KEY, Stream::Stdout, b"held").unwrap(); + peer.end_output(KEY, Stream::Stdout).unwrap(); + peer.end_output(KEY, Stream::Stderr).unwrap(); + if cancel_first { + peer.cancel(now); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Cancel(Cause::Cancelled)) + ); + peer.terminal(now, KEY, terminal(runner_cause)).unwrap(); + } else { + peer.terminal(now, KEY, terminal(runner_cause)).unwrap(); + peer.cancel(now); + // The runner finished, so the peer sends no cancellation. + assert_eq!(peer.next_message(&mut []), None); + } + // The stalled sink holds the output for the grace, then the peer discards it. + assert!( + !std::iter::from_fn(|| peer.next_event()) + .any(|event| matches!(event, peer::Event::Finished(_))) + ); + peer.tick(now + GRACE); + let finished = std::iter::from_fn(|| peer.next_event()) + .filter_map(|event| match event { + peer::Event::Finished(outcome) => Some(outcome), + _ => None, + }) + .collect::>(); + assert_eq!(finished.len(), 1); + let outcome = finished[0]; + // The runner's cause stands; the peer's own fills in for none. + assert_eq!(outcome.cause(), runner_cause.or(Some(Cause::Cancelled))); + assert!(outcome.is_truncated(Stream::Stdout)); + assert!(!outcome.is_truncated(Stream::Stderr)); + assert!(outcome.is_reusable()); + assert_eq!(peer.next_event(), None); + } + } +} + +#[test] +fn a_peer_detached_before_the_terminal_outcome_finishes_on_its_own() { + for ending in [Ending::Reset, Ending::Disconnect] { + let now = Instant::now(); + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + while peer.next_message(&mut []).is_some() {} + peer.output(KEY, Stream::Stdout, b"held").unwrap(); + peer.end_output(KEY, Stream::Stderr).unwrap(); + match ending { + Ending::Reset => peer.reset(now), + _ => peer.disconnect(now), + } + // The sink still takes what arrived, for the grace. + assert_eq!(peer.pending_output(Stream::Stdout), b"held"); + peer.consumed(Stream::Stdout, 4); + let events = std::iter::from_fn(|| peer.next_event()).collect::>(); + let Some(peer::Event::Finished(outcome)) = events.last() else { + panic!("{events:?}"); + }; + // Standard output never ended, so it may have lost bytes; standard error ended whole. + assert_eq!(outcome.status(), None); + assert_eq!(outcome.cause(), Some(ending.cause())); + assert!(outcome.is_truncated(Stream::Stdout)); + assert!(!outcome.is_truncated(Stream::Stderr)); + assert!(!outcome.is_reusable()); + assert_eq!(peer.next_message(&mut [0; 8]), None); + // What arrives after the detachment comes too late. + assert_eq!( + peer.terminal(now, KEY, Outcome::new(None)), + Ok(Receipt::Late) + ); + } +} + +#[test] +fn the_runner_s_messages_after_the_peer_detached_are_late_while_its_sink_drains() { + for ending in [Ending::Reset, Ending::Disconnect] { + let now = Instant::now(); + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + while peer.next_message(&mut []).is_some() {} + peer.output(KEY, Stream::Stdout, b"held").unwrap(); + match ending { + Ending::Reset => peer.reset(now), + _ => peer.disconnect(now), + } + // The runner's last messages arrive while the sink still holds what came before, and + // before the caller retrieved the peer's outcome: the peer's own outcome stands. + assert_eq!(peer.output(KEY, Stream::Stdout, b"more"), Ok(Receipt::Late)); + for stream in Stream::ALL { + assert_eq!(peer.end_output(KEY, stream), Ok(Receipt::Late)); + } + let terminal = Outcome::new(Some(Status::Exited(0))).with_reusable(); + assert_eq!(peer.terminal(now, KEY, terminal), Ok(Receipt::Late)); + assert_eq!(peer.pending_output(Stream::Stdout), b"held"); + peer.consumed(Stream::Stdout, 4); + let events = std::iter::from_fn(|| peer.next_event()).collect::>(); + let Some(peer::Event::Finished(outcome)) = events.last() else { + panic!("{events:?}"); + }; + assert_eq!(outcome.status(), None, "{ending:?}"); + assert_eq!(outcome.cause(), Some(ending.cause())); + assert!(outcome.is_truncated(Stream::Stdout) && outcome.is_truncated(Stream::Stderr)); + assert!(!outcome.is_reusable()); + } +} + +#[test] +fn the_runner_s_messages_after_the_peer_gave_up_on_it_are_late() { + // The runner answers the cancellation only once the peer gave up waiting for it, but before + // the peer's caller retrieved its outcome: the peer's own outcome stands. + let now = Instant::now(); + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + while peer.next_message(&mut []).is_some() {} + peer.started(KEY).unwrap(); + peer.cancel(now); + assert_eq!( + peer.next_message(&mut []), + Some(peer::Message::Cancel(Cause::Cancelled)) + ); + let late = now + GRACE + SETTLEMENT; + peer.tick(late); + assert_eq!(peer.output(KEY, Stream::Stdout, b"late"), Ok(Receipt::Late)); + for stream in Stream::ALL { + assert_eq!(peer.end_output(KEY, stream), Ok(Receipt::Late)); + } + let terminal = Outcome::new(Some(Status::Exited(0))).with_reusable(); + assert_eq!(peer.terminal(late, KEY, terminal), Ok(Receipt::Late)); + let events = std::iter::from_fn(|| peer.next_event()).collect::>(); + let Some(peer::Event::Finished(outcome)) = events.last() else { + panic!("{events:?}"); + }; + assert_eq!(outcome.status(), None); + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert!(outcome.is_truncated(Stream::Stdout) && outcome.is_truncated(Stream::Stderr)); + assert!(!outcome.is_reusable()); +} diff --git a/nvx-rs/src/exec/tests/streams.rs b/nvx-rs/src/exec/tests/streams.rs new file mode 100644 index 000000000..f1bec5c70 --- /dev/null +++ b/nvx-rs/src/exec/tests/streams.rs @@ -0,0 +1,222 @@ +//! The order of each stream, the end of the input after every accepted byte, a workload that +//! closes its input early, and a launch that fails. + +use super::*; + +/// Returns the input bytes `0, 1, 2, ...` of `length` bytes. +fn counting(length: usize) -> Vec { + (0..length).map(|index| index as u8).collect() +} + +#[test] +fn each_stream_keeps_its_order_and_its_end_follows_its_last_byte() { + let mut client = Client::new(); + client.rates = [700, 300]; + let mut pair = Pair::new( + bytes(1000, 4096, 3000), + timers(), + Workload::new(10_000, 7_000), + client, + ); + let outcome = pair.finish(); + // The client checked every byte against the workload's pattern as it took it. + assert_eq!(pair.client.received, [10_000, 7_000]); + assert_eq!(outcome, pair.terminal()); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(outcome.is_reusable()); + assert!(!outcome.is_truncated(Stream::Stdout) && !outcome.is_truncated(Stream::Stderr)); + + let sent = &pair.sent_by_runner; + let position = |wanted: runner::Message| sent.iter().position(|message| *message == wanted); + let started = position(runner::Message::Started).unwrap(); + for stream in Stream::ALL { + let end = position(runner::Message::End(stream)).unwrap(); + let outputs = sent + .iter() + .enumerate() + .filter(|(_, message)| { + matches!(message, runner::Message::Output { stream: of, .. } if *of == stream) + }) + .map(|(index, _)| index) + .collect::>(); + assert!(outputs.iter().all(|&index| started < index && index < end)); + } + assert!(matches!(sent.last(), Some(runner::Message::Terminal(_)))); + assert_eq!( + sent.iter() + .filter(|message| matches!(message, runner::Message::Terminal(_))) + .count(), + 1 + ); + // The peer tells its caller in the same order. + let events = &pair.client.events; + assert_eq!(events.first(), Some(&peer::Event::Started)); + assert!(matches!(events.last(), Some(peer::Event::Finished(_)))); + for stream in Stream::ALL { + assert_eq!( + events + .iter() + .filter(|event| **event == peer::Event::End(stream)) + .count(), + 1 + ); + } +} + +#[test] +fn the_input_ends_only_after_every_accepted_byte() { + let mut workload = Workload::new(0, 0); + workload.reads_to_end = true; + workload.input_rate = 3; + let mut client = Client::new(); + client.input = counting(50); + let mut pair = Pair::new(bytes(8, 16, 64), timers(), workload, client); + let outcome = pair.finish(); + // The workload took every byte before its input closed, which ended the program. + assert_eq!(pair.workload.input, counting(50)); + assert_eq!( + pair.workload + .requests + .iter() + .filter(|request| **request == runner::Request::CloseInput) + .count(), + 1 + ); + assert!(pair.workload.input_end); + // The peer ended the input, so the runner never said that it closed early. + assert!(!pair.sent_by_runner.contains(&runner::Message::InputClosed)); + assert!(!pair.client.events.contains(&peer::Event::InputClosed)); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable()); +} + +#[test] +fn a_workload_that_closes_its_input_early_never_leaves_the_source_waiting() { + let mut workload = Workload::new(2_000, 0); + workload.reads_to_end = true; + workload.input_limit = Some(5); + let mut client = Client::new(); + client.input = counting(100_000); + let mut pair = Pair::new(bytes(512, 1024, 1024), timers(), workload, client); + let outcome = pair.finish(); + assert_eq!(pair.workload.input, counting(5)); + assert!(pair.sent_by_runner.contains(&runner::Message::InputClosed)); + assert!(pair.client.events.contains(&peer::Event::InputClosed)); + // The source stopped once the input closed, short of its 100,000 bytes, and the output + // still arrived whole. + assert!(pair.client.sent < 100_000); + assert_eq!(pair.peer.write_allowance(), 0); + assert_eq!(pair.client.received, [2_000, 0]); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(outcome.is_reusable()); +} + +#[test] +fn input_after_the_workload_s_input_closed_is_dropped_within_the_window() { + let mut runner = Runner::new(KEY, bytes(4, 8, 8), timers(), Instant::now()); + runner.launched(Ok(())); + while runner.next_message(&mut []).is_some() {} + assert_eq!(runner.input(KEY, b"abcd"), Ok(Receipt::Applied)); + runner.input_closed(); + assert_eq!(runner.pending_input(), b""); + assert_eq!( + runner.next_message(&mut []), + Some(runner::Message::InputClosed) + ); + // The peer may still send what its credit allowed before it learned, and no more. + assert_eq!(runner.input(KEY, b"efgh"), Ok(Receipt::Late)); + assert_eq!(runner.input(KEY, b"i"), Err(Violation::Window)); + assert_eq!(runner.end_input(KEY), Ok(Receipt::Late)); + assert_eq!(runner.end_input(KEY), Err(Violation::Ended)); + assert_eq!(runner.next_request(), None); +} + +#[test] +fn closing_the_input_keeps_the_output_flowing() { + let mut workload = Workload::new(50_000, 3_000); + workload.reads_to_end = true; + let mut client = Client::new(); + client.input = counting(10); + client.rates = [1_000, 1_000]; + let mut pair = Pair::new(bytes(2048, 4096, 4096), timers(), workload, client); + let outcome = pair.finish(); + assert_eq!(pair.workload.input, counting(10)); + assert_eq!(pair.client.received, [50_000, 3_000]); + assert!(!outcome.is_truncated(Stream::Stdout)); +} + +#[test] +fn a_failed_launch_sends_no_start_and_reports_why() { + let mut workload = Workload::new(0, 0); + workload.launches = false; + let mut pair = Pair::new(bytes(64, 64, 64), timers(), workload, Client::new()); + let outcome = pair.finish(); + assert!(!pair.sent_by_runner.contains(&runner::Message::Started)); + assert!(!pair.client.events.contains(&peer::Event::Started)); + assert!(!pair.peer.has_started()); + assert_eq!(outcome.launch_failure(), Some(LaunchFailure::new(5, 2))); + assert_eq!(outcome.status(), Some(Status::Exited(125))); + assert!(outcome.is_reusable()); +} + +#[test] +fn a_second_start_end_or_terminal_is_out_of_order() { + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), Instant::now()); + let now = Instant::now(); + assert_eq!(peer.started(KEY), Ok(Receipt::Applied)); + assert_eq!(peer.started(KEY), Err(Violation::Order)); + let terminal = Outcome::new(Some(Status::Exited(0))).with_reusable(); + // The terminal outcome follows both streams' ends. + assert_eq!(peer.terminal(now, KEY, terminal), Err(Violation::Order)); + assert_eq!(peer.end_output(KEY, Stream::Stdout), Ok(Receipt::Applied)); + assert_eq!(peer.end_output(KEY, Stream::Stdout), Err(Violation::Ended)); + while peer.next_message(&mut []).is_some() {} + assert_eq!( + peer.output(KEY, Stream::Stdout, b"x"), + Err(Violation::Ended) + ); + assert_eq!(peer.terminal(now, KEY, terminal), Err(Violation::Order)); + assert_eq!(peer.end_output(KEY, Stream::Stderr), Ok(Receipt::Applied)); + assert_eq!(peer.terminal(now, KEY, terminal), Ok(Receipt::Applied)); + assert_eq!(peer.terminal(now, KEY, terminal), Err(Violation::Ended)); + let events = std::iter::from_fn(|| peer.next_event()).collect::>(); + assert_eq!( + events, + [ + peer::Event::Started, + peer::Event::InputClosed, + peer::Event::End(Stream::Stdout), + peer::Event::End(Stream::Stderr), + peer::Event::Finished(terminal), + ] + ); +} + +#[test] +fn a_start_after_the_terminal_outcome_of_a_failed_launch_is_out_of_order() { + // The runner reported that the launch failed, so no start follows, even before the peer's + // caller retrieved the outcome. + let now = Instant::now(); + let mut peer = Peer::new(KEY, bytes(8, 8, 8), timers(), now); + for stream in Stream::ALL { + assert_eq!(peer.end_output(KEY, stream), Ok(Receipt::Applied)); + } + let terminal = Outcome::new(Some(Status::Exited(125))) + .with_launch_failure(LaunchFailure::new(5, 2)) + .with_reusable(); + assert_eq!(peer.terminal(now, KEY, terminal), Ok(Receipt::Applied)); + assert_eq!(peer.started(KEY), Err(Violation::Order)); + assert!(!peer.has_started()); + let events = std::iter::from_fn(|| peer.next_event()).collect::>(); + assert_eq!( + events, + [ + peer::Event::InputClosed, + peer::Event::End(Stream::Stdout), + peer::Event::End(Stream::Stderr), + peer::Event::Finished(terminal), + ] + ); +} diff --git a/nvx-rs/src/exec/tests/truncation.rs b/nvx-rs/src/exec/tests/truncation.rs new file mode 100644 index 000000000..0e740c0f8 --- /dev/null +++ b/nvx-rs/src/exec/tests/truncation.rs @@ -0,0 +1,525 @@ +//! The abnormal paths and the normal one: a termination's grace and the truncation that follows +//! it, lossless normal completion, failed reads, failed and late settlements, the streams that an +//! execution that does not settle leaves open, and those that hung up, launch results that a +//! process holds back after the workload's end, and sides that give up and fence the execution. + +use super::*; + +/// Returns whether `outcome` marks either stream as truncated. +fn truncated(outcome: Outcome) -> bool { + Stream::ALL + .into_iter() + .any(|stream| outcome.is_truncated(stream)) +} + +#[test] +fn a_workload_that_ends_on_its_own_loses_no_byte_however_slow_the_sink() { + let mut client = Client::new(); + client.rates = [100, 37]; + let mut pair = Pair::new( + bytes(1024, 1024, 4096), + timers(), + Workload::new(100_000, 50_000), + client, + ); + // Each step takes longer than the grace and the settlement time together. + for _ in 0..STEPS { + if pair.peer.is_finished() { + break; + } + pair.advance(GRACE + SETTLEMENT); + pair.step(); + // The runner's terminal outcome waits for the output that the sink has yet to take, + // beyond what its windows hold. + if pair.runner.is_finished() { + assert!(pair.client.received[0] + 4096 >= 100_000); + assert!(pair.client.received[1] + 4096 >= 50_000); + } + } + let outcome = pair.client.outcome().unwrap(); + assert_eq!(pair.client.received, [100_000, 50_000]); + assert_eq!(outcome, pair.terminal()); + assert_eq!(outcome.cause(), None); + assert!(!truncated(outcome)); + assert!(outcome.is_reusable()); +} + +#[test] +fn a_termination_drains_pending_output_for_its_grace_then_marks_the_loss() { + let mut client = Client::new(); + client.rates = [0, 0]; + let mut pair = Pair::new( + bytes(4096, 4096, 16 * 1024), + timers(), + Workload::new(1 << 20, 100), + client, + ); + // The stalled sink exhausts both windows, and the workload blocks on its full pipe. + pair.steps(20); + assert_eq!(pair.runner.read_allowance(Stream::Stdout), 0); + assert!(pair.workload.status.is_none()); + let cancelled = pair.now; + pair.peer.cancel(cancelled); + pair.steps(20); + assert!( + pair.workload.status.is_some(), + "the cancellation ended the workload" + ); + assert!(!pair.runner.is_finished() && !pair.peer.is_finished()); + pair.advance(GRACE); + pair.steps(5); + assert!(pair.runner.is_finished() && pair.peer.is_finished()); + let terminal = pair.terminal(); + assert_eq!(terminal.cause(), Some(Cause::Cancelled)); + assert_eq!(terminal.status(), Some(Status::Signaled(9))); + assert!(terminal.is_truncated(Stream::Stdout)); + // Standard error's 100 bytes were in flight within its window, and the peer discarded them. + assert!(!terminal.is_truncated(Stream::Stderr)); + let outcome = pair.client.outcome().unwrap(); + assert!(outcome.is_truncated(Stream::Stdout) && outcome.is_truncated(Stream::Stderr)); + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert!(outcome.is_reusable()); + assert!( + pair.workload + .requests + .contains(&runner::Request::Discard(Stream::Stdout)) + ); + assert!(pair.workload.discarded[0] > 0); +} + +#[test] +fn a_termination_whose_pending_output_drains_within_the_grace_loses_nothing() { + let mut client = Client::new(); + client.rates = [0, 0]; + let mut pair = Pair::new( + bytes(4096, 4096, 16 * 1024), + timers(), + Workload::new(1 << 20, 0), + client, + ); + pair.steps(20); + pair.peer.cancel(pair.now); + // The sink takes everything again within the grace: what the workload wrote before its end + // all arrives, though the cancellation cut the workload short. + pair.client.rates = [usize::MAX, usize::MAX]; + pair.finish(); + let outcome = pair.client.outcome().unwrap(); + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert!(!truncated(outcome), "{outcome:?}"); + assert_eq!(pair.client.received[0], pair.workload.written[0]); + assert!(pair.workload.written[0] < 1 << 20); +} + +#[test] +fn a_failed_read_truncates_its_stream_without_a_cause() { + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(8, 8, 8), timers(), now); + runner.launched(Ok(())); + runner.output_failed(Stream::Stdout); + runner.output_closed(Stream::Stderr); + runner.exited(now, Some(Status::Exited(0))); + runner.settled(Settlement::Settled); + let messages = std::iter::from_fn(|| runner.next_message(&mut [0; 8])).collect::>(); + let Some(runner::Message::Terminal(outcome)) = messages.last() else { + panic!("{messages:?}"); + }; + assert_eq!(outcome.cause(), None); + assert!(outcome.is_truncated(Stream::Stdout)); + assert!(!outcome.is_truncated(Stream::Stderr)); + assert_eq!(runner.read_allowance(Stream::Stdout), 0); +} + +#[test] +fn a_failed_or_late_settlement_fences_the_execution() { + for settlement in [Some(Settlement::Failed), None] { + let mut workload = Workload::new(10, 0); + workload.settlement = settlement; + let mut pair = Pair::new(bytes(64, 64, 64), timers(), workload, Client::new()); + pair.run_until(|pair| pair.workload.requests.contains(&runner::Request::Settle)); + pair.steps(3); + if settlement.is_none() { + // The runner waits for the settlement for its settlement time. + assert!(!pair.runner.is_finished()); + assert!(pair.runner.awaits_settlement()); + assert!(pair.runner.deadline().is_some()); + pair.advance(SETTLEMENT); + } + let outcome = pair.finish(); + assert!(!pair.runner.awaits_settlement(), "{settlement:?}"); + assert!(!outcome.is_reusable(), "{settlement:?}"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(!truncated(outcome)); + // A result after the settlement time changes nothing. + pair.runner.settled(Settlement::Settled); + assert!(!pair.terminal().is_reusable()); + } +} + +#[test] +fn an_execution_that_does_not_settle_gives_up_on_the_streams_still_open_after_its_end() { + // A process that the workload left behind holds standard output open, and the settlement + // fails, or never comes. The runner leaves the stream the settlement time and then the + // grace after the workload's end to close, then gives up on it: it sends what it read, and + // marks only that stream truncated. + for settlement in [Some(Settlement::Failed), None] { + let mut workload = Workload::new(10_000, 100); + workload.held = [true, false]; + workload.settlement = settlement; + let mut pair = Pair::new(bytes(1024, 1024, 4096), timers(), workload, Client::new()); + pair.run_until(|pair| pair.workload.reported); + let exited = pair.now; + pair.run_until(|pair| { + pair.client.received == [10_000, 100] + && pair + .client + .events + .contains(&peer::Event::End(Stream::Stderr)) + }); + pair.steps(5); + assert!(!pair.runner.is_finished(), "{settlement:?}"); + assert!( + !pair + .client + .events + .contains(&peer::Event::End(Stream::Stdout)) + ); + if settlement.is_none() { + assert!(pair.runner.awaits_settlement()); + assert_eq!(pair.runner.deadline(), Some(exited + SETTLEMENT)); + pair.advance(SETTLEMENT); + pair.steps(5); + assert!(!pair.runner.awaits_settlement()); + assert!(!pair.runner.is_finished()); + } + assert_eq!( + pair.runner.deadline(), + Some(exited + SETTLEMENT + GRACE), + "{settlement:?}" + ); + pair.now = exited + SETTLEMENT + GRACE - Duration::from_millis(1); + pair.steps(5); + assert!(!pair.runner.is_finished(), "{settlement:?}"); + pair.now = exited + SETTLEMENT + GRACE; + let outcome = pair.finish(); + assert_eq!(outcome, pair.terminal()); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(outcome.is_truncated(Stream::Stdout)); + assert!(!outcome.is_truncated(Stream::Stderr)); + assert!(!outcome.is_reusable()); + assert_eq!(pair.client.received, [10_000, 100]); + assert_eq!(pair.runner.read_allowance(Stream::Stdout), 0); + assert!( + pair.sent_by_runner + .contains(&runner::Message::End(Stream::Stdout)) + ); + } +} + +#[test] +fn an_execution_that_does_not_settle_loses_nothing_of_the_streams_that_hung_up() { + // No process holds the streams once the workload ended, but the settlement fails, or never + // comes, and the sink takes nothing for long after the settlement time and the grace. The + // streams hung up, so the runner never gives up on them: it waits for the sink, and loses + // nothing, though the execution is fenced. + for settlement in [Some(Settlement::Failed), None] { + let mut workload = Workload::new(10_000, 5_000); + workload.settlement = settlement; + let mut client = Client::new(); + client.rates = [0, 0]; + let mut pair = Pair::new(bytes(1024, 1024, 4096), timers(), workload, client); + pair.run_until(|pair| pair.workload.reported); + pair.steps(5); + pair.advance((SETTLEMENT + GRACE) * 10); + pair.steps(5); + assert!(!pair.runner.is_finished(), "{settlement:?}"); + assert_eq!(pair.runner.deadline(), None, "{settlement:?}"); + pair.client.rates = [usize::MAX, usize::MAX]; + let outcome = pair.finish(); + assert_eq!(pair.client.received, [10_000, 5_000], "{settlement:?}"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(!truncated(outcome), "{settlement:?}"); + assert!(!outcome.is_reusable(), "{settlement:?}"); + } +} + +#[test] +fn a_settled_execution_waits_for_its_sink_however_long_its_streams_stay_open() { + // The settlement ends the process that held standard output, and the stalled sink takes the + // rest long after the settlement time and the grace: nothing is lost. + let mut workload = Workload::new(10_000, 0); + workload.held = [true, false]; + let mut client = Client::new(); + client.rates = [0, 0]; + let mut pair = Pair::new(bytes(1024, 1024, 4096), timers(), workload, client); + pair.run_until(|pair| pair.workload.reported); + pair.steps(5); + assert!(!pair.runner.awaits_settlement()); + assert_eq!(pair.runner.deadline(), None); + pair.advance((SETTLEMENT + GRACE) * 10); + pair.steps(5); + assert!(!pair.runner.is_finished()); + pair.client.rates = [usize::MAX, usize::MAX]; + let outcome = pair.finish(); + assert_eq!(pair.client.received, [10_000, 0]); + assert!(!truncated(outcome)); + assert!(outcome.is_reusable()); +} + +#[test] +fn the_runner_gives_up_on_a_launch_result_that_a_process_holds_back_after_the_workload_s_end() { + // A process holds back what reports the workload's launch, whether the execution settles, + // its settlement fails, or never comes, and only the runner has a timeout, which the + // workload's end disarmed before it passed. The runner leaves the launch result the + // settlement time and then the grace after the workload's end to arrive, holding the start + // and the output meanwhile, then gives up on it: it sends the output, which the streams that + // hung up keep whole, and the outcome, without a start, with the workload's status and no + // cause, and fences the execution. The result arrives once the runner sent output and an + // end, but before the terminal outcome, and counts for nothing, so no start follows them. + const TIMEOUT: Duration = Duration::from_secs(2); + let limits = bytes(64, 64, 64); + for settlement in [Some(Settlement::Settled), Some(Settlement::Failed), None] { + let mut workload = Workload::new(100, 10); + workload.launch_held = true; + workload.settlement = settlement; + let mut pair = Pair::new(limits, timers(), workload, Client::new()); + pair.runner = Runner::new(KEY, limits, timers().timeout(TIMEOUT), pair.now); + pair.run_until(|pair| pair.workload.reported); + let exited = pair.now; + pair.steps(5); + let held = |pair: &Pair| { + !pair.runner.is_finished() + && !pair.sent_by_runner.iter().any(|message| { + matches!( + message, + runner::Message::Started + | runner::Message::Output { .. } + | runner::Message::End(_) + ) + }) + }; + assert!(held(&pair), "{settlement:?}"); + if settlement.is_none() { + assert_eq!(pair.runner.deadline(), Some(exited + SETTLEMENT)); + pair.advance(SETTLEMENT); + pair.steps(5); + assert!(!pair.runner.awaits_settlement()); + assert!(held(&pair)); + } + assert_eq!( + pair.runner.deadline(), + Some(exited + SETTLEMENT + GRACE), + "{settlement:?}" + ); + pair.now = exited + SETTLEMENT + GRACE - Duration::from_millis(1); + pair.steps(5); + assert!(held(&pair), "{settlement:?}"); + pair.now = exited + SETTLEMENT + GRACE; + pair.run_until(|pair| !held(pair)); + assert!(!pair.runner.is_finished(), "{settlement:?}"); + pair.workload.launch_held = false; + let outcome = pair.finish(); + assert!(pair.workload.launched); + assert_eq!(outcome, pair.terminal()); + assert_eq!(outcome.status(), Some(Status::Exited(0)), "{settlement:?}"); + assert_eq!(outcome.cause(), None, "{settlement:?}"); + assert_eq!(outcome.launch_failure(), None); + assert!(!truncated(outcome), "{settlement:?}"); + assert!(!outcome.is_reusable(), "{settlement:?}"); + assert_eq!(pair.client.received, [100, 10]); + assert!(!pair.sent_by_runner.contains(&runner::Message::Started)); + assert!(!pair.peer.has_started()); + // A launch result after the terminal outcome changes nothing. + pair.runner.launched(Ok(())); + assert_eq!(pair.runner.next_message(&mut []), None); + } +} + +#[test] +fn a_launch_result_after_a_termination_s_give_up_counts_for_nothing() { + // A cancellation ends a workload whose launch result never arrives, and which never ends, so + // the runner gives up on it once the grace and the settlement time have passed. The result + // arrives once the runner sent a stream's end, but before the terminal outcome, and counts + // for nothing: no start follows the end. + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(8, 8, 8), timers(), now); + assert_eq!( + runner.cancel(now, KEY, Cause::Cancelled), + Ok(Receipt::Applied) + ); + runner.tick(now + GRACE + SETTLEMENT); + let mut sent = Vec::new(); + while let Some(message) = runner.next_message(&mut [0; 8]) { + sent.push(message); + if matches!(message, runner::Message::End(_)) { + break; + } + } + assert!( + matches!(sent.last(), Some(runner::Message::End(_))), + "{sent:?}" + ); + runner.launched(Ok(())); + sent.extend(std::iter::from_fn(|| runner.next_message(&mut [0; 8]))); + assert!(!sent.contains(&runner::Message::Started), "{sent:?}"); + let Some(runner::Message::Terminal(outcome)) = sent.last() else { + panic!("{sent:?}"); + }; + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert_eq!(outcome.launch_failure(), None); + assert!(!outcome.is_reusable()); +} + +#[test] +fn a_launch_result_that_arrives_after_the_workload_s_end_but_in_time_still_counts() { + // A process holds back what reports the workload's launch until just before the runner would + // give up on it: the runner sends the start first, the outcome is lossless, and the execution + // is reusable. + let mut workload = Workload::new(100, 10); + workload.launch_held = true; + let mut pair = Pair::new(bytes(64, 64, 64), timers(), workload, Client::new()); + pair.run_until(|pair| pair.workload.reported); + pair.now += SETTLEMENT + GRACE - Duration::from_millis(1); + pair.steps(5); + assert!(!pair.runner.is_finished()); + assert!(!pair.sent_by_runner.contains(&runner::Message::Started)); + pair.workload.launch_held = false; + let outcome = pair.finish(); + assert_eq!(outcome, pair.terminal()); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(!truncated(outcome)); + assert!(outcome.is_reusable()); + assert_eq!(pair.client.received, [100, 10]); + let started = pair + .sent_by_runner + .iter() + .position(|message| *message == runner::Message::Started) + .expect("the runner sent the start"); + assert!( + pair.sent_by_runner[..started] + .iter() + .all(|message| matches!( + message, + runner::Message::Credit { .. } | runner::Message::InputClosed + )), + "{:?}", + pair.sent_by_runner + ); +} + +#[test] +fn a_settlement_that_the_runner_did_not_request_counts_for_nothing() { + let now = Instant::now(); + let mut runner = Runner::new(KEY, bytes(8, 8, 8), timers(), now); + runner.launched(Ok(())); + runner.output_closed(Stream::Stdout); + runner.output_closed(Stream::Stderr); + // The workload still runs, so nothing settles it yet. + runner.settled(Settlement::Settled); + assert!(!runner.awaits_settlement()); + runner.exited(now, Some(Status::Exited(0))); + assert!(runner.awaits_settlement()); + assert_eq!(runner.next_request(), Some(runner::Request::Settle)); + while let Some(message) = runner.next_message(&mut []) { + assert!( + !matches!(message, runner::Message::Terminal(_)), + "{message:?}" + ); + } + runner.settled(Settlement::Failed); + assert!(!runner.awaits_settlement()); + let Some(runner::Message::Terminal(outcome)) = runner.next_message(&mut []) else { + panic!("the terminal outcome follows the settlement"); + }; + assert!(!outcome.is_reusable()); +} + +#[test] +fn the_runner_gives_up_on_a_workload_that_will_not_end_and_fences_it() { + let mut workload = Workload::new(1 << 20, 0); + workload.unkillable = true; + let mut client = Client::new(); + client.rates = [0, 0]; + let mut pair = Pair::new(bytes(4096, 4096, 4096), timers(), workload, client); + pair.steps(10); + let cancelled = pair.now; + pair.peer.cancel(cancelled); + pair.steps(10); + pair.advance(GRACE); + pair.steps(10); + // The workload still runs and holds its stream open, so the runner waits. + assert!(!pair.runner.is_finished()); + assert_eq!(pair.runner.deadline(), Some(cancelled + GRACE + SETTLEMENT)); + pair.advance(SETTLEMENT); + pair.steps(5); + assert!(pair.runner.is_finished() && pair.peer.is_finished()); + let terminal = pair.terminal(); + assert_eq!(terminal.status(), None); + assert_eq!(terminal.cause(), Some(Cause::Cancelled)); + assert!(terminal.is_truncated(Stream::Stdout)); + assert!(!terminal.is_reusable()); + // The runner still ended both streams before its terminal outcome. + for stream in Stream::ALL { + assert!(pair.sent_by_runner.contains(&runner::Message::End(stream))); + } + assert_eq!(pair.client.outcome(), Some(terminal)); + assert_eq!(pair.runner.read_allowance(Stream::Stdout), 0); +} + +#[test] +fn a_peer_that_ended_the_execution_gives_up_on_a_silent_runner() { + let mut pair = Pair::new( + bytes(64, 64, 64), + timers(), + Workload::new(1 << 20, 0), + Client::new(), + ); + pair.steps(5); + // The transport stops delivering the runner's messages. + pair.delivers_to_peer = false; + let cancelled = pair.now; + pair.peer.cancel(cancelled); + pair.steps(5); + pair.advance(GRACE); + pair.steps(5); + assert!(!pair.peer.is_finished()); + assert_eq!(pair.peer.deadline(), Some(cancelled + GRACE + SETTLEMENT)); + pair.advance(SETTLEMENT); + pair.steps(1); + let outcome = pair.client.outcome().expect("the peer gave up"); + assert_eq!(outcome.status(), None); + assert_eq!(outcome.cause(), Some(Cause::Cancelled)); + assert!(outcome.is_truncated(Stream::Stdout) && outcome.is_truncated(Stream::Stderr)); + assert!(!outcome.is_reusable()); +} + +#[test] +fn the_peer_s_own_timeout_cancels_and_names_itself() { + let mut client = Client::new(); + client.rates = [0, 0]; + let mut pair = Pair::new( + bytes(64, 64, 64), + timers().timeout(Duration::from_secs(30)), + Workload::new(1 << 20, 0), + client, + ); + // Only the peer's timer runs out: the runner's workload got the same timeout, but the + // runner reports the peer's cause, which arrives first. + pair.runner = Runner::new(KEY, bytes(64, 64, 64), timers(), pair.now); + pair.steps(5); + pair.advance(Duration::from_secs(30)); + pair.steps(5); + let terminations = pair + .workload + .requests + .iter() + .filter(|request| matches!(request, runner::Request::Terminate(_))) + .collect::>(); + assert_eq!(terminations, [&runner::Request::Terminate(Cause::TimedOut)]); + pair.advance(GRACE); + let outcome = pair.finish(); + assert_eq!(outcome.cause(), Some(Cause::TimedOut)); + assert_eq!(pair.terminal().cause(), Some(Cause::TimedOut)); +} diff --git a/nvx-rs/src/guest/exec.rs b/nvx-rs/src/guest/exec.rs index e0a10e593..62b6e634e 100644 --- a/nvx-rs/src/guest/exec.rs +++ b/nvx-rs/src/guest/exec.rs @@ -10,7 +10,10 @@ //! It needs neither NVX's control protocol nor an async runtime, and it keeps no policy: the //! caller decides which programs run and how many at once, where their output goes, how much of //! it to take, and what each outcome means. NVX's managed agent is one caller, which runs one -//! execution at a time and forwards its output and outcome as NVXC frames. +//! execution at a time and forwards its output and outcome as NVXC frames. A caller whose peer +//! runs the [execution engine](crate::exec)'s other side drives the execution through a +//! [`Binding`](binding::Binding) instead, which turns the engine's credit, input, endings, and +//! settlement into the execution's calls, and its events into the engine's inputs. //! //! # Driving //! @@ -100,6 +103,8 @@ use super::process::Child; use super::spawn::{LaunchFailure, LaunchReport, Spawned}; use super::sys::{self, Errno, FileDescriptor, ProcessId}; +pub mod binding; + /// One of a program's output streams. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub enum Stream { @@ -567,6 +572,15 @@ impl Execution { self.outputs.iter().any(Output::is_open) } + /// Returns whether every writer of `stream` closed it, without reading it: the stream ends + /// once the execution reads what its pipe still holds. A stream that closed hung up too. + pub(crate) fn output_hung_up(&self, stream: Stream) -> bool { + match &self.outputs[stream.index()].reader { + Some(reader) => sys::hung_up(reader.borrowed()).unwrap_or(false), + None => true, + } + } + /// Lets the execution read `bytes` more of `stream`: its flow-control seam. It reads a /// stream only while the stream has an allowance, and each output event uses up as much of /// it. An allowance that reaches `usize::MAX` never runs out. @@ -739,26 +753,13 @@ impl Execution { if let Some(event) = self.kill_failure() { return Ok(Some(event)); } - if self.report.as_ref().is_some_and(arrived) { - let report = self.report.take().expect("the report is pending"); - return Ok(Some(Event::Launch(report.wait()))); - } - if self.collects() { - match self.child.try_wait() { - Ok(Some(_)) => { - self.exited(); - if let Some(event) = self.kill_failure() { - return Ok(Some(event)); - } - } - Ok(None) => {} - Err(error) => { - if error.raw_os_error() == Some(Errno::ECHILD as i32) { - self.taken(); - } - return Err(error); - } - } + if let Some(report) = self.launch() { + return Ok(Some(Event::Launch(report))); + } + if self.collect()? + && let Some(event) = self.kill_failure() + { + return Ok(Some(event)); } if self .deadline() @@ -804,6 +805,46 @@ impl Execution { }) } + /// Takes the program's launch report without blocking once it has arrived, as + /// [`next_event`](Self::next_event) does within a step, and returns it, or `None` while it is + /// pending or once it was taken. + pub(crate) fn launch(&mut self) -> Option>> { + if !self.report.as_ref().is_some_and(arrived) { + return None; + } + let report = self.report.take().expect("the report is pending"); + Some(report.wait()) + } + + /// Collects the program's status without blocking, if the execution collects it itself and + /// the program has ended, as [`next_event`](Self::next_event) does within a step, and returns + /// whether it collected it now. It fails as `next_event` does if collecting the status fails. + pub(crate) fn collect(&mut self) -> io::Result { + if !self.collects() { + return Ok(false); + } + match self.child.try_wait() { + Ok(Some(_)) => { + self.exited(); + Ok(true) + } + Ok(None) => Ok(false), + Err(error) => { + if error.raw_os_error() == Some(Errno::ECHILD as i32) { + self.taken(); + } + Err(error) + } + } + } + + /// Gives the options' cgroup back to the caller, whose check found it empty after the + /// program ended: the execution kills its processes no more, not even when it drops, so + /// that the caller may reuse the cgroup at once. + pub(crate) fn release_cgroup(&mut self) { + self.cgroup = None; + } + /// Ends the program for `cause`, unless the execution has ended, and returns whether `cause` /// is why it ended the program: the first cause while the program is unreaped. fn end(&mut self, cause: Cause) -> bool { diff --git a/nvx-rs/src/guest/exec/binding.rs b/nvx-rs/src/guest/exec/binding.rs new file mode 100644 index 000000000..911af17ec --- /dev/null +++ b/nvx-rs/src/guest/exec/binding.rs @@ -0,0 +1,582 @@ +//! The runner of the [execution engine](crate::exec), bound to an [`Execution`] of the guest's +//! driver: the glue that every guest adapter would otherwise write. +//! +//! A [`Binding`] applies the [`Runner`]'s requests to the execution: its read allowances become +//! [grants](Execution::grant), its discards [discards](Execution::discard), the input that it +//! accepted [writes](Execution::write_input) and, once the peer ended it, +//! [a close](Execution::close_input), its terminations a [cancellation](Execution::cancel), a +//! [disconnection](Execution::disconnect), or a [termination](Execution::terminate), and its +//! settlement a kill and an occupancy check of the execution's [cgroup](super::Options::cgroup). It +//! feeds the execution's events back to the runner: the launch report, the output and its ends, +//! the bytes that the execution discarded, the program's end, once the execution collected its +//! status or the caller's own collector handed it over with [`observe`](Binding::observe), as +//! [`Options::caller_reaps`](super::Options::caller_reaps) has it, and, after the program's end, +//! the streams that [hung up](crate::exec::Runner::output_hung_up), which no process holds any +//! more. +//! +//! The adapter hands the binding the peer's messages, frames the runner's +//! [messages](crate::exec::runner::Message) that [`next`](Binding::next) returns, and drives the +//! binding without blocking, as it would the execution alone: it calls `next` until that returns +//! `None`, then waits until one of the binding's [descriptors](Binding::descriptors) polls +//! readable, its [input writer](Binding::input_writer) polls writable, its +//! [deadline](Binding::deadline) passes, or a message arrives. A stalled peer, sink, or transport +//! only withholds credit, so the binding stops reading the stream that waits on it, and nothing +//! else. Each call takes a bounded number of steps, its writes of the input among them, so that +//! no execution holds the adapter's loop: a call that stops with steps left sets the deadline to +//! its own instant, which the adapter's next wait finds passed already. Once the binding returned +//! a message, it steps the execution, if it can, before it returns another, so that input that +//! the adapter hands it between calls never keeps it from reading the program's output. +//! +//! Settlement goes through the cgroup that the execution's options name, if any, in the scope +//! that they give: once the program's status is known, the binding kills the cgroup's processes, +//! and reports the execution settled once the cgroup is empty, which it checks at most every 10 +//! ms. From then on, the cgroup is the caller's again: neither the execution nor its drop kills +//! the cgroup's processes, so the caller may reuse the cgroup once the outcome is reusable, +//! whether or not it dropped the binding. A zombie still occupies the cgroup until its parent +//! collects it, so a caller whose executions' orphans it adopts, such as a PID 1, collects their +//! statuses between its calls. +//! Without a cgroup, nothing proves that the program's descendants are gone, so the execution +//! never settles, and is fenced: the runner then gives up on the output streams that a process +//! may still hold the settlement time and the grace after the program's end, while it waits as +//! long as the peer takes for the rest of those that hung up. The runner's timers time the +//! execution: an execution with a [timeout](super::Options::timeout) of its own ends its program +//! without telling the runner why. + +use std::collections::VecDeque; +use std::io; +use std::os::fd::BorrowedFd; +use std::os::unix::process::ExitStatusExt; +use std::process::ExitStatus; +use std::time::{Duration, Instant}; + +use super::{Event as DriverEvent, Execution, KillTarget, Source, Stream as DriverStream}; +use crate::exec::{ + Cause, Key, LaunchFailure, Pool, Receipt, Runner, Settlement, Status, Stream, Violation, runner, +}; + +/// The interval between two checks of a settling cgroup. +const SETTLE_INTERVAL: Duration = Duration::from_millis(10); + +/// The most steps that one call of [`Binding::next`] takes: steps of the execution, each of which +/// reads at most the caller's buffer, and writes of the input, each of at most a chunk. So neither +/// output that the execution discards as fast as a process writes it, nor input that a process +/// reads as fast as the binding writes it, holds the caller's loop. +const STEPS: usize = 16; + +/// What one step of a [`Binding`] reports. +#[derive(Debug)] +#[non_exhaustive] +pub enum Event { + /// A message for the peer, which the adapter frames and sends in order. An output message's + /// bytes are at the start of the buffer that the adapter lent [`Binding::next`]. + Message(runner::Message), + /// A kill that ending the program needed failed, as the execution reports it. The binding + /// goes on: the runner's deadline bounds a program that the kill did not end. + KillFailed(KillTarget, io::Error), + /// Killing or checking the execution's cgroup failed, which fenced the execution. + SettlementFailed(io::Error), + /// A write of the program's input failed, other than because the program closed its end. The + /// binding closed the input, so that the program reads its end, and the runner tells the peer + /// that the input closed. + InputFailed(io::Error), +} + +/// An [`Execution`] that the execution engine's [`Runner`] drives: see the [module](self) +/// documentation. +/// +/// Dropping a binding drops its execution, which ends a program that has not ended, and the +/// processes of its cgroup unless the execution settled. +#[derive(Debug)] +pub struct Binding { + /// The execution. + execution: Execution, + /// The runner. + runner: Runner, + /// How many bytes of each stream the binding has granted the execution, and the execution + /// has not read. + granted: [usize; 2], + /// Whether the binding fed the execution's launch report to the runner. + launched: bool, + /// Whether the binding reported the program's end to the runner. + exited: bool, + /// Whether the binding told the runner that each stream hung up. + hung_up: [bool; 2], + /// Whether the execution ended: the driver has no step left. + ended: bool, + /// When the binding checks the settling cgroup next, while it settles it. + settling: Option, + /// The instant of the last call of [`next`](Self::next) if it took [`STEPS`] steps without an + /// event to return, and may have more to take: the binding's deadline, which has passed + /// already. + paused: Option, + /// Whether the execution's turn is due: once the binding returned a message, it takes a step + /// of the execution, if it has one, before it returns another, so that input credit, which + /// the writes of every call may free, never keeps it from reading the program's output. + execution_turn: bool, + /// The events that wait to be returned: failures, which the binding reports before messages. + events: VecDeque, +} + +impl Binding { + /// Binds `runner` to `execution`, whose program its adapter spawned for the runner's + /// execution. An execution without an input pipe gives its program no input that the peer + /// could send, so the runner tells the peer that the input closed. + pub fn new(execution: Execution, runner: Runner) -> Self { + let mut binding = Self { + execution, + runner, + granted: [0, 0], + launched: false, + exited: false, + hung_up: [false, false], + ended: false, + settling: None, + paused: None, + execution_turn: true, + events: VecDeque::new(), + }; + if binding.execution.input().is_none() { + binding.runner.input_closed(); + } + binding + } + + /// Returns the runner, whose state the adapter may inspect. + pub fn runner(&self) -> &Runner { + &self.runner + } + + /// Returns the execution, whose state the adapter may inspect. + pub fn execution(&self) -> &Execution { + &self.execution + } + + /// Returns the program's process ID. + pub fn id(&self) -> u32 { + self.execution.id() + } + + /// Hands the runner the peer's credit: see [`Runner::credit`]. + pub fn credit(&mut self, key: Key, pool: Pool, units: u32) -> Result { + self.runner.credit(key, pool, units) + } + + /// Hands the runner the peer's input: see [`Runner::input`]. + pub fn input(&mut self, key: Key, bytes: &[u8]) -> Result { + self.runner.input(key, bytes) + } + + /// Hands the runner the end of the peer's input: see [`Runner::end_input`]. + pub fn end_input(&mut self, key: Key) -> Result { + self.runner.end_input(key) + } + + /// Hands the runner the peer's cancellation: see [`Runner::cancel`]. + pub fn cancel(&mut self, now: Instant, key: Key, cause: Cause) -> Result { + self.runner.cancel(now, key, cause) + } + + /// Ends the execution because the adapter abandons it: see [`Runner::dropped`]. + pub fn dropped(&mut self, now: Instant) { + self.runner.dropped(now); + } + + /// Ends the execution because the session was reset: see [`Runner::reset`]. + pub fn reset(&mut self, now: Instant) { + self.runner.reset(now); + } + + /// Ends the execution because the transport was lost: see [`Runner::disconnect`]. + pub fn disconnect(&mut self, now: Instant) { + self.runner.disconnect(now); + } + + /// Hands the execution the status that the caller's own collector collected for the process + /// `id`, and returns whether it was the program's, as [`Execution::observe`] does. The next + /// call of [`next`](Self::next) reports the program's end to the runner before anything else. + pub fn observe(&mut self, id: u32, status: ExitStatus) -> bool { + self.execution.observe(id, status) + } + + /// Takes the next step without blocking, at `now`, and returns what it reports, or `None` if + /// nothing can happen until a descriptor polls ready, the deadline passes, or a message + /// arrives. Each read of an output stream goes into `buffer`, which bounds it, and each + /// output message's bytes are at the start of `buffer`. + /// + /// Each call first collects the program's status without blocking, unless the caller's own + /// collector takes it, and takes the program's launch report once it has arrived. Within a + /// step, the binding then reports the program's end to the runner once the execution has its + /// status, whether or not the launch report has arrived, which a process that another thread + /// forked meanwhile may hold back, tells the runner which streams hung up once a + /// timer is due after the program's end, fires the runner's timers, checks a settling + /// cgroup if a check is due, writes the input that the program's pipe takes, at most a chunk + /// at a time, applies the runner's requests, those that the last write raised included, such + /// as closing the input after its last byte, grants the reads that the runner allows, and + /// returns the runner's next message; failing that, it takes the execution's next step, feeds + /// its event to the runner, and starts over. So the program's end reaches the runner before + /// its timers fire or a queued message returns, even in a first call that comes after the + /// runner's timeout. Once the binding returned a message, the execution's turn comes before + /// the next: it takes the execution's next step, if it has one, and starts over, so that + /// input that the caller hands it between calls, which frees credit for each of them, never + /// keeps it from reading the program's output. The binding also starts over while anything + /// else changed, so that every request that a step raised is applied before it returns + /// `None`, and a due check runs however much output flows. It bounds the work of one call: + /// the steps of the execution and the writes of the input count against one bound, of which + /// the writes leave a step for the execution's turn while it is due, and a call that reaches + /// it without an event to return sets the [deadline](Self::deadline) to `now`, so that the + /// caller calls it again at once, after its other work. After the terminal outcome, it + /// returns `None`. + pub fn next(&mut self, now: Instant, buffer: &mut [u8]) -> Option { + self.paused = None; + self.collect_status(now); + self.take_launch_report(); + let mut steps = 0; + loop { + let mut progressed = self.report_status(now); + self.report_hang_ups(now); + self.runner.tick(now); + progressed |= self.settle(now); + progressed |= self.write_input(&mut steps); + progressed |= self.apply_requests(now); + self.grant(); + if let Some(event) = self.events.pop_front() { + return Some(event); + } + if std::mem::take(&mut self.execution_turn) && self.step_execution(now, buffer) { + steps += 1; + continue; + } + if let Some(message) = self.runner.next_message(buffer) { + self.execution_turn = true; + return Some(Event::Message(message)); + } + if self.runner.is_finished() { + return None; + } + if steps >= STEPS { + self.paused = Some(now); + return None; + } + if self.step_execution(now, buffer) { + steps += 1; + continue; + } + if !progressed { + return None; + } + } + } + + /// Returns the descriptors that move the execution on, each with what it waits on, which + /// the caller polls for reading, as [`Execution::descriptors`] describes. An output stream is + /// among them only while the runner lets the binding read it. + pub fn descriptors(&self) -> impl Iterator)> { + let active = !self.runner.is_finished(); + self.execution.descriptors().filter(move |_| active) + } + + /// Lends the writer of the program's input while the runner holds input for it, which the + /// caller polls for writing. + pub fn input_writer(&self) -> Option> { + if self.runner.pending_input().is_empty() { + return None; + } + self.execution.input() + } + + /// Returns when the binding must take a step even if no descriptor polls ready: the earliest + /// of the runner's deadline, the execution's, the next check of a settling cgroup, and the + /// instant of the last call of [`next`](Self::next) if it stopped with steps left. + pub fn deadline(&self) -> Option { + if self.runner.is_finished() { + return None; + } + [ + self.runner.deadline(), + self.execution.deadline(), + self.settling.filter(|_| self.runner.awaits_settlement()), + self.paused, + ] + .into_iter() + .flatten() + .min() + } + + /// Returns whether the runner returned its terminal outcome. + pub fn is_finished(&self) -> bool { + self.runner.is_finished() + } + + /// Applies the runner's requests to the execution, and returns whether there were any. + fn apply_requests(&mut self, now: Instant) -> bool { + let mut applied = false; + while let Some(request) = self.runner.next_request() { + applied = true; + match request { + runner::Request::Terminate(Cause::Cancelled) => { + self.execution.cancel(); + } + runner::Request::Terminate(Cause::Disconnected) => self.execution.disconnect(), + runner::Request::Terminate(_) => { + self.execution.terminate(); + } + runner::Request::Discard(stream) => self.execution.discard(driver_stream(stream)), + runner::Request::CloseInput => self.execution.close_input(), + runner::Request::Settle => self.begin_settlement(now), + } + } + applied + } + + /// Writes the input that the runner holds while the program's pipe takes it, at most a chunk + /// in each write, each of which counts as one of the call's `steps`, until they reach + /// [`STEPS`], or one less while the execution's turn is due, so that a step is left for it; + /// and tells the runner once the input closed: because the program closed its end, because + /// the execution closed it as the program ended, or because a write failed, after which the + /// binding closes it too, so that the program reads its end, and reports the failure. Returns + /// whether it wrote any byte. + fn write_input(&mut self, steps: &mut usize) -> bool { + let chunk = self.runner.limits().chunk(); + let limit = STEPS - usize::from(self.execution_turn); + let mut wrote = false; + while *steps < limit { + let pending = self.runner.pending_input(); + if pending.is_empty() { + break; + } + let pending = &pending[..pending.len().min(chunk)]; + match self.execution.write_input(pending) { + Ok(0) => break, + Ok(length) => { + wrote = true; + *steps += 1; + self.runner.input_written(length); + } + Err(error) if error.kind() == io::ErrorKind::WouldBlock => break, + Err(error) => { + // The execution closes the input itself only once the program closed its + // end, which is no failure: the binding closes it after any other failure, + // and reports that. + self.execution.close_input(); + self.runner.input_closed(); + if error.kind() != io::ErrorKind::BrokenPipe { + self.events.push_back(Event::InputFailed(error)); + } + break; + } + } + } + if self.execution.input().is_none() { + self.runner.input_closed(); + } + wrote + } + + /// Grants the execution the reads that the runner allows beyond what it granted already. + fn grant(&mut self) { + for stream in Stream::ALL { + let allowance = self.runner.read_allowance(stream); + let granted = &mut self.granted[index(stream)]; + if allowance > *granted { + self.execution + .grant(driver_stream(stream), allowance - *granted); + *granted = allowance; + } + } + } + + /// Takes the execution's next step at `now`, if it has one, feeds its event to the runner, + /// and reports the program's end once its status is known. Returns whether it took a step or + /// reported the end. + fn step_execution(&mut self, now: Instant, buffer: &mut [u8]) -> bool { + if self.ended { + return false; + } + let stepped = match self.execution.next_event(buffer) { + Ok(Some(event)) => { + self.feed(event); + true + } + Ok(None) => false, + // Another collector took the program's status, or collecting it failed: the + // program has ended, and its status is lost. + Err(_) => self.report_exit(now, None), + }; + let reported = self.report_status(now); + stepped || reported + } + + /// Feeds one event of the execution to the runner. + fn feed(&mut self, event: DriverEvent<'_>) { + match event { + DriverEvent::Launch(report) => { + self.launched = true; + self.runner.launched(match report { + Ok(None) => Ok(()), + Ok(Some(failure)) => Err(LaunchFailure::new( + failure.stage().code().unsigned_abs(), + failure.raw_os_error(), + )), + Err(error) => Err(LaunchFailure::new(0, error.raw_os_error().unwrap_or(0))), + }); + } + DriverEvent::Output(stream, bytes) => { + let stream = engine_stream(stream); + let granted = &mut self.granted[index(stream)]; + *granted = granted.saturating_sub(bytes.len()); + self.runner.output(stream, bytes); + } + DriverEvent::Discarded(stream, length) => { + self.runner.discarded(engine_stream(stream), length); + } + DriverEvent::Closed(stream) => self.runner.output_closed(engine_stream(stream)), + DriverEvent::ReadFailed(stream, _) => self.runner.output_failed(engine_stream(stream)), + DriverEvent::KillFailed(target, error) => { + self.events.push_back(Event::KillFailed(target, error)); + } + DriverEvent::Ended(_) => self.ended = true, + } + } + + /// Collects the program's status at `now` without blocking, if the execution collects it + /// itself. A failed collection reports the program's end with its status lost, as a failed + /// step does. + fn collect_status(&mut self, now: Instant) { + if self.execution.collect().is_err() { + self.report_exit(now, None); + } + } + + /// Feeds the program's launch report to the runner once it has arrived, unless the binding + /// did already. + fn take_launch_report(&mut self) { + if !self.launched + && let Some(report) = self.execution.launch() + { + self.feed(DriverEvent::Launch(report)); + } + } + + /// Reports the program's end at `now` to the runner once the execution has its status, unless + /// the binding did already, and returns whether it reported the end now. The end does not + /// wait for the launch report: a process that another thread forked while the program was + /// spawning holds the report's writer until it replaces itself with its program or ends, and + /// the runner's timeout would fire meanwhile. The runner holds the start, the output, and the + /// terminal outcome until the report arrives, or until the settlement time and the grace have + /// passed since the program's end, when it gives up on the report and fences the execution. + fn report_status(&mut self, now: Instant) -> bool { + match self.execution.status() { + Some(status) => self.report_exit(now, Some(status)), + None => false, + } + } + + /// Reports the program's end at `now` to the runner, with its status if known, unless it did + /// already, and returns whether it reported it now. + fn report_exit(&mut self, now: Instant, status: Option) -> bool { + if self.exited { + return false; + } + self.exited = true; + let status = status.and_then(|status| { + status + .code() + .map(Status::Exited) + .or_else(|| status.signal().map(Status::Signaled)) + }); + self.runner.exited(now, status); + true + } + + /// Tells the runner which output streams hung up, once it knows of the program's end and one + /// of its timers is due at `now`, before the timer fires: no process holds such a stream any + /// more, so the runner never gives up on it, even if the execution does not settle. + fn report_hang_ups(&mut self, now: Instant) { + if !self.exited || self.runner.deadline().is_none_or(|at| at > now) { + return; + } + for stream in Stream::ALL { + let hung_up = &mut self.hung_up[index(stream)]; + if !*hung_up && self.execution.output_hung_up(driver_stream(stream)) { + *hung_up = true; + self.runner.output_hung_up(stream); + } + } + } + + /// Begins to settle the execution at `now`: kills the processes of its cgroup, and checks + /// the cgroup from then on. Without a cgroup, or if the kill fails, the settlement fails. + fn begin_settlement(&mut self, now: Instant) { + let Some((cgroup, scope)) = &self.execution.cgroup else { + self.runner.settled(Settlement::Failed); + return; + }; + match cgroup.kill(*scope) { + Ok(()) => self.settling = Some(now), + Err(error) => { + self.runner.settled(Settlement::Failed); + self.events.push_back(Event::SettlementFailed(error)); + } + } + } + + /// Checks the settling cgroup at `now`, if a check is due: the execution settled once no + /// process occupies the cgroup, and its settlement failed if the check does. Returns whether + /// the settlement ended. + fn settle(&mut self, now: Instant) -> bool { + if !self.runner.awaits_settlement() { + self.settling = None; + } + if self.settling.is_none_or(|due| now < due) { + return false; + } + let Some((cgroup, scope)) = &self.execution.cgroup else { + return false; + }; + match cgroup.is_occupied(*scope) { + Ok(false) => { + self.settling = None; + self.execution.release_cgroup(); + self.runner.settled(Settlement::Settled); + true + } + Ok(true) => { + self.settling = now.checked_add(SETTLE_INTERVAL); + false + } + Err(error) => { + self.settling = None; + self.runner.settled(Settlement::Failed); + self.events.push_back(Event::SettlementFailed(error)); + true + } + } + } +} + +/// Returns `stream`'s position in [`Stream::ALL`]. +fn index(stream: Stream) -> usize { + match stream { + Stream::Stdout => 0, + Stream::Stderr => 1, + } +} + +/// Returns the driver's stream of the engine's `stream`. +fn driver_stream(stream: Stream) -> DriverStream { + match stream { + Stream::Stdout => DriverStream::Stdout, + Stream::Stderr => DriverStream::Stderr, + } +} + +/// Returns the engine's stream of the driver's `stream`. +fn engine_stream(stream: DriverStream) -> Stream { + match stream { + DriverStream::Stdout => Stream::Stdout, + DriverStream::Stderr => Stream::Stderr, + } +} + +#[cfg(test)] +mod tests; diff --git a/nvx-rs/src/guest/exec/binding/tests.rs b/nvx-rs/src/guest/exec/binding/tests.rs new file mode 100644 index 000000000..09434eef6 --- /dev/null +++ b/nvx-rs/src/guest/exec/binding/tests.rs @@ -0,0 +1,1726 @@ +//! Checks of bound executions of real programs, each with a peer in the same process: streams +//! beyond their windows, input beyond its window, an input that the program closes early, one +//! that closes with its last byte while output waits, and one whose write fails, settlement before +//! reuse and the fences of failed cleanups, the bounded wait for output that a descendant holds +//! once an execution does not settle, due settlement checks while output drains slowly, statuses +//! that the execution or the caller collects, which reach the runner ahead of queued output and of +//! its timeout, even in a first call after it, and while the launch report is late, the bounded +//! wait for a launch report that a process holds for good, the bounds on the steps and on the +//! writes of one call, output that flows while the caller hands over input before every call, and +//! two executions in one loop, one of them with a stalled sink or a pipe writer that escaped its +//! kill. The loop waits only for what the bindings and peers say that they wait on, so a step that +//! nothing leads to stalls a check rather than passing it. + +use std::ffi::CString; +use std::os::fd::OwnedFd; +use std::path::Path; +use std::time::{Duration, Instant}; + +use nix::poll::{PollFd, PollFlags, PollTimeout}; +use nix::sys::socket::{self, AddressFamily, SockFlag, SockType, sockopt}; + +use super::*; +use crate::exec::{Limits, Outcome, Peer, Timers, Window, peer}; +use crate::guest::cgroup::{ExecutionCgroup, Layout, Scope}; +use crate::guest::exec::{Options, Pipes}; +use crate::guest::process::Child; +use crate::guest::spawn::Command; +use crate::guest::sys::{self, ProcessId}; + +/// How long a test waits for its executions before it fails. +const PATIENCE: Duration = Duration::from_secs(30); +/// The grace of pending output in these tests. +const GRACE: Duration = Duration::from_millis(300); +/// The settlement time in these tests. +const SETTLEMENT: Duration = Duration::from_secs(10); +/// The key of each execution in these tests. +const KEY: Key = Key::new(1, 1); +/// The longest that a deadline may be overrun while the test's loop serves other executions. +const SLACK: Duration = Duration::from_secs(2); + +/// Returns limits with chunks of 16 KiB and byte windows of `window` bytes. +fn limits(window: u32) -> Limits { + Limits::new(16 * 1024, Window::bytes(window), Window::bytes(window)).unwrap() +} + +/// Returns the timers of these tests. +fn timers() -> Timers { + Timers::new(GRACE, SETTLEMENT) +} + +/// Returns a command that runs `/bin/sh -c script`, with `$0` set to `sh`. +fn shell(script: &str) -> Command { + Command::new( + c"/bin/sh".to_owned(), + vec![ + c"sh".to_owned(), + c"-c".to_owned(), + CString::new(script).unwrap(), + ], + ) +} + +/// Returns `seq 1 count`'s output. +fn sequence(count: usize) -> Vec { + (1..=count) + .map(|number| format!("{number}\n")) + .collect::() + .into_bytes() +} + +/// Returns a fake cgroup in `root`: a workload cgroup whose `cgroup.kill` records the kills, +/// and whose `pids.current` says `pids`. A test plays the kernel's part. +fn fake_cgroup(root: &Path, pids: &str) -> ExecutionCgroup { + let layout = Layout::new(root.join("agent"), root.join("workload")).unwrap(); + std::fs::create_dir_all(layout.workload()).unwrap(); + std::fs::write(layout.workload().join("cgroup.kill"), b"").unwrap(); + std::fs::write(layout.workload().join("pids.current"), pids).unwrap(); + ExecutionCgroup::open(&layout, None).unwrap() +} + +/// One bound execution and its peer, connected directly, with the peer's sink and source. +struct Side { + /// The runner's side: the binding of the execution. + binding: Binding, + /// The peer. + peer: Peer, + /// The bytes of each stream that the sink took. + output: [Vec; 2], + /// Whether the sink takes nothing. + stalled: bool, + /// The input that the source hands over. + input: Vec, + /// The bytes of input that the source handed over. + sent: usize, + /// The peer's events. + events: Vec, + /// The runner's terminal outcome, once it returned it. + terminal: Option, + /// The failures that the binding reported. + failures: Vec, +} + +impl Side { + /// Spawns `command` with pipes, its standard input piped if `input` and the null device + /// otherwise, in a process group of its own, and binds a runner with `limits` and `timers` + /// to its execution with `options`, with a peer of its own. + fn start( + command: Command, + input: bool, + options: Options, + limits: Limits, + timers: Timers, + ) -> Self { + let (pipes, ends) = Pipes::new(input).unwrap(); + let stdin = match ends.stdin { + Some(stdin) => stdin, + None => OwnedFd::from(std::fs::File::open("/dev/null").unwrap()), + }; + let spawned = command + .stdin(stdin) + .stdout(ends.stdout) + .stderr(ends.stderr) + .new_process_group() + .spawn() + .unwrap(); + let execution = Execution::new(spawned, pipes, options.process_group()).unwrap(); + Self::bound(execution, limits, timers) + } + + /// Binds a runner with `limits` and `timers` to `execution`, with a peer of its own. + fn bound(execution: Execution, limits: Limits, timers: Timers) -> Self { + let now = Instant::now(); + Self { + binding: Binding::new(execution, Runner::new(KEY, limits, timers, now)), + peer: Peer::new(KEY, limits, timers, now), + output: [Vec::new(), Vec::new()], + stalled: false, + input: Vec::new(), + sent: 0, + events: Vec::new(), + terminal: None, + failures: Vec::new(), + } + } + + /// Takes every step that the binding and the peer can take at `now` without waiting, and + /// returns whether anything happened. + fn pump(&mut self, now: Instant, buffer: &mut [u8]) -> bool { + let mut progressed = false; + while let Some(event) = self.binding.next(now, buffer) { + progressed = true; + self.handle(now, event, buffer); + } + self.serve(now, buffer) || progressed + } + + /// Takes what one step of the binding at `now` reported: hands the peer a message, whose + /// output is in `buffer`, and records a failure. + fn handle(&mut self, now: Instant, event: Event, buffer: &[u8]) { + match event { + Event::Message(message) => self.deliver(now, message, buffer), + Event::KillFailed(target, error) => { + self.failures.push(format!("kill {target:?}: {error}")); + } + Event::SettlementFailed(error) => { + self.failures.push(format!("settlement: {error}")); + } + Event::InputFailed(error) => { + self.failures.push(format!("input: {error}")); + } + } + } + + /// Takes every step that the peer, its source, and its sink can take at `now` without + /// waiting, hands the peer's messages to the binding at once, and returns whether anything + /// happened. + fn serve(&mut self, now: Instant, buffer: &mut [u8]) -> bool { + let mut progressed = false; + if self.sent < self.input.len() { + let length = self.peer.input(&self.input[self.sent..]); + self.sent += length; + progressed |= length > 0; + } + if self.sent == self.input.len() { + self.peer.end_input(); + } + if !self.stalled { + for stream in Stream::ALL { + loop { + let pending = self.peer.pending_output(stream); + if pending.is_empty() { + break; + } + let length = pending.len(); + self.output[index(stream)].extend_from_slice(pending); + self.peer.consumed(stream, length); + progressed = true; + } + } + } + self.peer.tick(now); + while let Some(event) = self.peer.next_event() { + progressed = true; + self.events.push(event); + } + while let Some(message) = self.peer.next_message(buffer) { + progressed = true; + match message { + peer::Message::Cancel(cause) => self.binding.cancel(now, KEY, cause), + peer::Message::Credit { pool, units } => self.binding.credit(KEY, pool, units), + peer::Message::Input { length } => self.binding.input(KEY, &buffer[..length]), + peer::Message::InputEnd => self.binding.end_input(KEY), + } + .expect("the runner accepts the peer's message"); + } + progressed + } + + /// Hands the peer the runner's `message` at `now`, whose output is in `buffer`. + fn deliver(&mut self, now: Instant, message: runner::Message, buffer: &[u8]) { + let receipt = match message { + runner::Message::Credit { units } => self.peer.credit(KEY, units), + runner::Message::Started => self.peer.started(KEY), + runner::Message::Output { stream, length } => { + self.peer.output(KEY, stream, &buffer[..length]) + } + runner::Message::End(stream) => self.peer.end_output(KEY, stream), + runner::Message::InputClosed => self.peer.input_closed(KEY), + runner::Message::Terminal(outcome) => { + assert!(self.terminal.is_none(), "one terminal outcome"); + self.terminal = Some(outcome); + if self.binding.runner().is_detached() { + return; + } + self.peer.terminal(now, KEY, outcome) + } + }; + receipt.expect("the peer accepts the runner's message"); + } + + /// Returns whether both sides finished. + fn is_finished(&self) -> bool { + self.binding.is_finished() && self.peer.is_finished() + } + + /// Returns the peer's outcome, once it finished. + fn outcome(&self) -> Option { + self.events.iter().find_map(|event| match event { + peer::Event::Finished(outcome) => Some(*outcome), + _ => None, + }) + } +} + +/// Waits until a descriptor of `sides` polls ready, or the earliest of their deadlines and +/// `until` passes, as an adapter's loop does. +fn wait(sides: &[&mut Side], until: Instant) { + let mut descriptors = Vec::new(); + for side in sides { + for (_, descriptor) in side.binding.descriptors() { + descriptors.push(PollFd::new(descriptor, PollFlags::POLLIN)); + } + if let Some(descriptor) = side.binding.input_writer() { + descriptors.push(PollFd::new(descriptor, PollFlags::POLLOUT)); + } + } + let deadline = sides + .iter() + .flat_map(|side| [side.binding.deadline(), side.peer.deadline()]) + .flatten() + .fold(until, Instant::min); + let timeout = deadline.saturating_duration_since(Instant::now()); + // Poll counts whole milliseconds, so a shorter wait would return before its deadline. + let timeout = if timeout.is_zero() { + timeout + } else { + timeout.max(Duration::from_millis(1)) + }; + match nix::poll::poll(&mut descriptors, PollTimeout::try_from(timeout).unwrap()) { + Ok(_) | Err(nix::errno::Errno::EINTR) => {} + Err(error) => panic!("poll failed: {error}"), + } +} + +/// Drives `sides` in one loop, as an adapter does, until `done` holds: it takes every step that +/// they can take, then waits for what their bindings and peers wait on, and for `wake` if any. +/// It fails unless `done` holds within the test's patience, and fails as soon as a wait runs +/// into it, so a step that nothing they wait on leads to stalls the test rather than passing it. +fn run(sides: &mut [&mut Side], wake: Option, mut done: impl FnMut(&[&mut Side]) -> bool) { + let patience = Instant::now() + PATIENCE; + let until = wake.map_or(patience, |wake| wake.min(patience)); + let mut buffer = vec![0u8; 64 * 1024]; + loop { + let mut progressed = false; + for side in sides.iter_mut() { + progressed |= side.pump(Instant::now(), &mut buffer); + } + if done(sides) { + return; + } + if !progressed { + wait(sides, until); + } + assert!(Instant::now() < patience, "the executions moved on in time"); + } +} + +/// Drives `sides` in one loop until `done` holds: see [`run`]. +fn drive(sides: &mut [&mut Side], done: impl FnMut(&[&mut Side]) -> bool) { + run(sides, None, done); +} + +/// Drives `sides` in one loop for `duration`: see [`run`]. +fn drive_for(sides: &mut [&mut Side], duration: Duration) { + let until = Instant::now() + duration; + run(sides, Some(until), |_| Instant::now() >= until); +} + +/// Drives `side` alone until both of its sides finished, and returns the peer's outcome. +fn finish(side: &mut Side) -> Outcome { + drive(&mut [side], |sides| sides[0].is_finished()); + side.outcome().unwrap() +} + +#[test] +fn output_beyond_its_windows_arrives_whole_and_the_outcome_is_lossless() { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut side = Side::start( + shell("seq 1 400000; seq 1 20000 >&2"), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(64 * 1024), + timers(), + ); + let outcome = finish(&mut side); + assert_eq!(side.output[0], sequence(400_000)); + assert_eq!(side.output[1], sequence(20_000)); + assert_eq!(Some(outcome), side.terminal); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(!outcome.is_truncated(Stream::Stdout) && !outcome.is_truncated(Stream::Stderr)); + assert!(outcome.is_reusable()); + assert!(side.failures.is_empty(), "{:?}", side.failures); + let runner = side.binding.runner(); + assert!(runner.high_water() <= runner.memory_bound()); + assert!(side.peer.high_water() <= side.peer.memory_bound()); + // Settlement killed the execution's cgroup, whose emptiness the binding checked. + assert_eq!( + std::fs::read(cgroup.path().join("cgroup.kill")).unwrap(), + b"1" + ); + assert_eq!( + side.binding.next(Instant::now(), &mut [0; 16]).map(drop), + None + ); + assert!(side.binding.descriptors().next().is_none()); + assert_eq!(side.binding.deadline(), None); +} + +#[test] +fn input_longer_than_its_window_reaches_the_program_intact() { + // With byte windows, and with frames of up to 64 KiB less 256 bytes in a shared pool. + let frames = Limits::new(64 * 1024 - 256, Window::frames(4), Window::frames(64)) + .unwrap() + .shared_output(); + for limits in [limits(16 * 1024), frames] { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut side = Side::start( + shell("exec cat"), + true, + Options::new().cgroup(&cgroup, Scope::Execution), + limits, + timers(), + ); + side.input = (0..3usize << 20) + .map(|index| (index * 7 + index / 251) as u8) + .collect(); + let outcome = finish(&mut side); + assert_eq!(side.output[0], side.input, "{limits:?}"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(outcome.is_reusable()); + let runner = side.binding.runner(); + assert!(runner.high_water() <= runner.memory_bound()); + assert!(side.peer.high_water() <= side.peer.memory_bound()); + } +} + +#[test] +fn a_program_that_closes_its_input_early_stops_the_peer_s_source() { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut side = Side::start( + shell("head -c 5; printf done >&2"), + true, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(16 * 1024), + timers(), + ); + side.input = vec![b'x'; 16 << 20]; + let outcome = finish(&mut side); + assert_eq!(side.output, [b"xxxxx".to_vec(), b"done".to_vec()]); + assert!(side.events.contains(&peer::Event::InputClosed)); + // The source stopped long before its 16 MiB. + assert!(side.sent < 1 << 20, "{}", side.sent); + // A program that closes its input is no failure. + assert!(side.failures.is_empty(), "{:?}", side.failures); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(outcome.is_reusable()); +} + +#[test] +fn the_input_closes_with_its_last_byte_while_output_waits_to_be_returned() { + // The program writes a line, then copies its input to its output. The runner holds the line + // until the peer's credit reaches the binding, together with the input and its end: the + // call that writes the input's last byte returns the line, and closes the program's input + // before it does. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut side = Side::start( + shell("echo ready; exec cat"), + true, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(4096), + timers(), + ); + // The runner's messages reach the peer, and the peer's wait, until the runner holds the + // line, and the peer knows that the program started: a call of the binding may read the line + // after it looked for the launch report. + let mut buffer = vec![0u8; 64 * 1024]; + let patience = Instant::now() + PATIENCE; + loop { + let now = Instant::now(); + let mut progressed = false; + while let Some(event) = side.binding.next(now, &mut buffer) { + progressed = true; + side.handle(now, event, &buffer); + } + if side.binding.runner().buffered() == b"ready\n".len() && side.peer.has_started() { + break; + } + if !progressed { + wait(&[&mut side], patience); + } + assert!(Instant::now() < patience, "the program wrote in time"); + } + side.input = b"data".to_vec(); + let now = Instant::now(); + side.serve(now, &mut buffer); + let event = side.binding.next(now, &mut buffer); + assert!( + matches!( + event, + Some(Event::Message(runner::Message::Output { + stream: Stream::Stdout, + length: 6, + })) + ), + "{event:?}" + ); + assert!(side.binding.execution().input().is_none()); + side.handle(now, event.unwrap(), &buffer); + let outcome = finish(&mut side); + assert_eq!(side.output[0], b"ready\ndata"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(outcome.is_reusable()); +} + +#[test] +fn a_write_that_fails_closes_the_program_s_input() { + // No pipe refuses a write but for `EPIPE`, so the program reads a socket of records instead, + // whose writer refuses a record longer than its buffer with `EMSGSIZE`, as it refuses the + // binding's first chunk. The binding closes the writer then, and reports the failure, as the + // runner drops the rest of the input, so the program reads the input's end, and ends on its + // own. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let (reader, writer) = socket::socketpair( + AddressFamily::Unix, + SockType::SeqPacket, + None, + SockFlag::SOCK_CLOEXEC, + ) + .unwrap(); + // Linux doubles the size, and refuses records longer than the result, less 32 bytes. + socket::setsockopt(&writer, sockopt::SndBuf, &4096).unwrap(); + let writer = sys::FileDescriptor::from_owned(writer); + writer.set_nonblocking().unwrap(); + let (mut pipes, ends) = Pipes::new(true).unwrap(); + pipes.input = Some(writer); + let spawned = shell("exec cat") + .stdin(reader) + .stdout(ends.stdout) + .stderr(ends.stderr) + .new_process_group() + .spawn() + .unwrap(); + let options = Options::new() + .cgroup(&cgroup, Scope::Execution) + .process_group(); + let execution = Execution::new(spawned, pipes, options).unwrap(); + let mut side = Side::bound(execution, limits(64 * 1024), timers()); + side.input = vec![b'i'; 16 * 1024]; + let outcome = finish(&mut side); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(side.output.iter().all(Vec::is_empty)); + assert!(outcome.is_reusable()); + let refused = io::Error::from(nix::errno::Errno::EMSGSIZE); + assert_eq!(side.failures, [format!("input: {refused}")]); +} + +/// Returns the process that `output` names by its ID on one line, opened by a pidfd that polls +/// readable once it ends. +fn descendant(output: &[u8]) -> (ProcessId, crate::guest::sys::FileDescriptor) { + let id = std::str::from_utf8(output).unwrap().trim().parse().unwrap(); + let process = ProcessId::from_positive(id).unwrap(); + (process, sys::open_pidfd(process).unwrap()) +} + +#[test] +fn descendants_and_pipe_writers_settle_before_reuse_in_a_cgroup_that_the_test_plays() { + // The fake cgroup counts the program and two descendants that outlive it: one in a session + // of its own that holds the program's standard output, and one that holds nothing. The test + // plays the kernel: it ends the descendants once the cgroup is killed, and only then counts + // no process. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "3\n"); + let kill = cgroup.path().join("cgroup.kill"); + let pids = cgroup.path().join("pids.current"); + let mut side = Side::start( + shell( + "setsid sleep 60 2>/dev/null & echo $!; \ + sleep 60 /dev/null 2>&1 & echo $! >&2", + ), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(16 * 1024), + timers(), + ); + // The program ends at once, and its standard error with it, as no descendant holds it. Its + // standard output stays open, and the cgroup is killed. + drive(&mut [&mut side], |sides| { + sides[0].output.iter().all(|output| output.ends_with(b"\n")) + && sides[0].events.contains(&peer::Event::End(Stream::Stderr)) + && sides[0].binding.execution().status().is_some() + }); + assert_eq!(std::fs::read(&kill).unwrap(), b"1"); + assert!(!side.events.contains(&peer::Event::End(Stream::Stdout))); + let holder = descendant(&side.output[0]); + let detached = descendant(&side.output[1]); + // The kernel's kill: the descendants end, and the pipe's last writer with them. + for (process, _) in [&holder, &detached] { + sys::kill_process(*process).unwrap(); + } + for (_, pidfd) in [&holder, &detached] { + let patience = u16::try_from(PATIENCE.as_millis()).unwrap(); + assert_eq!( + sys::readable_or_hung_up(&[pidfd.borrowed()], patience).unwrap(), + [true] + ); + } + // The program's end, the streams' ends, and the descendants' end are not settlement: the + // cgroup still counts processes, so the execution stays unsettled. + drive(&mut [&mut side], |sides| { + sides[0].events.contains(&peer::Event::End(Stream::Stdout)) + }); + drive_for(&mut [&mut side], Duration::from_millis(300)); + assert!(!side.binding.is_finished()); + assert!(side.terminal.is_none()); + assert!(side.binding.runner().awaits_settlement()); + // Once the cgroup is empty, the execution settles, and is reusable. + std::fs::write(&pids, b"0\n").unwrap(); + let outcome = finish(&mut side); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable()); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(side.failures.is_empty(), "{:?}", side.failures); +} + +#[test] +fn the_settlement_kills_what_the_program_leaves_behind_in_a_cgroup_that_the_test_plays() { + // The program's descendant holds neither output stream, and both streams end while the + // program runs, so that once it ends, the settlement's kill is the cgroup's only one. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "2\n"); + let kill = cgroup.path().join("cgroup.kill"); + let go = temporary.path().join("go"); + let mut side = Side::start( + shell(&format!( + "sleep 60 /dev/null 2>&1 & echo $!; exec >&- 2>&-; \ + until [ -e '{}' ]; do sleep 0.01; done", + go.display() + )), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(16 * 1024), + timers(), + ); + drive(&mut [&mut side], |sides| { + Stream::ALL + .into_iter() + .all(|stream| sides[0].events.contains(&peer::Event::End(stream))) + }); + let (process, pidfd) = descendant(&side.output[0]); + assert!(side.binding.execution().status().is_none()); + assert_eq!(std::fs::read(&kill).unwrap(), b""); + // The program ends; the settlement kills the cgroup, and checks it from then on. + std::fs::write(&go, b"").unwrap(); + drive(&mut [&mut side], |sides| { + sides[0].binding.execution().status().is_some() + }); + assert_eq!(std::fs::read(&kill).unwrap(), b"1"); + assert!(side.binding.runner().awaits_settlement()); + assert!(!side.binding.is_finished()); + // The kernel's kill ends the descendant, after which the cgroup counts no process. + sys::kill_process(process).unwrap(); + let patience = u16::try_from(PATIENCE.as_millis()).unwrap(); + assert_eq!( + sys::readable_or_hung_up(&[pidfd.borrowed()], patience).unwrap(), + [true] + ); + std::fs::write(cgroup.path().join("pids.current"), b"0\n").unwrap(); + let outcome = finish(&mut side); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable()); + assert!(side.failures.is_empty(), "{:?}", side.failures); +} + +#[test] +fn the_settlement_begins_as_the_program_ends_while_a_descendant_holds_its_output() { + // A descendant in a session of its own holds both of the program's output streams, and the + // fake cgroup counts no process. The program ends once the test lets it, when nothing is + // left to read, so that nothing but its end leads to the settlement, which is over before + // the output ends. Once the outcome is reusable, the caller reuses the cgroup before it drops + // the binding, which then kills nothing in it. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let go = temporary.path().join("go"); + let mut side = Side::start( + shell(&format!( + "setsid sleep 60 & echo $!; until [ -e '{}' ]; do sleep 0.01; done", + go.display() + )), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(16 * 1024), + timers(), + ); + drive(&mut [&mut side], |sides| { + sides[0].output[0].ends_with(b"\n") + }); + let holder = Stray(descendant(&side.output[0])); + std::fs::write(&go, b"").unwrap(); + drive(&mut [&mut side], |sides| { + sides[0].binding.execution().status().is_some() + && !sides[0].binding.runner().awaits_settlement() + }); + assert!( + side.events + .iter() + .all(|event| !matches!(event, peer::Event::End(_))) + ); + assert!(!side.binding.is_finished()); + // The output's end is the last that the terminal outcome waits for. + drop(holder); + let outcome = finish(&mut side); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable()); + assert!(!outcome.is_truncated(Stream::Stdout)); + // The settlement killed the cgroup, which is the caller's from then on. + let kill = cgroup.path().join("cgroup.kill"); + assert_eq!(std::fs::read(&kill).unwrap(), b"1"); + std::fs::write(&kill, b"").unwrap(); + drop(side); + assert_eq!(std::fs::read(&kill).unwrap(), b""); +} + +/// A workload cgroup in the real hierarchy, which the test removes, with everything in it, when +/// it ends. +struct RealWorkload(Layout); + +impl RealWorkload { + /// Creates a new workload cgroup below the root of the hierarchy at `/sys/fs/cgroup`, with + /// the test's own cgroup as the agent's. It returns `None` unless the test runs as root in a + /// cgroup v2 hierarchy with the `pids` controller, in which it can create a cgroup. + fn new(name: &str) -> Option { + let root = Path::new("/sys/fs/cgroup"); + let controllers = std::fs::read_to_string(root.join("cgroup.controllers")).ok()?; + if !nix::unistd::geteuid().is_root() + || !controllers + .split_whitespace() + .any(|controller| controller == "pids") + { + return None; + } + let own = std::fs::read_to_string("/proc/self/cgroup").ok()?; + let own = own.lines().find_map(|line| line.strip_prefix("0::"))?; + let agent = root.join(own.trim_start_matches('/')); + let workload = root.join(format!("nvx-binding-{}-{name}", std::process::id())); + let layout = Layout::new(agent, workload).ok()?; + std::fs::create_dir(layout.workload()).ok()?; + Some(Self(layout)) + } +} + +impl Drop for RealWorkload { + /// Kills whatever is left in the workload cgroup, then removes its cgroups. + fn drop(&mut self) { + let workload = self.0.workload(); + let _ = std::fs::write(workload.join("cgroup.kill"), b"1"); + for _ in 0..1000 { + if std::fs::read(workload.join("pids.current")).is_ok_and(|count| count == b"0\n") { + break; + } + std::thread::sleep(Duration::from_millis(10)); + } + if let Ok(entries) = std::fs::read_dir(workload) { + for entry in entries.flatten() { + if entry.file_type().is_ok_and(|kind| kind.is_dir()) { + let _ = std::fs::remove_dir(entry.path()); + } + } + } + let _ = std::fs::remove_dir(workload); + } +} + +#[test] +fn descendants_setsid_and_pipe_writers_settle_before_reuse_in_a_real_cgroup() { + // The test changes the real hierarchy, which needs root; elsewhere it does nothing. + let Some(workload) = RealWorkload::new("settle") else { + return; + }; + let cgroup = ExecutionCgroup::prepare(&workload.0, Some("exec"), &[]).unwrap(); + // The program ends only once the test has opened its descendants, which the settlement + // kills as soon as the binding collects the program's status. + let temporary = tempfile::tempdir().unwrap(); + let opened = temporary.path().join("opened"); + let mut side = Side::start( + shell(&format!( + "setsid sleep 60 & echo $!; \ + sleep 60 /dev/null 2>&1 & echo $! >&2; \ + while [ ! -e '{}' ]; do sleep 0.01; done", + opened.display() + )) + .cgroup(cgroup.placement().unwrap()), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(16 * 1024), + timers(), + ); + drive(&mut [&mut side], |sides| { + sides[0].output.iter().all(|output| output.ends_with(b"\n")) + }); + let holder = descendant(&side.output[0]); + let detached = descendant(&side.output[1]); + std::fs::write(&opened, b"").unwrap(); + let outcome = finish(&mut side); + // The terminal outcome came once the cgroup was empty: both descendants, the one in a + // session of its own that held the output included, are gone. + assert!(!cgroup.is_occupied(Scope::Execution).unwrap()); + for (_, pidfd) in [&holder, &detached] { + assert_eq!( + sys::readable_or_hung_up(&[pidfd.borrowed()], 0).unwrap(), + [true] + ); + } + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable(), "{:?}", side.failures); + assert!(!outcome.is_truncated(Stream::Stdout)); + // The caller reuses the cgroup at once, for a new program, before it drops the binding: the + // new program outlives the drop. + let spawned = shell("exec sleep 60") + .cgroup(cgroup.placement().unwrap()) + .spawn() + .unwrap(); + let id = spawned.id(); + let pidfd = sys::open_pidfd(ProcessId::from_positive(i64::from(id)).unwrap()).unwrap(); + assert_eq!(spawned.into_report().wait().unwrap(), None); + drop(side); + assert!(cgroup.is_occupied(Scope::Execution).unwrap()); + assert_eq!( + sys::readable_or_hung_up(&[pidfd.borrowed()], 0).unwrap(), + [false] + ); + cgroup.kill(Scope::Execution).unwrap(); + let status = Child::new(id).unwrap().wait().unwrap(); + assert_eq!(status.signal(), Some(9)); + cgroup.remove().unwrap(); +} + +#[test] +fn a_cleanup_failure_fences_the_execution() { + // A cgroup whose kill fails, one that never empties, and none at all. + let temporary = tempfile::tempdir().unwrap(); + let unkillable = fake_cgroup(&temporary.path().join("unkillable"), "0\n"); + std::fs::remove_file(unkillable.path().join("cgroup.kill")).unwrap(); + let occupied = fake_cgroup(&temporary.path().join("occupied"), "1\n"); + let short = Timers::new(GRACE, Duration::from_millis(300)); + for (options, timers, failed) in [ + ( + Options::new().cgroup(&unkillable, Scope::Execution), + timers(), + true, + ), + ( + Options::new().cgroup(&occupied, Scope::Execution), + short, + false, + ), + (Options::new(), timers(), false), + ] { + let mut side = Side::start(shell("printf ran"), false, options, limits(4096), timers); + let outcome = finish(&mut side); + assert_eq!(side.output[0], b"ran"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(!outcome.is_reusable()); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert_eq!( + side.failures + .iter() + .any(|failure| failure.starts_with("settlement")), + failed, + "{:?}", + side.failures + ); + } +} + +#[test] +fn an_unrelated_child_s_status_stays_with_its_owner() { + let mut unrelated = std::process::Command::new("/bin/sh") + .args(["-c", "exit 7"]) + .spawn() + .unwrap(); + while !sys::has_exited(ProcessId::from_child(&unrelated)).unwrap() { + std::thread::yield_now(); + } + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut side = Side::start( + shell("printf ran"), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(4096), + timers(), + ); + assert!(finish(&mut side).is_reusable()); + // Nothing collected the unrelated child's status, which its own wait still finds. + assert_eq!(unrelated.wait().unwrap().code(), Some(7)); +} + +#[test] +fn a_caller_that_reaps_hands_the_program_s_status_over() { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut side = Side::start( + shell("printf ran; exit 6"), + false, + Options::new() + .cgroup(&cgroup, Scope::Execution) + .caller_reaps(), + limits(4096), + timers(), + ); + let id = side.binding.id(); + // The binding never waits for the program, nor lists its pidfd. + assert!( + side.binding + .descriptors() + .all(|(source, _)| source != Source::Exit) + ); + // The program ends, which nothing in the binding waits for. + drive(&mut [&mut side], |sides| { + sides[0].output[0] == b"ran" + && Stream::ALL + .into_iter() + .all(|stream| sides[0].events.contains(&peer::Event::End(stream))) + }); + let process = ProcessId::from_positive(i64::from(id)).unwrap(); + let started = Instant::now(); + while !sys::has_exited(process).unwrap() { + assert!(started.elapsed() < PATIENCE, "the program ended in time"); + std::thread::sleep(Duration::from_millis(1)); + } + drive_for(&mut [&mut side], Duration::from_millis(100)); + assert!(!side.binding.is_finished()); + // The caller's own collector takes the status, and hands it over. + let status = Child::new(id).unwrap().wait().unwrap(); + assert!(side.binding.observe(id, status)); + let outcome = finish(&mut side); + assert_eq!(outcome.status(), Some(Status::Exited(6))); + assert!(outcome.is_reusable()); +} + +#[test] +fn the_program_s_end_reaches_the_runner_before_queued_output_whoever_collects_its_status() { + // The program writes 24 KiB to a sink that takes nothing: the peer holds a window, 4 KiB, + // the runner a chunk, 16 KiB, and the pipe the rest, 4 KiB, which even a pipe of one page + // holds. Then it exits 3, once the test lets it, before the runner's timeout, while nothing + // collects its status: neither the execution, which the test does not drive meanwhile, nor + // the caller's own collector, which then hands it over. Every call from then on comes after + // the timeout, and the sink takes the output 64 bytes at a time, so that every call has + // queued output to return: the program's end reached the runner first, so the timeout never + // fires. + const TIMEOUT: Duration = Duration::from_secs(20); + const PAYLOAD: usize = 24 * 1024; + let limits = limits(4096); + for caller_reaps in [false, true] { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let gate = temporary.path().join("gate"); + let mut options = Options::new().cgroup(&cgroup, Scope::Execution); + if caller_reaps { + options = options.caller_reaps(); + } + let mut side = Side::start( + shell(&format!( + "head -c {PAYLOAD} /dev/zero; while [ ! -e '{}' ]; do sleep 0.01; done; exit 3", + gate.display() + )), + false, + options, + limits, + Timers::new(GRACE, TIMEOUT * 3).timeout(TIMEOUT), + ); + // Only the runner has the timeout. The calls after it come within its settlement time. + side.peer = Peer::new(KEY, limits, timers(), Instant::now()); + let late = Instant::now() + TIMEOUT; + side.stalled = true; + drive(&mut [&mut side], |sides| { + sides[0].peer.buffered() == 4096 + && sides[0].binding.runner().read_allowance(Stream::Stdout) == 0 + }); + std::fs::write(&gate, b"").unwrap(); + let id = side.binding.id(); + let process = ProcessId::from_positive(i64::from(id)).unwrap(); + let started = Instant::now(); + while !sys::has_exited(process).unwrap() { + assert!(started.elapsed() < PATIENCE, "the program ended in time"); + std::thread::sleep(Duration::from_millis(1)); + } + if caller_reaps { + let status = Child::new(id).unwrap().wait().unwrap(); + assert!(side.binding.observe(id, status)); + } + side.stalled = false; + let mut buffer = [0u8; 64]; + side.serve(late, &mut buffer); + let event = side.binding.next(late, &mut buffer); + assert!( + matches!( + event, + Some(Event::Message(runner::Message::Output { + stream: Stream::Stdout, + .. + })) + ), + "{event:?}" + ); + side.handle(late, event.unwrap(), &buffer); + let patience = Instant::now() + PATIENCE; + while !side.is_finished() { + assert!(Instant::now() < patience, "the execution finished in time"); + if !side.pump(late, &mut buffer) { + wait(&[&mut side], Instant::now() + Duration::from_millis(10)); + } + } + let terminal = side.terminal.unwrap(); + assert_eq!(terminal.status(), Some(Status::Exited(3)), "{caller_reaps}"); + assert_eq!(terminal.cause(), None, "{caller_reaps}"); + assert!(!terminal.is_truncated(Stream::Stdout), "{caller_reaps}"); + assert!(terminal.is_reusable(), "{caller_reaps}"); + assert_eq!(side.output[0], vec![0; PAYLOAD], "{caller_reaps}"); + assert!(side.failures.is_empty(), "{:?}", side.failures); + } +} + +#[test] +fn the_program_s_end_reaches_the_runner_before_its_timeout_in_a_first_call_after_it() { + // The program writes a word and exits 4 at once, and its launch report arrives, but the first + // call comes only after the runner's timeout, whoever collects the program's status: the + // launch report and the program's end reach the runner before the timeout fires, so the + // output arrives whole, and the outcome names no cause. + const TIMEOUT: Duration = Duration::from_secs(20); + let limits = limits(4096); + for caller_reaps in [false, true] { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut options = Options::new().cgroup(&cgroup, Scope::Execution); + if caller_reaps { + options = options.caller_reaps(); + } + let mut side = Side::start( + shell("printf hello; exit 4"), + false, + options, + limits, + Timers::new(GRACE, TIMEOUT * 3).timeout(TIMEOUT), + ); + // Only the runner has the timeout. + side.peer = Peer::new(KEY, limits, timers(), Instant::now()); + let late = Instant::now() + TIMEOUT; + let id = side.binding.id(); + let process = ProcessId::from_positive(i64::from(id)).unwrap(); + let started = Instant::now(); + while !sys::has_exited(process).unwrap() { + assert!(started.elapsed() < PATIENCE, "the program ended in time"); + std::thread::sleep(Duration::from_millis(1)); + } + if caller_reaps { + let status = Child::new(id).unwrap().wait().unwrap(); + assert!(side.binding.observe(id, status)); + } + let report = side + .binding + .descriptors() + .find_map(|(source, descriptor)| matches!(source, Source::Launch).then_some(descriptor)) + .expect("the launch report is pending"); + let patience = u16::try_from(PATIENCE.as_millis()).unwrap(); + assert_eq!( + sys::readable_or_hung_up(&[report], patience).unwrap(), + [true] + ); + let mut buffer = [0u8; 64]; + let patience = Instant::now() + PATIENCE; + while !side.is_finished() { + assert!(Instant::now() < patience, "the execution finished in time"); + if !side.pump(late, &mut buffer) { + wait(&[&mut side], Instant::now() + Duration::from_millis(10)); + } + } + let terminal = side.terminal.unwrap(); + assert_eq!(terminal.status(), Some(Status::Exited(4)), "{caller_reaps}"); + assert_eq!(terminal.cause(), None, "{caller_reaps}"); + assert!(!terminal.is_truncated(Stream::Stdout), "{caller_reaps}"); + assert!(terminal.is_reusable(), "{caller_reaps}"); + assert_eq!(side.output[0], b"hello", "{caller_reaps}"); + assert!(side.failures.is_empty(), "{:?}", side.failures); + } +} + +#[test] +fn the_program_s_end_reaches_the_runner_before_its_timeout_while_its_launch_report_is_late() { + // The program writes a word and exits 4 at once, but the test holds a copy of its launch + // report's writer, as a process that another thread forked while the program was spawning + // would, so the report arrives only once the program has ended and the first calls, which + // come after the runner's timeout, have taken every step that they could. The program's end + // reaches the runner before the timeout fires all the same, whoever collects its status, and + // the runner holds the program's start and output until the report arrives: the output + // arrives whole, and the outcome names no cause. + const TIMEOUT: Duration = Duration::from_secs(20); + let limits = limits(4096); + for caller_reaps in [false, true] { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let mut options = Options::new().cgroup(&cgroup, Scope::Execution); + if caller_reaps { + options = options.caller_reaps(); + } + let (pipes, ends) = Pipes::new(false).unwrap(); + let (spawned, writer) = shell("printf hello; exit 4") + .stdin(OwnedFd::from(std::fs::File::open("/dev/null").unwrap())) + .stdout(ends.stdout) + .stderr(ends.stderr) + .new_process_group() + .spawn_holding_report() + .unwrap(); + let execution = Execution::new(spawned, pipes, options.process_group()).unwrap(); + let mut side = Side::bound( + execution, + limits, + Timers::new(GRACE, TIMEOUT * 3).timeout(TIMEOUT), + ); + // Only the runner has the timeout. + side.peer = Peer::new(KEY, limits, timers(), Instant::now()); + let late = Instant::now() + TIMEOUT; + let id = side.binding.id(); + let process = ProcessId::from_positive(i64::from(id)).unwrap(); + let started = Instant::now(); + while !sys::has_exited(process).unwrap() { + assert!(started.elapsed() < PATIENCE, "the program ended in time"); + std::thread::sleep(Duration::from_millis(1)); + } + if caller_reaps { + let status = Child::new(id).unwrap().wait().unwrap(); + assert!(side.binding.observe(id, status)); + } + let mut buffer = [0u8; 64]; + while side.pump(late, &mut buffer) {} + assert!(!side.peer.has_started(), "{caller_reaps}"); + assert!(side.output[0].is_empty(), "{caller_reaps}"); + assert!(side.terminal.is_none(), "{caller_reaps}"); + drop(writer); + let patience = Instant::now() + PATIENCE; + while !side.is_finished() { + assert!(Instant::now() < patience, "the execution finished in time"); + if !side.pump(late, &mut buffer) { + wait(&[&mut side], Instant::now() + Duration::from_millis(10)); + } + } + let terminal = side.terminal.unwrap(); + assert_eq!(terminal.status(), Some(Status::Exited(4)), "{caller_reaps}"); + assert_eq!(terminal.cause(), None, "{caller_reaps}"); + assert!(!terminal.is_truncated(Stream::Stdout), "{caller_reaps}"); + assert!(terminal.is_reusable(), "{caller_reaps}"); + assert_eq!(side.output[0], b"hello", "{caller_reaps}"); + assert!(side.failures.is_empty(), "{:?}", side.failures); + } +} + +#[test] +fn the_runner_gives_up_in_time_on_a_launch_report_that_a_process_holds_after_the_program_s_end() { + // The program writes a word and exits 4 at once, but the test holds a copy of its launch + // report's writer throughout, as a process that another thread forked while the program was + // spawning would if it neither replaced itself with its program nor ended, whether the + // execution settles or, without a cgroup, does not. Only the runner has a timeout, and the + // first calls come after it. The runner holds the program's start and output until the + // settlement time and the grace have passed since the program's end, then gives up on the + // report: the output arrives whole, without a start, and the outcome keeps the program's + // status, names no cause, and is fenced. + const TIMEOUT: Duration = Duration::from_secs(20); + let limits = limits(4096); + for settles in [true, false] { + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let options = if settles { + Options::new().cgroup(&cgroup, Scope::Execution) + } else { + Options::new() + }; + let (pipes, ends) = Pipes::new(false).unwrap(); + let (spawned, writer) = shell("printf hello; exit 4") + .stdin(OwnedFd::from(std::fs::File::open("/dev/null").unwrap())) + .stdout(ends.stdout) + .stderr(ends.stderr) + .new_process_group() + .spawn_holding_report() + .unwrap(); + let execution = Execution::new(spawned, pipes, options.process_group()).unwrap(); + let mut side = Side::bound( + execution, + limits, + Timers::new(GRACE, SETTLEMENT).timeout(TIMEOUT), + ); + // Only the runner has the timeout. + side.peer = Peer::new(KEY, limits, timers(), Instant::now()); + let late = Instant::now() + TIMEOUT; + let id = side.binding.id(); + let process = ProcessId::from_positive(i64::from(id)).unwrap(); + let started = Instant::now(); + while !sys::has_exited(process).unwrap() { + assert!(started.elapsed() < PATIENCE, "the program ended in time"); + std::thread::sleep(Duration::from_millis(1)); + } + let mut buffer = [0u8; 64]; + // A process that another test's thread forked while the program was spawning may hold + // the program's output pipes for a moment too, and the runner gives up on the streams + // still open at the instant at which it gives up on the report, so the test waits for + // both streams to close first. + let patience = Instant::now() + PATIENCE; + while Stream::ALL + .into_iter() + .any(|stream| side.binding.runner().read_allowance(stream) > 0) + { + assert!( + Instant::now() < patience, + "the program's streams closed in time" + ); + if !side.pump(late, &mut buffer) { + wait(&[&mut side], Instant::now() + Duration::from_millis(10)); + } + } + while side.pump(late, &mut buffer) {} + let given_up = late + SETTLEMENT + GRACE; + assert_eq!(side.binding.deadline(), Some(given_up), "{settles}"); + while side.pump(given_up - Duration::from_millis(1), &mut buffer) {} + assert!(!side.peer.has_started(), "{settles}"); + assert!(side.output[0].is_empty(), "{settles}"); + assert!(side.terminal.is_none(), "{settles}"); + let patience = Instant::now() + PATIENCE; + while !side.is_finished() { + assert!(Instant::now() < patience, "the execution finished in time"); + if !side.pump(given_up, &mut buffer) { + wait(&[&mut side], Instant::now() + Duration::from_millis(10)); + } + } + let terminal = side.terminal.unwrap(); + assert_eq!(terminal.status(), Some(Status::Exited(4)), "{settles}"); + assert_eq!(terminal.cause(), None, "{settles}"); + assert_eq!(terminal.launch_failure(), None, "{settles}"); + assert!(!terminal.is_truncated(Stream::Stdout), "{settles}"); + assert!(!terminal.is_reusable(), "{settles}"); + assert_eq!(side.output[0], b"hello", "{settles}"); + assert!(!side.peer.has_started(), "{settles}"); + assert!(side.failures.is_empty(), "{:?}", side.failures); + drop(writer); + } +} + +/// How the liveness harness ends the stalled execution. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Ending { + /// The peer's caller cancels it. + Cancel, + /// The runner's timeout passes. + Timeout, + /// The session is reset. + Reset, +} + +#[test] +fn a_stalled_sink_stalls_only_its_own_execution_and_endings_still_finish_in_time() { + const TIMEOUT: Duration = Duration::from_secs(2); + for ending in [Ending::Cancel, Ending::Timeout, Ending::Reset] { + let temporary = tempfile::tempdir().unwrap(); + let stalled_cgroup = fake_cgroup(&temporary.path().join("stalled"), "0\n"); + let other_cgroup = fake_cgroup(&temporary.path().join("other"), "0\n"); + let stalled_timers = match ending { + Ending::Timeout => timers().timeout(TIMEOUT), + _ => timers(), + }; + let mut stalled = Side::start( + shell("exec head -c 16777216 /dev/zero"), + false, + Options::new().cgroup(&stalled_cgroup, Scope::Execution), + limits(64 * 1024), + stalled_timers, + ); + let created = Instant::now(); + stalled.stalled = true; + // The stalled sink exhausts its window, and its program blocks on its full pipe. + drive(&mut [&mut stalled], |sides| { + sides[0].binding.runner().read_allowance(Stream::Stdout) == 0 + && sides[0].peer.buffered() == 64 * 1024 + }); + // Another execution streams meanwhile, and keeps streaming after a MiB. + let mut other = Side::start( + shell("seq 1 600000"), + false, + Options::new().cgroup(&other_cgroup, Scope::Execution), + limits(64 * 1024), + timers(), + ); + drive(&mut [&mut stalled, &mut other], |sides| { + sides[1].output[0].len() > 1 << 20 + }); + assert!(stalled.binding.execution().status().is_none(), "{ending:?}"); + assert!(stalled.output[0].is_empty()); + assert_eq!(stalled.peer.buffered(), 64 * 1024, "{ending:?}"); + let streamed = other.output[0].len(); + let ended = match ending { + Ending::Cancel => { + let now = Instant::now(); + stalled.peer.cancel(now); + // The cancellation reaches the runner at once, so that both sides' graces end + // together: a runner whose grace outlasted the peer's would take the window that + // the peer's ended grace frees, and could send the rest without a loss of its own. + stalled.serve(now, &mut [0; 64]); + now + } + Ending::Timeout => created + TIMEOUT, + Ending::Reset => { + let now = Instant::now(); + stalled.binding.reset(now); + stalled.peer.reset(now); + now + } + }; + // The stalled execution ends, while the other one streams on. + let mut finished = None; + drive(&mut [&mut stalled, &mut other], |sides| { + if sides[0].is_finished() { + finished.get_or_insert_with(Instant::now); + } + finished.is_some() && sides[1].output[0].len() > streamed + }); + // Each side finished within its grace, and the peer, which learns of the timeout only + // from the terminal outcome, within a second one. + let finished = finished.unwrap(); + let grace = match ending { + Ending::Timeout => GRACE + GRACE, + _ => GRACE, + }; + assert!( + finished <= ended + grace + SLACK, + "{ending:?} took {:?}", + finished - ended + ); + let cause = match ending { + Ending::Cancel => Cause::Cancelled, + Ending::Timeout => Cause::TimedOut, + Ending::Reset => Cause::Reset, + }; + let terminal = stalled.terminal.unwrap(); + assert_eq!(terminal.cause(), Some(cause), "{ending:?}"); + assert_eq!(terminal.status(), Some(Status::Signaled(9)), "{ending:?}"); + assert!(terminal.is_truncated(Stream::Stdout), "{ending:?}"); + assert!(terminal.is_reusable(), "{ending:?}"); + let outcome = stalled.outcome().unwrap(); + assert_eq!(outcome.cause(), Some(cause), "{ending:?}"); + assert!(outcome.is_truncated(Stream::Stdout), "{ending:?}"); + // The other execution still finishes whole. + let outcome = finish(&mut other); + assert_eq!(other.output[0], sequence(600_000), "{ending:?}"); + assert!(!outcome.is_truncated(Stream::Stdout)); + } +} + +#[test] +fn a_call_takes_a_bounded_number_of_steps_and_asks_to_be_called_again() { + // The binding reads at most a window and a chunk, 20 KiB, of the program's 24 KiB, so the + // rest, 4 KiB, which any pipe holds, is left in the pipe once the program wrote everything. + // The session is reset then, and the binding discards the rest with a buffer of 16 bytes: + // in more steps than one call takes. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let written = temporary.path().join("written"); + let mut side = Side::start( + shell(&format!( + "head -c 24576 /dev/zero; : >'{}'; exec sleep 60", + written.display() + )), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(4096), + timers(), + ); + side.stalled = true; + drive(&mut [&mut side], |sides| { + sides[0].peer.buffered() == 4096 + && sides[0].binding.runner().read_allowance(Stream::Stdout) == 0 + }); + // The rest fits in the pipe, so the program writes it without another read. + let started = Instant::now(); + while !written.exists() { + assert!(started.elapsed() < PATIENCE, "the program wrote in time"); + std::thread::sleep(Duration::from_millis(1)); + } + let now = Instant::now(); + side.binding.reset(now); + side.peer.reset(now); + assert!(side.binding.next(now, &mut [0; 16]).is_none()); + assert_eq!(side.binding.deadline(), Some(now)); + let outcome = finish(&mut side); + assert_eq!(outcome.cause(), Some(Cause::Reset)); + let terminal = side.terminal.unwrap(); + assert_eq!(terminal.cause(), Some(Cause::Reset)); + assert!(terminal.is_truncated(Stream::Stdout)); + assert!(terminal.is_reusable()); +} + +#[test] +fn a_call_writes_a_bounded_number_of_chunks_of_input_and_asks_to_be_called_again() { + // The runner holds a whole window of input, 8 KiB in chunks of 128 bytes, before the binding + // writes any of it, and the program reads all of it. Each call writes at most a chunk at a + // time, and at most `STEPS` times, so 2 KiB, where a pipe, which takes a page at least, + // would take 4 KiB or more at once. A call that stops at that bound with input left sets the + // deadline to its own instant. + const CHUNK: usize = 128; + const WINDOW: usize = 8192; + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let window = Window::bytes(u32::try_from(WINDOW).unwrap()); + let chunk = u32::try_from(CHUNK).unwrap(); + let mut side = Side::start( + shell("exec cat >/dev/null"), + true, + Options::new().cgroup(&cgroup, Scope::Execution), + Limits::new(chunk, window, Window::bytes(4096)).unwrap(), + timers(), + ); + side.input = vec![b'i'; WINDOW]; + let mut buffer = [0u8; 4096]; + let now = Instant::now(); + while let Some(event) = side.binding.next(now, &mut buffer) { + side.handle(now, event, &buffer); + } + while side.sent < WINDOW { + assert!( + side.serve(now, &mut buffer), + "the peer hands its input over" + ); + } + assert_eq!(side.binding.runner().buffered(), WINDOW); + let mut paused = false; + let patience = Instant::now() + PATIENCE; + while side.binding.runner().buffered() > 0 { + assert!( + Instant::now() < patience, + "the program took the input in time" + ); + let before = side.binding.runner().buffered(); + let now = Instant::now(); + let event = side.binding.next(now, &mut buffer); + let written = before - side.binding.runner().buffered(); + assert!(written <= STEPS * CHUNK, "one call wrote {written} bytes"); + match event { + Some(event) => side.handle(now, event, &buffer), + // A pipe takes a write of a chunk, which is atomic, whole or not at all, so a call + // that wrote this much wrote `STEPS` times. + None if written == STEPS * CHUNK && side.binding.runner().buffered() > 0 => { + assert_eq!(side.binding.deadline(), Some(now)); + paused = true; + } + None => wait(&[&mut side], patience), + } + } + assert!(paused, "a call stopped at the bound with input left"); + let outcome = finish(&mut side); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(outcome.is_reusable()); +} + +#[test] +fn output_flows_while_the_caller_hands_over_input_before_every_call() { + // The program writes its output once the test lets it, which waits in its pipe, then reads + // its input to its end. Once the peer knows that the program started, which its output + // waits for, the caller takes one event of the binding at a time, and before each, hands the + // binding the input that the peer's credit allows, which the program's pipe takes at once, + // so that every call frees credit. The source holds a page of input, which any pipe takes + // whole, so no write waits for the program to read: the output must arrive before the + // source runs out. + const INPUT: usize = 4096; + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let go = temporary.path().join("go"); + let written = temporary.path().join("written"); + let mut side = Side::start( + shell(&format!( + "while [ ! -e '{}' ]; do sleep 0.01; done; printf ready; : >'{}'; \ + exec cat >/dev/null", + go.display(), + written.display() + )), + true, + Options::new().cgroup(&cgroup, Scope::Execution), + Limits::new(16, Window::bytes(64), Window::bytes(4096)).unwrap(), + timers(), + ); + let mut buffer = [0u8; 4096]; + let patience = Instant::now() + PATIENCE; + while !side.peer.has_started() { + assert!(Instant::now() < patience, "the program started in time"); + let now = Instant::now(); + match side.binding.next(now, &mut buffer) { + Some(event) => side.handle(now, event, &buffer), + None => wait(&[&mut side], patience), + } + } + std::fs::write(&go, b"").unwrap(); + while !written.exists() { + assert!(Instant::now() < patience, "the program wrote in time"); + std::thread::sleep(Duration::from_millis(1)); + } + side.input = vec![b'i'; INPUT]; + while side.output[0].is_empty() { + assert!( + side.sent < INPUT, + "the output arrived while the input flowed" + ); + let now = Instant::now(); + side.serve(now, &mut buffer); + if let Some(event) = side.binding.next(now, &mut buffer) { + side.handle(now, event, &buffer); + } + } + assert!(side.sent > 0); + // The source ends its input there, and the program ends with it. + side.input.truncate(side.sent); + let outcome = finish(&mut side); + assert_eq!(side.output[0], b"ready"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(outcome.is_reusable()); +} + +/// A process that escaped its execution, which the test kills when it ends, however it ends, +/// unless the process ended already. +struct Stray((ProcessId, sys::FileDescriptor)); + +impl Drop for Stray { + /// Kills the process unless its pidfd says that it ended. + fn drop(&mut self) { + let (process, pidfd) = &self.0; + if sys::readable_or_hung_up(&[pidfd.borrowed()], 0).is_ok_and(|ended| ended == [false]) { + let _ = sys::kill_process(*process); + } + } +} + +/// Waits until the program names its descendant in `file`, by its ID on one line, and returns +/// the descendant. +fn stray(file: &Path) -> Stray { + let started = Instant::now(); + loop { + if let Ok(line) = std::fs::read(file) + && line.ends_with(b"\n") + { + return Stray(descendant(&line)); + } + assert!( + started.elapsed() < PATIENCE, + "the program named its descendant in time" + ); + std::thread::sleep(Duration::from_millis(1)); + } +} + +#[test] +fn a_pipe_writer_that_escapes_the_kill_fences_its_execution_in_time_and_holds_no_other() { + // The fake cgroup kills nothing: the program's descendant, in a session of its own, writes + // to the program's standard output as fast as it can, and outlives the program's + // cancellation, as a process that escaped the cgroup would. Its sink takes nothing, so its + // execution discards what it writes once the grace ends, and gives up once the settlement + // time after the grace ends too. + let temporary = tempfile::tempdir().unwrap(); + let escaped_cgroup = fake_cgroup(&temporary.path().join("escaped"), "0\n"); + let other_cgroup = fake_cgroup(&temporary.path().join("other"), "0\n"); + let writer = temporary.path().join("writer"); + let settlement = Duration::from_millis(500); + let mut escaped = Side::start( + shell(&format!( + "setsid cat /dev/zero 2>/dev/null & echo $! >'{}'; exec sleep 60", + writer.display() + )), + false, + Options::new().cgroup(&escaped_cgroup, Scope::Execution), + limits(64 * 1024), + Timers::new(GRACE, settlement), + ); + escaped.stalled = true; + let _stray = stray(&writer); + drive(&mut [&mut escaped], |sides| { + sides[0].peer.buffered() == 64 * 1024 + }); + // Another execution streams meanwhile, and keeps streaming after a MiB. + let mut other = Side::start( + shell("seq 1 600000"), + false, + Options::new().cgroup(&other_cgroup, Scope::Execution), + limits(64 * 1024), + timers(), + ); + drive(&mut [&mut escaped, &mut other], |sides| { + sides[1].output[0].len() > 1 << 20 + }); + let streamed = other.output[0].len(); + let cancelled = Instant::now(); + escaped.peer.cancel(cancelled); + let mut finished = None; + drive(&mut [&mut escaped, &mut other], |sides| { + if sides[0].is_finished() { + finished.get_or_insert_with(Instant::now); + } + finished.is_some() && sides[1].output[0].len() > streamed + }); + let finished = finished.unwrap(); + assert!( + finished <= cancelled + GRACE + settlement + SLACK, + "took {:?}", + finished - cancelled + ); + // The program ended and its cgroup settled, but its output never did, so the execution is + // fenced, and its standard output truncated. + let terminal = escaped.terminal.unwrap(); + assert_eq!(terminal.cause(), Some(Cause::Cancelled)); + assert_eq!(terminal.status(), Some(Status::Signaled(9))); + assert!(terminal.is_truncated(Stream::Stdout)); + assert!(!terminal.is_reusable()); + assert!(!escaped.outcome().unwrap().is_reusable()); + // The other execution still finishes whole. + let outcome = finish(&mut other); + assert_eq!(other.output[0], sequence(600_000)); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(outcome.is_reusable()); +} + +#[test] +fn an_execution_that_does_not_settle_gives_up_in_time_on_the_output_that_a_descendant_holds() { + // The program ends at once, while a descendant in a session of its own holds its standard + // output, which nothing ends: neither the fake cgroup, which kills nothing and always counts + // a process, nor the execution without a cgroup, whose settlement fails at once. The + // execution does not settle, so the runner gives up on the stream the settlement time and + // the grace after the program's end, and still sends what it read. + let settlement = Duration::from_millis(300); + let temporary = tempfile::tempdir().unwrap(); + let occupied = fake_cgroup(temporary.path(), "1\n"); + for cgroup in [Some(&occupied), None] { + let options = match cgroup { + Some(cgroup) => Options::new().cgroup(cgroup, Scope::Execution), + None => Options::new(), + }; + let mut side = Side::start( + shell("setsid sleep 60 2>/dev/null & echo $!"), + false, + options, + limits(16 * 1024), + Timers::new(GRACE, settlement), + ); + drive(&mut [&mut side], |sides| { + sides[0].output[0].ends_with(b"\n") && sides[0].binding.execution().status().is_some() + }); + let exited = Instant::now(); + let _holder = Stray(descendant(&side.output[0])); + let outcome = finish(&mut side); + let finished = Instant::now(); + assert!( + finished <= exited + settlement + GRACE + SLACK, + "took {:?}", + finished - exited + ); + let terminal = side.terminal.unwrap(); + assert_eq!(outcome, terminal); + assert_eq!(terminal.status(), Some(Status::Exited(0))); + assert_eq!(terminal.cause(), None); + assert!( + terminal.is_truncated(Stream::Stdout), + "with a cgroup: {}", + cgroup.is_some() + ); + assert!(!terminal.is_truncated(Stream::Stderr)); + assert!(!terminal.is_reusable()); + assert!(side.failures.is_empty(), "{:?}", side.failures); + } +} + +#[test] +fn an_execution_that_does_not_settle_loses_nothing_of_the_output_that_no_process_holds() { + // The program leaves nothing behind, and ends once its output fits in the window of its + // stalled sink, the runner's queue, and its pipe, which holds the rest, about 1 KiB, as any + // pipe does, but the execution does not settle: the fake cgroup always counts a process, and + // the execution without a cgroup fails its settlement at once. Its streams hung up, so the + // runner waits for the sink long after the settlement time and the grace, and loses nothing. + let settlement = Duration::from_millis(100); + let temporary = tempfile::tempdir().unwrap(); + let occupied = fake_cgroup(temporary.path(), "1\n"); + for cgroup in [Some(&occupied), None] { + let options = match cgroup { + Some(cgroup) => Options::new().cgroup(cgroup, Scope::Execution), + None => Options::new(), + }; + let mut side = Side::start( + shell("seq 1 7000; seq 1 100 >&2"), + false, + options, + limits(16 * 1024), + Timers::new(GRACE, settlement), + ); + side.stalled = true; + drive(&mut [&mut side], |sides| { + sides[0].binding.execution().status().is_some() + }); + // Past the time at which the runner gives up on the streams that a process may hold. + drive_for(&mut [&mut side], (settlement + GRACE) * 2); + let with = cgroup.is_some(); + assert!(!side.is_finished(), "with a cgroup: {with}"); + side.stalled = false; + let outcome = finish(&mut side); + assert_eq!(side.output[0], sequence(7000), "with a cgroup: {with}"); + assert_eq!(side.output[1], sequence(100), "with a cgroup: {with}"); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert_eq!(outcome.cause(), None); + assert!( + !outcome.is_truncated(Stream::Stdout) && !outcome.is_truncated(Stream::Stderr), + "with a cgroup: {with}" + ); + assert!(!outcome.is_reusable()); + assert!(side.failures.is_empty(), "{:?}", side.failures); + } +} + +#[test] +fn a_due_settlement_check_runs_while_output_drains_slowly() { + // The program writes 32 KiB, which its pipe holds, and ends. The adapter takes 64 bytes at a + // time, and 2 ms to send each message, so that every call has output to return for longer + // than the settlement time after the program's end: the settlement's due checks still run, + // and the execution settles. + let temporary = tempfile::tempdir().unwrap(); + let cgroup = fake_cgroup(temporary.path(), "0\n"); + let settlement = Duration::from_millis(100); + let mut side = Side::start( + shell("head -c 32768 /dev/zero"), + false, + Options::new().cgroup(&cgroup, Scope::Execution), + limits(16 * 1024), + Timers::new(GRACE, settlement), + ); + let mut buffer = [0u8; 64]; + let started = Instant::now(); + let mut exited = None; + while !side.is_finished() { + assert!( + started.elapsed() < PATIENCE, + "the execution finished in time" + ); + let now = Instant::now(); + match side.binding.next(now, &mut buffer) { + Some(event) => { + side.handle(now, event, &buffer); + std::thread::sleep(Duration::from_millis(2)); + } + None => wait(&[&mut side], started + PATIENCE), + } + side.serve(now, &mut buffer); + if exited.is_none() && side.binding.execution().status().is_some() { + exited = Some(Instant::now()); + } + } + // The output drained for longer than the settlement time after the program's end. + let drained = Instant::now() - exited.unwrap(); + assert!(drained > settlement, "drained in {drained:?}"); + assert_eq!(side.output[0], vec![0; 32768]); + let outcome = side.outcome().unwrap(); + assert_eq!(outcome.status(), Some(Status::Exited(0))); + assert!(outcome.is_reusable()); + assert!(!outcome.is_truncated(Stream::Stdout)); + assert!(side.failures.is_empty(), "{:?}", side.failures); +} diff --git a/nvx-rs/src/guest/exec/tests.rs b/nvx-rs/src/guest/exec/tests.rs index d51faa539..9b6ed5e2b 100644 --- a/nvx-rs/src/guest/exec/tests.rs +++ b/nvx-rs/src/guest/exec/tests.rs @@ -829,8 +829,11 @@ fn both_streams_take_turns_while_each_has_output() { Options::new(), ); let mut record = Record::default(); - record.drive_until(&mut execution, false, |_, execution| { - execution.status().is_some() + // The launch report may arrive after the status: a process that another test's thread forked + // while the program was spawning holds the report's writer until it replaces itself with its + // program or ends. + record.drive_until(&mut execution, false, |record, execution| { + record.launch.is_some() && execution.status().is_some() }); assert!(record.order.is_empty()); execution.grant(Stream::Stdout, usize::MAX); diff --git a/nvx-rs/src/guest/mod.rs b/nvx-rs/src/guest/mod.rs index 764d6d656..ca2da9034 100644 --- a/nvx-rs/src/guest/mod.rs +++ b/nvx-rs/src/guest/mod.rs @@ -5,9 +5,10 @@ //! of no other; [`cgroup`], through which it places, accounts, kills, and settles the processes of //! its executions; [`mappings`], through which it mounts host mappings beneath the descriptors of //! the roots that it names; [`exec`], which drives each program as one execution, with its -//! launch, its bounded output and input, its ending, and its outcome; [`root`], through which it -//! bootstraps the guest for a layered root, assembles the root, mounts live shares and mappings in -//! it, and tears it down; and [`gpt`], through which it reads the partition table of a GPT disk +//! launch, its bounded output and input, its ending, and its outcome, alone or, through its +//! [`binding`](exec::binding), under the [execution engine](crate::exec); [`root`], through which +//! it bootstraps the guest for a layered root, assembles the root, mounts live shares and mappings +//! in it, and tears it down; and [`gpt`], through which it reads the partition table of a GPT disk //! that the kernel attached as a block device. //! //! Each other module holds one program, and the crate's binary of the same name runs its `main`. diff --git a/nvx-rs/src/guest/spawn.rs b/nvx-rs/src/guest/spawn.rs index 059617dd5..f45f17521 100644 --- a/nvx-rs/src/guest/spawn.rs +++ b/nvx-rs/src/guest/spawn.rs @@ -328,6 +328,22 @@ impl Command { PreparedChild::new(self.spec).expect("the command prepares") } + /// Starts the child as [`spawn`](Self::spawn) does, and returns with it a copy of its launch + /// report's writer: the report arrives only once the copy closes too, as it does when a + /// process that another thread forked while the child was spawning holds the writer, for + /// focused tests. + #[cfg(test)] + pub(crate) fn spawn_holding_report(self) -> io::Result<(Spawned, OwnedFd)> { + let child = PreparedChild::new(self.spec)?; + let writer = child.report_writer_copy()?; + let (process, report) = child.spawn()?; + let spawned = Spawned { + process, + report: LaunchReport { reader: report }, + }; + Ok((spawned, writer.into())) + } + /// Replaces the system calls of the child's credential stage, which then succeed, or fail /// with `failure`, without privileges. #[cfg(test)] diff --git a/nvx-rs/src/guest/sys/mod.rs b/nvx-rs/src/guest/sys/mod.rs index 5c00e8c86..c0b008b25 100644 --- a/nvx-rs/src/guest/sys/mod.rs +++ b/nvx-rs/src/guest/sys/mod.rs @@ -96,7 +96,7 @@ pub(crate) use ioctl::{ pub(crate) use mapping::MemoryMapping; #[cfg(target_arch = "x86_64")] pub(crate) use msr::{MsrAccess, MsrFile}; -pub(crate) use poll::{Pollable, poll, readable_or_hung_up}; +pub(crate) use poll::{Pollable, hung_up, poll, readable_or_hung_up}; #[cfg(all(target_arch = "x86_64", test))] pub(crate) use port::PortBackend; pub(crate) use port::{PORT_DEVICE, Port}; diff --git a/nvx-rs/src/guest/sys/poll.rs b/nvx-rs/src/guest/sys/poll.rs index 561e49759..07e3423e4 100644 --- a/nvx-rs/src/guest/sys/poll.rs +++ b/nvx-rs/src/guest/sys/poll.rs @@ -27,6 +27,16 @@ pub(crate) fn readable_or_hung_up( .collect()) } +/// Returns whether `descriptor` hung up, without waiting: for the read end of a pipe, whether +/// every writer closed it, whatever bytes it still holds. +pub(crate) fn hung_up(descriptor: BorrowedFd<'_>) -> Result { + let mut polls = [PollFd::new(descriptor, PollFlags::empty())]; + nix::poll::poll(&mut polls, PollTimeout::ZERO)?; + Ok(polls[0] + .revents() + .is_some_and(|events| events.contains(PollFlags::POLLHUP))) +} + /// A descriptor owner that [`poll`] waits on for input. #[derive(Clone, Copy, Debug)] pub(crate) enum Pollable<'a> { diff --git a/nvx-rs/src/guest/sys/spawn.rs b/nvx-rs/src/guest/sys/spawn.rs index cffbc2e55..14057d15c 100644 --- a/nvx-rs/src/guest/sys/spawn.rs +++ b/nvx-rs/src/guest/sys/spawn.rs @@ -536,6 +536,19 @@ impl PreparedChild { pub(crate) fn has_cgroup(&self) -> bool { self.cgroup.is_some() } + + /// Returns a close-on-exec copy of the launch report's writer, which holds the report back + /// while it is open, as a process that another thread forks before the child does, for + /// focused tests. + #[cfg(test)] + pub(crate) fn report_writer_copy(&self) -> Result { + let copy = nix::fcntl::fcntl( + self.report_writer.borrowed(), + FcntlArg::F_DUPFD_CLOEXEC(FIRST_NONSTANDARD), + )?; + // SAFETY: `F_DUPFD_CLOEXEC` returned a new descriptor, which nothing else owns. + Ok(unsafe { FileDescriptor::from_raw(copy) }) + } } /// Returns `descriptor` with close-on-exec set and a number above the standard streams', diff --git a/nvx-rs/src/lib.rs b/nvx-rs/src/lib.rs index b0a6cbc6e..03fcb0eda 100644 --- a/nvx-rs/src/lib.rs +++ b/nvx-rs/src/lib.rs @@ -10,6 +10,11 @@ //! not take. It has no dependencies, so the host, the guest, and test doubles share it. //! - [`contract`]: the control-contract versions that this crate speaks, and the compatibility of //! NVX releases with them. +//! - [`exec`]: the credit and stdio engine of an execution, which the side that runs a workload +//! and the side that requested it share with any adapter, whatever its framing: the state of +//! each stream, the credit that bounds the bytes in flight, the order of each stream's data and +//! end, and the execution's outcome and settlement. It needs only the standard library, so +//! every configuration has it. //! - [`net`]: IP networks in CIDR notation and their subtraction, and the forms of a VM's //! hostname and guest network address. //! - [`gpt`]: GUID Partition Table disks whose first partition configures an adapter and whose @@ -33,11 +38,12 @@ //! (`guest::process`), places, kills, and settles the processes of its executions in cgroups //! (`guest::cgroup`), mounts host mappings beneath the descriptors of the roots that it names //! (`guest::mappings`), drives each program as one execution, with its launch, bounded -//! output, input, ending, and outcome (`guest::exec`), bootstraps the guest for a layered root, -//! assembles the root, mounts live shares and mappings in it, and tears it down (`guest::root`), -//! and reads the partition table of a GPT disk's block device (`guest::gpt`), and the typed -//! calls through which it runs the time ABI's steps (`guest::time`) and reseeds after a restore -//! (`guest::reseed`) in its own process. The crate's `nvx-*` guest binaries run the programs. +//! output, input, ending, and outcome (`guest::exec`), alone or under the execution engine +//! (`guest::exec::binding`), bootstraps the guest for a layered root, assembles the root, mounts +//! live shares and mappings in it, and tears it down (`guest::root`), and reads the partition +//! table of a GPT disk's block device (`guest::gpt`), and the typed calls through which it runs +//! the time ABI's steps (`guest::time`) and reseeds after a restore (`guest::reseed`) in its own +//! process. The crate's `nvx-*` guest binaries run the programs. //! //! # Example //! @@ -110,6 +116,7 @@ pub mod contract; mod error; +pub mod exec; pub mod gpt; mod hypervisor; pub mod net;