End-to-end tier, sentinel armed at the first change, terminal-model tests (3.5.0-DEV21, B138, DEV22)

.github/workflows/ci.yml

@@ -38,3 +38,21 @@ jobs:
 
       - name: Lint, suite and gates
         run: bash tools/ci.sh "$GITHUB_WORKSPACE"
+
+  # The end-to-end tier (3.5.0-DEV21): the same scenarios `make e2e` runs under
+  # podman, here directly in the job's fresh container, sharing one system. A
+  # `pre-release` prerequisite outside GATES, so listed in the Makefile's
+  # CI_JOBS; tests/test_ci_tooling.py holds the two to each other.
+  e2e:
+    runs-on: ubuntu-latest
+    container: docker.io/library/archlinux:base-devel
+    steps:
+      - name: Install git and make
+        run: pacman -Syu --noconfirm --needed git make
+
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: End-to-end scenarios
+        run: make e2e E2E_HERE=1

Makefile

@@ -7,7 +7,7 @@
         preflight release-major release-minor release-patch release-resume \
         vm-deps vm-image vm-boot vm-boot-gui vm-snapshot vm-loadvm vm-iso vm-monitor vm-console vm-savevm vm-snapshots vm-ssh vm-ssh-builder vm-stop vm-clean \
         vm-pkg-stable vm-pkg-git vm-pkg-all vm-install-stable vm-install-git vm-smoke vm-test \
-        container-build container-smoke container-smoke-cachyos container-shell container-clean \
+        container-build container-smoke container-smoke-cachyos container-shell container-clean e2e \
         ci-local
 
 # ---------------------------------------------------------------------------
@@ -42,7 +42,7 @@ DEV_DEPS_CORE      = python-pytest ruff shellcheck scdoc uv git base-devel zsh
 DEV_DEPS_PKG       = devtools
 # QEMU test VM: vm-boot, vm-snapshot, vm-console, vm-monitor (socat), …
 DEV_DEPS_VM        = qemu-desktop edk2-ovmf gtk-vnc socat
-# Container test tier: container-smoke, container-smoke-cachyos.
+# Container test tier: container-smoke, container-smoke-cachyos, e2e.
 DEV_DEPS_CONTAINER = podman
 # Release: GPG-signed commit/tag/tarball + the GitHub release upload.
 DEV_DEPS_RELEASE   = gnupg github-cli
@@ -374,10 +374,13 @@ sync-config: ## Merge new shipped defaults into the live config dir
 ##@ Release
 # The gate list, in run order, written once (3.5.0-DEV8): `pre-release`
 # depends on it and tools/ci.sh reads it through `print-gates`, so the release
-# gate and CI cannot drift apart.
+# gate and CI cannot drift apart. CI_JOBS are the rest of `pre-release`: CI
+# runs each in a job of its own (`make <job>` in .github/workflows/ci.yml), not
+# as a gate of tools/ci.sh (3.5.0-DEV21).
 GATES := lint typecheck test check-shipped check-personal check-design check-roadmap-table check-standards check-surface
+CI_JOBS := e2e
 
-pre-release: $(GATES) ## Composite gate: lint, types, tests, all checks
+pre-release: $(GATES) $(CI_JOBS) ## Composite gate: lint, types, tests, all checks, end-to-end tier
 
 print-gates: ## Print the gate list pre-release and CI run
 	@echo $(GATES)
@@ -647,6 +650,14 @@ container-shell: ## Open a shell in the test container
 container-clean: ## Remove the test container and image
 	./tools/container/harness.sh clean --distro=$(or $(DISTRO),arch)
 
+# End-to-end tier (3.5.0-DEV21): the documented paths run as an unprivileged
+# user on a fresh system, each scenario in its own container from the demo
+# image (tools/e2e/README.md). SCENARIO= runs only the named ones; E2E_COMMIT=
+# builds that revision instead of HEAD. E2E_HERE=1 runs them on this system
+# as root, without podman: CI's e2e job, in its own fresh container.
+e2e: ## Run the end-to-end scenarios in fresh Arch containers (SCENARIO=, needs podman)
+	E2E_COMMIT=$(E2E_COMMIT) bash tools/e2e/run.sh $(if $(E2E_HERE),--here) $(SCENARIO)
+
 # Replay the CI job (.github/workflows/ci.yml, 3.4.0-DEV1) against the working
 # tree, uncommitted edits included, in the same base image. The tree is mounted
 # read-only: tools/ci.sh copies it to an unprivileged user before running the

REUSE.toml

@@ -15,6 +15,7 @@ path = [
     "tools/vm/**",
     "tools/container/**",
     "tools/demo/**",
+    "tools/e2e/**",
     "**.md",
     "**.toml",
     "**.lock",

ROADMAP.md

@@ -111,46 +111,12 @@ canonical ordering.
 <!-- BEGIN roadmap-table (generated by tools/gen_roadmap_table.py — do not edit by hand; run `make roadmap-table`) -->
 | ID | Item | Priority | Effort | Bump |
 |----|------|----------|--------|------|
-| `3.5.0-DEV21` | an end-to-end test tier that runs the documented paths on a fresh system | high | large | patch |
-| `3.5.0-B138` | a verb that fails before installing anything leaves a stale sentinel | med | medium | patch |
 <!-- END roadmap-table -->
 
 ### Features
 
 ### Bugs
 
-- **`3.5.0-B138` — a verb that fails before installing anything leaves a stale sentinel.**
-  `verbs.runner.run_verb` wraps all of a `requires_sentinel` verb's `execute` in
-  `stage_sentinel.sentinel_scope`, which keeps the record on any exception. So `sysforge build`
-  failing on a missing `pkgctl` (a `RuntimeError` raised before any install) leaves
-  `stage = "build"` behind, and the next run refuses to start ("Stale stage-in-progress sentinel
-  present"): a recovery prompt at a terminal, a hard stop for a timer or cron run. Found by the
-  demo-recording container on 2026-10-23. Two shapes to choose between: arm the sentinel at the
-  first mutation (the install) rather than at `execute` entry, or move pre-mutation checks such
-  as the `pkgctl` probe into `pre_check`, which runs before the scope opens. The first fixes the
-  class; the second only the cases found so far.
-  *Priority: med · Effort: medium · Bump: patch* — blocks unattended runs after an ordinary,
-  harmless failure.
-
 ### Developer tooling
 
-- **`3.5.0-DEV21` — an end-to-end test tier that runs the documented paths on a fresh system.**
-  The unit suite stubs out exactly where six bugs found on 2026-10-23 lived: root versus the user
-  and group membership that only applies at the next login (`3.5.0-B134`, `B135`, `B137`), state one
-  command leaves for the next (`B136`, whose test called `mark_done` directly, a state the real
-  code never produced) and real terminal behaviour (`B132`, `B133`). The demo-recording container
-  (`tools/demo/`, `make demo-gif`) caught them by following the README. Add `make e2e`: shell
-  scenarios run as an unprivileged user in that same image, asserting on exit codes, `pacman -Qi`,
-  the state files and leftover sentinels (not vhs). Start with four: the Quick start as one login
-  session (setup, AUR build, `update`); a repo package's opt-in; flag drift after a profile change;
-  and a failure before install leaving no sentinel (`3.5.0-B138`'s regression test once fixed).
-  Prefer local PKGBUILDs, with at most one networked scenario, so outages don't flake the tier. CI
-  already runs in a fresh `archlinux:base-devel` container, so the same scripts can run there
-  directly without podman; wire `make e2e` into `pre-release` and `docs/RELEASE-CHECKLIST.md`. A
-  cheaper complement for the terminal class: a test helper that replays the progress bar's output
-  through a terminal model and asserts where the cursor ends up, under both xterm's and xterm.js's
-  saved-cursor rules, which would have caught B132 and B133 in the unit suite.
-  *Priority: high · Effort: large · Bump: patch* — the only tier that exercises privilege
-  boundaries, multi-command sequences and the documented first run, where users meet sysforge.
-
 ### Open questions

docs/RELEASE-CHECKLIST.md

@@ -15,6 +15,7 @@ make next-bump                  # 1. what level the release notes require
 make design roadmap-table       # 2. regenerate, then commit anything that moved
 make ci-local                   # 3. every static gate + full suite, clean container
 make coverage-ratchet audit     # 3. the two checks no gate runs
+make e2e                                                                     # 4a
 make vm-pkg-stable && make container-smoke && make container-smoke-cachyos   # 4a
 make vm-snapshot && make vm-test && make vm-stop                             # 4b (+ git flavor)
 # 5. /release-notes skill
@@ -70,6 +71,7 @@ In more detail:
 | `lint` (ruff + shellcheck) | yes | yes | yes | yes |
 | `typecheck` (pyright) | yes | yes | no | yes |
 | full test suite | yes (`test`) | yes (`test`) | yes (`coverage`) | no |
+| end-to-end tier (`e2e`) | yes | no (its own CI job) | no | no |
 | `check-shipped` (includes version markers, completions) | yes | yes | subset | yes |
 | `check-personal`, `check-design`, `check-roadmap-table`, `check-standards` | yes | yes | no | yes |
 | clean environment (not your workstation) | no | yes | no | no |
@@ -143,6 +145,20 @@ pass: fix it or record the tier as unrun.
 
 ### 4a. Container tier: fast, run first
 
+The end-to-end tier first: the documented paths (the Quick start, a repo package's opt-in, flag
+drift, a failed build's cleanup) run as an unprivileged user on a fresh system, one container per
+scenario. It builds sysforge from the checkout itself, so it needs no `vm-pkg-stable`. CI runs the
+same scenarios on every pull request; run it here for the release commit. An `XFAIL` is a known,
+filed bug and does not block; an `XPASS` fails the run (remove the scenario's `xfail` marker).
+
+```bash
+make e2e                        # tools/e2e/README.md
+```
+
+- [ ] `make e2e` exits 0
+
+The smokes then check the packaging itself.
+
 Seconds rather than a boot. The CachyOS arm is the one that actually exercises portability (repo/AUR
 shadowing, a different `makepkg.conf` baseline, bumped pkgrels on core packages).
 

docs/design/08-cli-verb-framework.md

@@ -75,16 +75,16 @@ spelling per meaning, across every verb:
 
 **Result types** (`PreCheckResult` and `ExecResult`) are plain dataclasses with `ctx` / `artifacts` dicts; the runner does not inspect their contents. This keeps phase boundaries loose enough for ad-hoc data flow within a verb without inventing a per-verb context class.
 
-**Sentinel handling.** Verbs whose `execute` mutates the live system set `requires_sentinel = True`. The runner wraps `execute + post_validate` in `sentinel_scope(state_dir, verb.name, recovery=…, retry_cmd=…, **metadata)` (`recovery` is a `stage_sentinel.Recovery` from `verb.sentinel_recovery`, stored as a kind plus arguments, `3.5.0-B43`) from `primitives/stage_sentinel.py`. On entry, the sentinel writes `stage_in_progress.toml`; on normal completion (both phases pass), it clears. On `RuntimeError` or `CleanExitRequested`, the sentinel is left in place so the next sysforge invocation blocks at the CLI-entry recovery prompt. `sentinel_scope` also installs an `InterruptScope`, so verbs participate in the same first-Ctrl-C-defers-to-safe-boundary behaviour as the toolchain stage: `build` and `update` stop before their next package (`interrupt.checkpoint()` in `build_core`), and a verb with no checkpoint stops when its `execute` + `post_validate` complete (`3.5.0-B44`). It additionally holds the `stage_in_progress.lock` liveness lock for the scope: if another run already holds it, entering the scope raises `RuntimeError` **before** `mark_started`, which the runner's existing handler converts to exit 1 without clobbering the live run's sentinel (§Pipeline layer, *Liveness guard*). The toolchain pipeline stage uses the same primitive — there is one implementation, shared.
+**Sentinel handling.** Verbs whose `execute` mutates the live system set `requires_sentinel = True`. The runner wraps `execute + post_validate` in `sentinel_scope(state_dir, verb.name, recovery=…, retry_cmd=…, **metadata)` (`recovery` is a `stage_sentinel.Recovery` from `verb.sentinel_recovery`, stored as a kind plus arguments, `3.5.0-B43`) from `primitives/stage_sentinel.py`, with `arm_on_mutation=True` (`3.5.0-B138`): the scope takes the liveness lock on entry but writes `stage_in_progress.toml` only at the first escalation that changes the system, which `privilege.privileged_argv` announces to `privilege.on_mutation` listeners (§Privilege seam). A verb's `execute` does its checks first (`build` probes for `pkgctl`, resolves targets, builds), and a failure there leaves nothing for the next run to refuse on; before the fix it left `stage = "build"` behind and blocked a timer or cron run. On normal completion (both phases pass), the record clears. On `RuntimeError` or `CleanExitRequested` after the record was written, it is left in place so the next sysforge invocation blocks at the CLI-entry recovery prompt; a Ctrl-C before it says nothing changed. The pipeline stages keep the default (record on entry), since they open their scope around the install itself. `sentinel_scope` also installs an `InterruptScope`, so verbs participate in the same first-Ctrl-C-defers-to-safe-boundary behaviour as the toolchain stage: `build` and `update` stop before their next package (`interrupt.checkpoint()` in `build_core`), and a verb with no checkpoint stops when its `execute` + `post_validate` complete (`3.5.0-B44`). It additionally holds the `stage_in_progress.lock` liveness lock for the scope: if another run already holds it, entering the scope raises `RuntimeError` **before** `mark_started`, which the runner's existing handler converts to exit 1 without clobbering the live run's sentinel (§Pipeline layer, *Liveness guard*). The toolchain pipeline stage uses the same primitive — there is one implementation, shared.
 
 **Read-only verbs** (`env`, `help`, `resolve`, `log`, `config check`, `state list`, `state orphans` without `--delete`, `state failed` without `--clear`/`--clear-all`, `state profiles` without `--delete`, `state boot-entries`, `bench toolchain`, `packages list`, `doctor` without `--apply`) implement `execute` (the work is printing) and return `ExecResult()`; `post_validate` defaults to no-op and `requires_sentinel = False`. They use the same dispatch path as mutating verbs — no second code path.
 
 **`state boot-entries`** (`3.3.0-F5`, `StateBootEntriesVerb`, read-only, `requires_sentinel = False`) lists every `vmlinuz-*` in `/boot` with the loader entry that boots it. The table goes through the pager like `state profiles` (`--no-pager` disables it): columns KERNEL / ENTRY / OWNER (`you`, `sysforge` or `none`) / TEMPLATE, plus `(missing)` rows for managed entries whose kernel is gone. Under `bootloader = grub`/`none` it prints a one-line note instead; with `boot_entries = "off"` it notes that and still lists. Rows come from `boot_entries.rows_for`.
 
 **Error model.**
-- `RuntimeError` raised from any phase → `_log.error(msg)`, return 1. Sentinel preserved if active.
+- `RuntimeError` raised from any phase → `_log.error(msg)`, return 1. Sentinel preserved if written (a change had begun).
 - `SystemExit` → propagate verbatim (lets tests and signal handlers see the raw exit).
-- `CleanExitRequested` → caught inside `sentinel_scope`, logged, re-raised as `RuntimeError` with the verb's `retry_cmd` and recovery command in the message; sentinel preserved.
+- `CleanExitRequested` → caught inside `sentinel_scope`, logged, re-raised as `RuntimeError` with the verb's `retry_cmd` and recovery command in the message; sentinel preserved. Before the first change, the message says nothing changed and no record is left.
 
 **Per-verb phase mapping** (current shape; phases reflect where each piece of work lives, not which primitives it calls):
 

docs/design/09-primitives-layer.md

@@ -32,6 +32,8 @@ Bottom-anchored status line for batch operations (`[3/10] building · htop`). Du
 - **When the bar repaints** — `_draw()` is the sole writer. It builds the whole frame (any pending resize, region establishment or release, and the painted row, glyph-downgraded then width-truncated) as one string and emits it with a **single write**, so another thread's log line can land before or after a frame but never inside it; an unchanged frame writes nothing. A module lock guards mutate-then-draw in every public entry point and the ticker, and is never held across a blocking call or a yield body. `SIGWINCH` only sets `resize_pending` (no I/O, no lock — a signal can interrupt a paint holding both); the next frame applies it. **A resize release erases from the content cursor down (`ESC[J`), never the old absolute row** (`3.5.0-B4`): a taller window makes the emulator pull scrollback in and shift everything, cursor and bar alike, down by the rows gained, so the old row then holds a content line, and establishment's index would scroll the stale bar into the output. Order is what a resize preserves, and the bar is always below the cursor. The emulator also resets DECSTBM on resize, so `pty_runner` repaints before forwarding the first chunk after a `SIGWINCH` instead of waiting out its one-second throttle. Events (tick, phase, yield exit) draw immediately; **exactly one scheduled painter** keeps the clock moving between them, chosen by whether a raw byte stream is being forwarded to the terminal. Single-write frames are safe against line-oriented writers (`log.*`, `print`, TTY-inheriting pacman/git), but a raw forwarder can be mid-escape or mid-UTF-8. By default the painter is a daemon **ticker** thread, started on the first TTY draw and stopped by `shutdown()`/`clear()`, which redraws every second unless yielded or forwarding; exceptions are swallowed and traced. While `pty_runner` forwards makepkg's raw output it enters `forwarding_output()` and becomes the painter itself, calling `refresh()` at most once a second and only at a stream boundary (§`pty_runner.py`). Cost: one wakeup a second, and about one short write a second only while a tracker is past the elapsed floor. **`heartbeat(detail)`** (`3.2.0-B13`) only sets `detail` — the scheduled painter shows it. It exists because a whole package build sits inside a single `tick()`: a trace once showed a 379-second gap between paints. The detail replaces rather than appends and the next tick/note/resume/phase clears it.
 - **Lifecycle** — `cli.main` wraps `_main()` in `progress.session()`, whose exit calls `shutdown()` (stop the ticker, release the region). The session exits before the `KeyboardInterrupt`/`BrokenPipeError` handlers, so the region is gone before `aborted (Ctrl-C)` prints, and a plain `sys.exit` or `log.fatal` releases on the way out too. `log` does not import `ui`. The `atexit` hook remains as the backstop for importers that don't go through `cli.main`; `shutdown()` is idempotent.
 
+**Tested on a terminal model** (`3.5.0-DEV22`). `tests/test_progress.py` checks the sequences the bar writes; `tests/test_terminal_model.py` replays them, and `update`'s `_TERMINAL_RESET`, through `tests/term_model.py`, a VT model of the sequences sysforge emits that raises on any other, and asserts what the screen holds: the command line survives, output follows it in order, the bar leaves nothing behind, the next prompt lands after the last line. Each runs under both rules terminals have for a saved cursor (DECSC) across a scroll: xterm keeps the screen row, xterm.js (VS Code's terminal, vhs) follows the content line. `3.5.0-B132` and `B133` both turned on that difference, and each has a test replaying its pre-fix sequence to show the model reproduces it. A new escape sequence in the bar is taught to the model in the same change.
+
 **One tracker at a time.** A tracker opened while another is open is a no-op that yields `progress_hooks._NoOpTick` and logs at debug, leaving the outer untouched; under tests (`_strict_nesting`, set by an autouse `tests/conftest.py` fixture that also gives every test a fresh `_bar` and no ticker) it raises `RuntimeError("tracker '<inner>' opened inside '<outer>'")`. A sub-step inside an item uses `tick.note()`/`tick.resume()` instead. The functions that open trackers call `require_no_tracker(owner)` (same strict/lenient policy) first thing, so a mis-nested caller fails before any sync or resolution side effect — the build-⊂-update invariant invites routing more callers through `build_and_install`. A stale `tick` after its tracker exits is a no-op. Tracker owners:
 
 | Tracker | Owner | Runs under |