KeithRaghubar · Oct 26, 2026 · Oct 26, 2026 · Oct 26, 2026 · Oct 26, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -56,3 +56,22 @@ jobs:
 
       - name: End-to-end scenarios
         run: make e2e E2E_HERE=1
+
+  # Tests against the live Arch mirrors (3.5.0-B146): a canary for a repo
+  # change the suite's stubs cannot see, such as clang 23.1.3-4 splitting out
+  # clang-libs. In CI_JOBS, like e2e. llvm-libs brings the stock compiler's
+  # runtime libraries, and is the host copy the test checks it does not load.
+  live:
+    runs-on: ubuntu-latest
+    container: docker.io/library/archlinux:base-devel
+    steps:
+      - name: Install git, make and llvm-libs
+        run: pacman -Syu --noconfirm --needed git make llvm-libs
+
+      - uses: actions/checkout@v4
+
+      - name: Dev dependencies
+        run: make dev-deps-core PACMAN_FLAGS=--noconfirm
+
+      - name: Live mirror tests
+        run: make test-live
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -102,6 +102,10 @@ Shipped-file edits must pass `make check-shipped`; doc/design edits `make check-
   `verbs/registry.py` `COMMANDS`, directly or via its namespace's `VerbGroup`. §CLI Verb Framework.
 - **Dual-toolchain test parity**: logic branching on resolved compiler (gcc vs llvm) ships both a
   gcc-path and an llvm-path test in the same change.
+- **A fix whose defect crossed a boundary the unit suite stubs also ships a test in that boundary's
+  tier**: root/user, cross-command state or a real terminal → `make e2e`; what the live repos hold
+  (a package split, rename, file move) → `make test-live`. The PR names the tier, or says why unit
+  tests suffice. Rubric: §Release Process "Which tier a test belongs in".
 - **Log levels follow the rubric in `docs/design/12-logging.md`** (§Logging; standards row 25): `ui()`
   is the answer the user asked for, `warn()`/`info()`/`debug()` are narration gated behind `-v`/`-vv`/`-vvv`
   — `ui()` is not a "always show this" escape hatch. A new pipeline stage requests the

diff --git a/DESIGN.md b/DESIGN.md
diff --git a/Makefile b/Makefile
@@ -1,4 +1,4 @@
-.PHONY: all help surface check-surface install-hooks dev venv build install dev-install dev-uninstall clean distclean test test-x lint typecheck coverage coverage-ratchet coverage-ratchet-update man demo-gif readme-preview \
+.PHONY: all help surface check-surface install-hooks dev venv build install dev-install dev-uninstall clean distclean test test-x test-live lint typecheck coverage coverage-ratchet coverage-ratchet-update man demo-gif readme-preview \
         dev-deps dev-deps-core dev-deps-pkg dev-deps-container dev-deps-release dev-deps-docs dev-deps-list \
         lint-py lint-sh \
         bench-completions check-shipped check-personal check-standards check-bump check-standards-at next-id next-bump design check-design pre-release print-gates audit \
@@ -179,6 +179,12 @@ test: ## Run the full suite (ARGS= extra pytest args)
 test-x: ## Run the suite, stop on first failure
 	uv run --no-sync --with pytest --with tomlkit pytest -x $(ARGS)
 
+# Tests that read the live Arch mirrors (3.5.0-B146), marked `network` and
+# skipped by `make test`: a canary for repo changes the suite's stubs cannot
+# see, such as a package split. CI runs it as a job of its own (CI_JOBS).
+test-live: ## Run the live-mirror tests (downloads the stock toolchain packages)
+	SYSFORGE_LIVE_TESTS=1 uv run --no-sync --with pytest --with tomlkit pytest -m network $(ARGS)
+
 # Both linters. Shell is not a second-class language here: 12 of the repo's
 # scripts are the test/release tooling, where a quoting bug fails a release or
 # silently tests the wrong tree.
@@ -378,7 +384,7 @@ sync-config: ## Merge new shipped defaults into the live config dir
 # runs each in a job of its own (`make <job>` in .github/workflows/ci.yml), not
 # as a gate of tools/ci.sh (3.5.0-DEV21).
 GATES := lint typecheck test check-shipped check-personal check-design check-roadmap-table check-standards check-surface
-CI_JOBS := e2e
+CI_JOBS := e2e test-live
 
 pre-release: $(GATES) $(CI_JOBS) ## Composite gate: lint, types, tests, all checks, end-to-end tier
 

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -112,6 +112,17 @@ canonical ordering.
 <!-- BEGIN roadmap-table (generated by tools/gen_roadmap_table.py — do not edit by hand; run `make roadmap-table`) -->
 | ID | Item | Priority | Effort | Bump |
 |----|------|----------|--------|------|
+| `3.5.0-DEV27` | an end-to-end scenario for update --devel on a -git package | high | medium | patch |
+| `3.5.0-DEV25` | an end-to-end scenario for the toolchain → kernel handoff | med | small | patch |
+| `3.5.0-DEV28` | an end-to-end scenario for a stock package installed over a sysforge build | med | small | patch |
+| `3.5.0-DEV30` | an end-to-end scenario for config merge | med | small | patch |
+| `3.5.0-DEV31` | an end-to-end scenario for build --interactive and update --interactive | med | small | patch |
+| `3.5.0-DEV29` | the build sandbox end to end, in the VM tier | med | large | patch |
+| `3.5.0-DEV33` | an end-to-end scenario for unattended runs | low | small | patch |
+| `3.5.0-DEV34` | an end-to-end scenario for the --frozen and --thaw flags | low | small | patch |
+| `3.5.0-Q1` | should sysforge measure a kernel built with its toolchain against one built with stock clang? | low | small | patch |
+| `3.5.0-DEV26` | a release-time run of the real toolchain → kernel chain on real hardware | low | medium | patch |
+| `3.5.0-DEV32` | an end-to-end scenario for package PGO (build --pgo) | low | medium | patch |
 <!-- END roadmap-table -->
 
 ### Features
@@ -120,4 +131,145 @@ canonical ordering.
 
 ### Developer tooling
 
+- **`3.5.0-DEV25` — an end-to-end scenario for the toolchain → kernel handoff.** `95-stage-dry-run`
+  previews `run toolchain` and `run kernel` separately, so nothing checks what one leaves for the
+  other: the compiler, its paths and its variant in `pipeline_state.toml`, which the kernel stage
+  (and `packages`) read. A scenario under `tools/e2e/scenarios/` runs a real `run toolchain` with
+  `toolchain.toml compiler = "llvm"` and `skip_build = true` (it registers the container's clang
+  without building, in seconds), asserts the stage result it saved, then asserts `run kernel
+  --dry-run` plans with that compiler (`compiler: llvm (from …)`, `cc=/usr/bin/clang`,
+  `variant: stock_llvm`), and again after switching back to `compiler = "gcc"`. A real kernel build
+  in the container is a follow-up only if a spike shows it fits: a `tinyconfig` build is minutes,
+  but the stage's boot-readiness check expects mkinitcpio and a bootloader the container lacks
+  (`state boot-entries` is already excused in `tools/e2e/uncovered.toml` for that reason). Tier
+  rubric row "state one command leaves for the next" (§Release Process).
+  *Priority: med · Effort: small · Bump: patch* — the cross-stage contract behind every LLVM kernel
+  build is untested end to end; the scenario is seconds long.
+
+---
+
+- **`3.5.0-DEV26` — a release-time run of the real toolchain → kernel chain on real hardware.**
+  `DEV25` checks the handoff with a registered clang; what only a real build shows is that the kernel
+  is compiled by the toolchain sysforge just built and is recorded that way. A script under `tools/`
+  (and a Make target, with a step in `docs/RELEASE-CHECKLIST.md` for minor and major releases) runs
+  `run toolchain` then `run kernel` on the workstation, and asserts on provenance, never on speed:
+  the built `vmlinux`'s `.comment` section names the clang version `build_state.toml` recorded for
+  the toolchain; the kernel's `build_state` entry carries that toolchain's variant (`pgo_llvm`); and
+  `update --dry-run` reports no toolchain drift for it. Then the drift leg: with a stock clang
+  installed from the pacman cache, `update --dry-run` reports the kernel as drifted, and the script
+  puts the sysforge toolchain back from its cached archives (`run toolchain --install-only`) before
+  it exits, on success or failure. Not CI and not the VM tier: the last PGO toolchain build here took
+  about 3.2 hours, and the VM (4 cores, 4 GiB, `VM_DISK_SIZE` 40G) cannot link libLLVM, though its
+  real bootloader would suit a kernel-only leg if one is wanted later. Kernel half of the rubric row
+  "state one command leaves for the next" (§Release Process).
+  *Priority: low · Effort: medium · Bump: patch* — hours long and run by hand, so a release
+  checklist item; it catches a chain the CI scenario can only stand in for.
+
+---
+
+- **`3.5.0-DEV27` — an end-to-end scenario for `update --devel` on a `-git` package.** `update --devel` is
+  the most-run flag in the maintainer's own use and no scenario passes it; about ten 3.5.0 fixes were
+  VCS-package bugs (`3.5.0-B144` among the latest). A stand-in under `tools/e2e/pkgs/` named `*-git`,
+  whose `source=()` is a `git+file://` repository the scenario creates, keeps it off the network:
+  build it, then `update --devel` with no new commit reports it current and rebuilds nothing; after a
+  commit to that repository it reports the new `pkgver` and rebuilds; `build_state.toml` records the
+  commit it built. The AUR-repo half of `3.5.0-B144` (an AUR packaging repo fetched only when the AUR
+  says it changed) needs a real AUR package, so it belongs in `10-quickstart`'s networked run if it is
+  covered at all.
+  *Priority: high · Effort: medium · Bump: patch* — the maintainer runs this daily and it has the densest recent bug history of any untested path.
+
+---
+
+- **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** A
+  user's own `pacman -S` of a package sysforge built from source should demote its record to a plain
+  `pacman` marker on the next `update` (`primitives/install_reconcile.py`, through the
+  `sysforge-buildstate.hook` sentinel), and a toolchain package changed outside sysforge should leave
+  the toolchain reminder (`sysforge-toolchain.hook`). The hooks run inside every scenario, but nothing
+  asserts their effect, and the ordering between the hook's events and sysforge's own installs has
+  needed two fixes (`3.5.0-B64`, `3.5.0-B120`). The scenario builds `pkgs/tree` from source, runs
+  `sudo pacman -S tree`, then asserts `update` demotes it and does not rebuild it; then `sudo pacman -S
+  lld` and asserts the reminder appears once and the sentinel is consumed. Needs the mirrors, as
+  `60-revert-to-stock` already does.
+  *Priority: med · Effort: small · Bump: patch* — a documented behaviour with a bug history and no end-to-end check; one short scenario.
+
+---
+
+- **`3.5.0-DEV29` — the build sandbox end to end, in the VM tier.** `[security] sandbox_builds = true`
+  (clean chroot, local repo) is how every build on the maintainer's system runs, about ten 3.5.0
+  fixes were sandbox bugs, and no scenario enables it. It cannot run in the container tier:
+  `makechrootpkg` needs `systemd-nspawn`, which an unprivileged podman or Actions container cannot
+  start. The VM has full systemd, so the e2e scenarios gain a sandbox mode run there
+  (`tools/e2e/run.sh --here` inside the VM, with `sandbox_builds` on): at least an AUR build, a build
+  whose dependency is a sysforge build served from the local repo, and a `-git` build, asserting the
+  package came out of the chroot and that `build_state.toml` records the sandbox toolchain. A
+  release-checklist step, beside `make vm-test`.
+  *Priority: med · Effort: large · Bump: patch* — the default build path on the maintainer's system is untested end to end, but it needs VM plumbing.
+
+---
+
+- **`3.5.0-DEV30` — an end-to-end scenario for `config merge`.** `config merge` is excused in
+  `tools/e2e/uncovered.toml` because it "needs a .sfnew and an interactive merge tool", but the tool is
+  `sysforge.toml [ui] merge`, which a scenario can set to a script. The scenario leaves a `.sfnew`
+  beside an edited config (as a package upgrade would), sets `merge` to a script that takes the new
+  file's added key and keeps the user's edit, runs `config merge`, and asserts the merged file, that
+  the `.sfnew` is gone and that `config check` is clean. The excusal comes out of `uncovered.toml` in
+  the same change.
+  *Priority: med · Effort: small · Bump: patch* — a command the maintainer runs after upgrades, excused for a reason that does not hold.
+
+---
+
+- **`3.5.0-DEV31` — an end-to-end scenario for `build --interactive` and `update --interactive`.** `--interactive`
+  runs makepkg without `--noconfirm` and hands it the terminal, so pacman's own questions (a package
+  conflict, a provider choice) reach the user instead of failing under `--noconfirm` (`3.3.0-B6`). No
+  scenario passes it, and prompts and terminal handling account for about seventeen 3.5.0 fixes. A
+  stand-in package that conflicts with an installed one, built with `--interactive` and the conflict
+  question answered through `tools/e2e/answer.py`, asserts the swap happened; the same without the
+  flag asserts the build stops and installs nothing.
+  *Priority: med · Effort: small · Bump: patch* — a frequently used mode whose bug class is the largest after the two stages.
+
+---
+
+- **`3.5.0-DEV32` — an end-to-end scenario for package PGO (`build --pgo`).** `docs/guides/pgo.md`
+  documents an instrumented build, a training run and a rebuild with the profile; the e2e tier runs
+  none of it. A stand-in C package whose program has a hot path the training run exercises: `build
+  --pgo` (instrument), run it, rebuild with the profile, then assert the profile store holds its data,
+  `state profiles` lists it, and the second build used it (the recorded flags). Asserts on what was
+  recorded and used, not on speed.
+  *Priority: low · Effort: medium · Bump: patch* — a documented guide path, used occasionally; the stand-in package is the work.
+
+---
+
+- **`3.5.0-DEV33` — an end-to-end scenario for unattended runs.** `--unattended` and `update --yes`
+  (`3.5.0-F20`) are for timers and scripts: they must never block on a prompt and must refuse what
+  needs consent instead of assuming it. The scenario runs `update --unattended` with stdin closed and
+  no terminal, against a state that would prompt (a repo package awaiting opt-in, a PKGBUILD change
+  awaiting review), and asserts it exits within a timeout, installs nothing it needed consent for, and
+  says why; then `uninstall --yes` completes without a terminal.
+  *Priority: low · Effort: small · Bump: patch* — new and documented in the kernel guide; the maintainer runs no timers, so lower value here.
+
+---
+
+- **`3.5.0-DEV34` — an end-to-end scenario for the `--frozen` and `--thaw` flags.** `90-source-policy`
+  covers `freeze_sources` in config but not the flags `docs/guides/freeze.md` documents for a single
+  run. Add to it: `--frozen` refuses a clone with the config off, `--no-frozen` lets one through with
+  it on, and `--thaw` behaves as the guide says, each asserting no checkout appears when refused.
+  *Priority: low · Effort: small · Bump: patch* — documented flags on an already-covered command; a few lines in an existing scenario.
+
+---
+
 ### Open questions
+
+- **`3.5.0-Q1` — should sysforge measure a kernel built with its toolchain against one built with
+  stock clang?** `bench toolchain` times the compiler itself; a user who builds the toolchain to get
+  a faster kernel has no sysforge answer to whether the kernel got faster, or whether its build did.
+  The two readings are different features. Build time (the kernel compiled by each clang, interleaved
+  as the toolchain bench does) fits `bench`'s model: read-only, the stock clang unpacked to a temp
+  dir. Runtime (the kernel *running* faster) needs a reboot into each kernel and a workload, which is
+  outside anything sysforge does unattended. Either way it is a measurement with no pass/fail
+  threshold (a cutoff tuned on one machine means nothing on another), so it is not a test; the
+  provenance checks belong to `DEV26`. Resolve by choosing build time, runtime, or neither
+  (a measurement left to the user, as `kernel-bench.sh` is today).
+  *Priority: low · Effort: small · Bump: patch* — a decision, not an implementation; the effort is the
+  spike that informs it.
+
+---
diff --git a/docs/RELEASE-CHECKLIST.md b/docs/RELEASE-CHECKLIST.md
@@ -16,6 +16,7 @@ make design roadmap-table       # 2. regenerate, then commit anything that moved
 make ci-local                   # 3. every static gate + full suite, clean container
 make coverage-ratchet audit     # 3. the two checks no gate runs
 make e2e                                                                     # 4a
+make test-live                                                               # 4a
 make vm-pkg-stable && make container-smoke && make container-smoke-cachyos   # 4a
 make vm-snapshot && make vm-test && make vm-stop                             # 4b (+ git flavor)
 # 5. /release-notes skill
@@ -72,6 +73,7 @@ In more detail:
 | `typecheck` (pyright) | yes | yes | no | yes |
 | full test suite | yes (`test`) | yes (`test`) | yes (`coverage`) | no |
 | end-to-end tier (`e2e`) | yes | no (its own CI job) | no | no |
+| live-mirror tier (`test-live`) | yes | no (its own CI job) | no | no |
 | `check-shipped` (includes version markers, completions) | yes | yes | subset | yes |
 | `check-personal`, `check-design`, `check-roadmap-table`, `check-standards` | yes | yes | no | yes |
 | clean environment (not your workstation) | no | yes | no | no |
@@ -157,6 +159,16 @@ make e2e                        # tools/e2e/README.md
 
 - [ ] `make e2e` exits 0
 
+Then the live-mirror tier: the stock toolchain the bench compares against, fetched from the repos
+as they are today, so a repo-side change such as a package split shows up before users hit it.
+CI runs it in a `live` job; run it here for the release commit.
+
+```bash
+make test-live                  # tests/test_live_stock_toolchain.py
+```
+
+- [ ] `make test-live` exits 0
+
 The smokes then check the packaging itself.
 
 Seconds rather than a boot. The CachyOS arm is the one that actually exercises portability (repo/AUR