From 6967320a5ab89a238e2e28c4c967d4f927041469 Mon Sep 17 00:00:00 2001 From: Keith Raghubar Date: Mon, 26 Oct 2026 10:43:30 -0400 Subject: [PATCH 1/3] Bench fetches stock clang-libs and refuses an incomplete stock prefix (3.5.0-B146) Arch's clang 23.1.3-4 moved the builtin headers and libclang-cpp into a new clang-libs package. The bench fetched only clang and llvm-libs, so the stock compiler failed on after loading the host's (sysforge) libclang-cpp through the LD_LIBRARY_PATH fallback. - stock_pkg.resolve takes optional names, each looked up on its own since one unknown target fails a whole pacman -Sp call; a local-repo hit is refused - stock_pkg.check_complete requires libclang-cpp, libLLVM and the builtin headers in the prefix before the stock binary runs - design section and release note updated Co-Authored-By: Claude Opus 5.5 --- DESIGN.md | 2 +- docs/design/08-cli-verb-framework.md | 2 +- docs/release-notes/unreleased.md | 4 ++ sysforge/bench_core.py | 9 +++- sysforge/primitives/stock_pkg.py | 66 +++++++++++++++++++++----- tests/test_bench_cmd.py | 46 ++++++++++++++++--- tests/test_stock_pkg.py | 69 ++++++++++++++++++++++++++++ 7 files changed, 177 insertions(+), 21 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 38f814bc..06becc44 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1454,7 +1454,7 @@ Read-only lifecycle verb (`requires_sentinel = False`). One term is queried agai Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspect tier, `requires_sentinel = False`). `bench_cmd.py` is only the argparse surface; the engine is `bench_core.run_toolchain_bench`, the one home shared with `state profiles` (GAIN) and the toolchain stage (`3.4.0-F11`), because verb modules are peers and may not import each other (`test_verb_modules_do_not_import_siblings`, which now also catches the `from sysforge import x_cmd` form). It answers the project's central question for the toolchain: is the installed sysforge-built clang actually faster than the stock repo clang? `bench` is a namespace because other targets (mesa via a shader-compile replay, the kernel) are follow-ups. -**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs` comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x); the installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock `, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. +**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs`, plus `clang-libs` when the sync DB carries it, comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x). **A repo split (`3.5.0-B146`).** `LD_LIBRARY_PATH` only puts the prefix first: a library missing from it is loaded from the host instead. Arch's clang 23.1.3-4 moved the resource dir (builtin headers) and `libclang-cpp` into a new `clang-libs` package, so the stock compiler ran with the sysforge `libclang-cpp` and failed on `#include `. `clang-libs` is passed to `stock_pkg.resolve` as `optional`, looked up on its own because one unknown target fails a whole `pacman -Sp` call, so a sync DB from before the split still benches; an optional package served by the local repo is refused like a required one. Before the stock binary runs, `stock_pkg.check_complete` requires `libclang-cpp`, `libLLVM` and the builtin headers in the prefix and names whatever is missing, so a future split fails the bench instead of borrowing the host's files. The installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock `, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. **Report.** Through `ui()`: the corpus label and file count, each compiler's version, median and min–max, and `stock/sysforge: x`. When the versions differ a line says the ratio includes upstream changes; across a major-version gap there is no ratio, only the reason. There is **no verdict and no threshold**: a cutoff tuned on one host means nothing on another. diff --git a/docs/design/08-cli-verb-framework.md b/docs/design/08-cli-verb-framework.md index 23bf962d..4baba011 100644 --- a/docs/design/08-cli-verb-framework.md +++ b/docs/design/08-cli-verb-framework.md @@ -141,7 +141,7 @@ Read-only lifecycle verb (`requires_sentinel = False`). One term is queried agai Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspect tier, `requires_sentinel = False`). `bench_cmd.py` is only the argparse surface; the engine is `bench_core.run_toolchain_bench`, the one home shared with `state profiles` (GAIN) and the toolchain stage (`3.4.0-F11`), because verb modules are peers and may not import each other (`test_verb_modules_do_not_import_siblings`, which now also catches the `from sysforge import x_cmd` form). It answers the project's central question for the toolchain: is the installed sysforge-built clang actually faster than the stock repo clang? `bench` is a namespace because other targets (mesa via a shader-compile replay, the kernel) are follow-ups. -**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs` comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x); the installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock `, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. +**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs`, plus `clang-libs` when the sync DB carries it, comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x). **A repo split (`3.5.0-B146`).** `LD_LIBRARY_PATH` only puts the prefix first: a library missing from it is loaded from the host instead. Arch's clang 23.1.3-4 moved the resource dir (builtin headers) and `libclang-cpp` into a new `clang-libs` package, so the stock compiler ran with the sysforge `libclang-cpp` and failed on `#include `. `clang-libs` is passed to `stock_pkg.resolve` as `optional`, looked up on its own because one unknown target fails a whole `pacman -Sp` call, so a sync DB from before the split still benches; an optional package served by the local repo is refused like a required one. Before the stock binary runs, `stock_pkg.check_complete` requires `libclang-cpp`, `libLLVM` and the builtin headers in the prefix and names whatever is missing, so a future split fails the bench instead of borrowing the host's files. The installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock `, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. **Report.** Through `ui()`: the corpus label and file count, each compiler's version, median and min–max, and `stock/sysforge: x`. When the versions differ a line says the ratio includes upstream changes; across a major-version gap there is no ratio, only the reason. There is **no verdict and no threshold**: a cutoff tuned on one host means nothing on another. diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index ed8dac54..ed4a256c 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -915,6 +915,10 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-B145` — Rust packages built in the sandbox no longer show up as toolchain drift on every run.** In the build sandbox, a package that builds with Arch's `rust` is compiled with the repo's LLVM, because that is the LLVM Arch's `rust` is linked against. sysforge recorded that honestly, but then compared it with your own toolchain on every `update`, so with a sysforge-built LLVM every such package (the COSMIC `-git` packages, `mesa`, and other Rust-based packages) was reported as toolchain drift each time. With `rebuild_on_drift` or `--rebuild-on-toolchain-drift` on, they were rebuilt on every run, only to record the same thing again. Toolchain drift now leaves such a package alone while its next build would still use the repo LLVM, as it already did for a package kept on the other compiler family on purpose. It is reported again if you turn the sandbox off for it, or it stops needing Arch's `rust`. An update of the repo LLVM itself is not detected for these packages. +--- + +- **`3.5.0-B146` — `bench toolchain` works with Arch's new `clang-libs` package, and never times part of your build as stock.** From clang 23.1.3-4, Arch ships the compiler's builtin headers and `libclang-cpp` in a separate `clang-libs` package. The bench fetched only `clang` and `llvm-libs`, so its stock compiler stopped with `stddef.h` not found; before failing, it had also loaded your own `libclang-cpp` in place of the stock one. The bench now fetches `clang-libs` too, and still works against a repo from before the split. It also checks that the stock files it needs are all there before timing anything, and stops with a message naming what is missing, rather than measuring a stock compiler that is partly your own build. + ## Security - **`3.5.0-B37` — the root pacman hook no longer follows links in the state directory.** The hook helper runs as root in every pacman transaction and wrote into `/var/lib/sysforge/sentinels`, whose parent any `sysforge` group member can write to. A member could swap `sentinels` for a symlink to another directory, which the next transaction made group-writable, or symlink one of its files so the hook appended to any file as root. The helper now enters the directory and acts only if it is the real `sentinels`, owned by root. It appends only to a regular file, opened so that a symlink is refused and a planted FIFO cannot stall the transaction. As before, it never fails a transaction. diff --git a/sysforge/bench_core.py b/sysforge/bench_core.py index 0108c2ed..314f8d1a 100644 --- a/sysforge/bench_core.py +++ b/sysforge/bench_core.py @@ -35,6 +35,9 @@ TARGET = "toolchain" _STOCK_PKGS = ("clang", "llvm-libs") +# Split out of clang in 23.1.3-4 (resource dir, libclang-cpp); a sync DB from +# before the split has no such package (3.5.0-B146). +_STOCK_OPTIONAL = ("clang-libs",) _INSTALLED_CC = "/usr/bin/clang" _INSTALLED_CXX = "/usr/bin/clang++" DEFAULT_RUNS = 5 @@ -121,6 +124,7 @@ def _stock_env(prefix: Path) -> dict[str, str]: def _probe_stock(prefix: Path) -> None: + stock_pkg.check_complete(prefix) res = run.probe([str(prefix / "usr/bin/clang"), "--version"], env=_stock_env(prefix)) if res.returncode != 0: raise stock_pkg.StockUnavailable( @@ -202,7 +206,7 @@ def _fetch_stock(tmp: Path, offline: bool): repo: retry once against a private fresh copy. ``--offline`` never gets here, because an offline miss is not a download failure. """ - pkgs = stock_pkg.resolve(list(_STOCK_PKGS)) + pkgs = stock_pkg.resolve(list(_STOCK_PKGS), optional=list(_STOCK_OPTIONAL)) try: return pkgs, _obtain_all(pkgs, tmp, offline) except stock_pkg.StockDownloadFailed as exc: @@ -214,7 +218,8 @@ def _fetch_stock(tmp: Path, offline: bool): raise stock_pkg.StockUnavailable( f"{exc}; {sync_exc}. The sync database is probably stale: " "run `pacman -Syu`, then bench again") from exc - pkgs = stock_pkg.resolve(list(_STOCK_PKGS), dbpath=dbpath) + pkgs = stock_pkg.resolve(list(_STOCK_PKGS), optional=list(_STOCK_OPTIONAL), + dbpath=dbpath) return pkgs, _obtain_all(pkgs, tmp, offline) diff --git a/sysforge/primitives/stock_pkg.py b/sysforge/primitives/stock_pkg.py index 3aa02eb5..a3a488b4 100644 --- a/sysforge/primitives/stock_pkg.py +++ b/sysforge/primitives/stock_pkg.py @@ -49,7 +49,8 @@ _TIMEOUT_S = 60 _DEFAULT_DBPATH = Path("/var/lib/pacman") # What a stock clang needs to compile -O2 -c: the driver and its symlinks, the -# two shared libraries it loads, and the resource dir (builtin headers). +# two shared libraries it loads, and the resource dir (builtin headers). Globs, +# so a file moving between packages is still found (``check_complete``). EXTRACT_MEMBERS = ("usr/bin/clang", "usr/bin/clang++", "usr/bin/clang-[0-9]*", "usr/lib/libclang-cpp.so*", "usr/lib/libLLVM*.so*", "usr/lib/clang") @@ -73,13 +74,10 @@ class StockPkg: sha256: str -def resolve(names: Sequence[str], *, dbpath: Path | None = None) -> list[StockPkg]: - """The sync-DB build of each of *names*, in order. Raises ``StockUnavailable``. +def _lookup(names: Sequence[str], dbpath: Path | None) -> tuple[dict[str, StockPkg], str]: + """``pacman -Sp -dd`` rows for *names*, and pacman's stderr. - ``-dd`` skips dependency resolution: the bench needs the named archives - only, and resolving against the installed system lets an unrelated conflict - or versioned dependency fail the lookup. Only the requested names are kept. - *dbpath* reads a ``private_sync`` copy instead of the system sync DB. + One unknown target fails the whole call with no rows at all. """ argv = ["pacman", "-Sp", "-dd", "--print-format", _FORMAT] if dbpath is not None: @@ -90,15 +88,38 @@ def resolve(names: Sequence[str], *, dbpath: Path | None = None) -> list[StockPk parts = line.strip().split("|") if len(parts) == 6: rows[parts[0]] = StockPkg(*parts) + return rows, (res.stderr or "").strip() + + +def resolve(names: Sequence[str], *, optional: Sequence[str] = (), + dbpath: Path | None = None) -> list[StockPkg]: + """The sync-DB build of each of *names*, then of each *optional* name the + repo carries, in order. Raises ``StockUnavailable``. + + ``-dd`` skips dependency resolution: the bench needs the named archives + only, and resolving against the installed system lets an unrelated conflict + or versioned dependency fail the lookup. Only the requested names are kept. + *optional* names a package a repo split may or may not have produced yet + (3.5.0-B146); each is looked up on its own, since an unknown target would + fail the required lookup too. *dbpath* reads a ``private_sync`` copy + instead of the system sync DB. + """ + rows, stderr = _lookup(names, dbpath) missing = [n for n in names if n not in rows] if missing: - detail = (res.stderr or "").strip() or "not in any sync database" + detail = stderr or "not in any sync database" raise StockUnavailable(f"no stock repo carries {', '.join(missing)} ({detail})") - local = [n for n in names if rows[n].repo == REPO_NAME] + found = list(names) + for name in optional: + extra, _ = _lookup([name], dbpath) + if name in extra: + rows[name] = extra[name] + found.append(name) + local = [n for n in found if rows[n].repo == REPO_NAME] if local: raise StockUnavailable( f"{', '.join(local)} resolves to {REPO_NAME}, not a stock repo") - return [rows[n] for n in names] + return [rows[n] for n in found] def _sha256(path: Path) -> str: @@ -223,7 +244,8 @@ def _members(archive: Path) -> list[str]: """The :data:`EXTRACT_MEMBERS` patterns present in *archive*. bsdtar fails on a pattern an archive lacks, and each package carries only - part of the set (``libLLVM`` is in ``llvm-libs``, the rest in ``clang``). + part of the set (``libLLVM`` is in ``llvm-libs``; from clang 23.1.3-4 the + resource dir and ``libclang-cpp`` are in ``clang-libs``, the driver in ``clang``). """ res = run.probe(["bsdtar", "-tf", str(archive)]) if res.returncode != 0: @@ -250,3 +272,25 @@ def extract(archives: Sequence[Path], prefix: Path, *, f"could not extract {archive}: {(res.stderr or '').strip()}") if downloaded_in is not None and Path(archive).parent == Path(downloaded_in): Path(archive).unlink(missing_ok=True) + + +# What the extracted prefix must hold, as (glob under the prefix, what it is). +# A library missing here is not an error to the loader: it falls back to the +# host's copy, which is the sysforge build (3.5.0-B146). +_REQUIRED_PARTS = (("usr/lib/libclang-cpp.so*", "libclang-cpp"), + ("usr/lib/libLLVM*.so*", "libLLVM"), + ("usr/lib/clang/*/include/stddef.h", "builtin headers")) + + +def check_complete(prefix: Path) -> None: + """Raise ``StockUnavailable`` naming each part missing from *prefix*. + + The stock packages can move files between them on a repo split (clang + 23.1.3-4 moved the resource dir and ``libclang-cpp`` to ``clang-libs``). + """ + missing = [what for pattern, what in _REQUIRED_PARTS + if not any(Path(prefix).glob(pattern))] + if missing: + raise StockUnavailable( + f"the stock packages lack {', '.join(missing)}: the repo may have " + "split them into a package the bench does not fetch") diff --git a/tests/test_bench_cmd.py b/tests/test_bench_cmd.py index 3bf2f605..6174a909 100644 --- a/tests/test_bench_cmd.py +++ b/tests/test_bench_cmd.py @@ -35,9 +35,9 @@ def happy(tmp_path, monkeypatch): "built_at": "2026-10-10T08:00:00Z"} if env["build_mode"] else None)) monkeypatch.setattr(bench_core, "load_tcfg", lambda: None) - monkeypatch.setattr(sp, "resolve", lambda names, dbpath=None: [ + monkeypatch.setattr(sp, "resolve", lambda names, optional=(), dbpath=None: [ sp.StockPkg(n, env["stock"], "extra", f"{n}.pkg.tar.zst", "https://m", "x") - for n in names]) + for n in [*names, *optional]]) monkeypatch.setattr(sp, "obtain", lambda pkg, workdir, offline: Path(workdir) / pkg.filename) monkeypatch.setattr(sp, "extract", lambda archives, prefix, **kw: None) @@ -179,6 +179,7 @@ def test_phase_line_names_each_step_then_clears_for_the_report(tmp_path, happy, assert seen == [ "bench toolchain: fetching stock clang 23.1.1-1", "bench toolchain: fetching stock llvm-libs 23.1.1-1", + "bench toolchain: fetching stock clang-libs 23.1.1-1", "bench toolchain: unpacking stock clang", "bench toolchain: warm-up", "bench toolchain: round 1/2", @@ -210,10 +211,10 @@ def _stale_db(monkeypatch, env, *, sync_ok=True): sync names 23.1.3-3. Returns the list of private_sync calls.""" synced = [] - def resolve(names, dbpath=None): + def resolve(names, optional=(), dbpath=None): ver = "23.1.3-3" if dbpath else "23.1.3-2" return [sp.StockPkg(n, ver, "extra", f"{n}-{ver}.pkg.tar.zst", "https://m", "x") - for n in names] + for n in [*names, *optional]] def obtain(pkg, workdir, offline): if pkg.version == "23.1.3-2": @@ -277,10 +278,10 @@ def test_stale_attempt_drops_its_downloads_before_refreshing(tmp_path, happy, mo on the tmpfs next to the fresh one.""" left = [] - def resolve(names, dbpath=None): + def resolve(names, optional=(), dbpath=None): ver = "23.1.3-3" if dbpath else "23.1.3-2" return [sp.StockPkg(n, ver, "extra", f"{n}-{ver}.pkg.tar.zst", "https://m", "x") - for n in names] + for n in [*names, *optional]] def obtain(pkg, workdir, offline): if pkg.name == "llvm-libs" and pkg.version == "23.1.3-2": @@ -298,3 +299,36 @@ def private_sync(workdir): monkeypatch.setattr(sp, "private_sync", private_sync) assert bench_cmd.cmd_bench_toolchain(_args(tmp_path)) == 0 assert left == [] + + +# --- clang 23.1.3-4 split out clang-libs (3.5.0-B146) ------------------------ + + +def test_stock_fetch_asks_for_clang_libs_on_both_sync_paths(tmp_path, happy, monkeypatch): + """clang-libs carries the resource dir and libclang-cpp from 23.1.3-4; it is + optional so a sync DB from before the split still benches.""" + synced = _stale_db(monkeypatch, happy) + asked = [] + stale_resolve = sp.resolve + + def resolve(names, optional=(), dbpath=None): + asked.append((tuple(names), tuple(optional), dbpath is not None)) + return stale_resolve(names, optional=optional, dbpath=dbpath) + + monkeypatch.setattr(sp, "resolve", resolve) + assert bench_cmd.cmd_bench_toolchain(_args(tmp_path)) == 0 + assert synced + assert asked == [(("clang", "llvm-libs"), ("clang-libs",), False), + (("clang", "llvm-libs"), ("clang-libs",), True)] + + +def test_probe_stock_refuses_an_incomplete_prefix_before_running_it(tmp_path, monkeypatch): + """Missing libclang-cpp would load the host's (sysforge) copy and time the + sysforge frontend as "stock": refuse before the binary ever runs.""" + ran = [] + monkeypatch.setattr(bench_core.run, "probe", lambda argv, **kw: ran.append(argv)) + (tmp_path / "usr/bin").mkdir(parents=True) + (tmp_path / "usr/bin/clang").write_bytes(b"") + with pytest.raises(sp.StockUnavailable, match="libclang-cpp"): + bench_core._probe_stock(tmp_path) + assert ran == [] diff --git a/tests/test_stock_pkg.py b/tests/test_stock_pkg.py index 0c7552a7..f23122c1 100644 --- a/tests/test_stock_pkg.py +++ b/tests/test_stock_pkg.py @@ -350,3 +350,72 @@ def test_local_db_follows_pacman_conf_dbpath(monkeypatch): def test_local_db_defaults_when_pacman_conf_fails(monkeypatch): monkeypatch.setattr(sp.run, "probe", _probe("", rc=1)) assert sp._local_db() == Path("/var/lib/pacman/local") + + +# --- clang 23.1.3-4 split out clang-libs (3.5.0-B146) ------------------------ + + +def _sync(rows): + """pacman -Sp over a sync DB holding *rows* ({name: repo}): one unknown + target fails the whole call and prints nothing, as pacman does.""" + def probe(argv, **kw): + names = [a for a in argv[argv.index(sp._FORMAT) + 1:] if not a.startswith("-")] + unknown = [n for n in names if n not in rows] + if unknown: + return SimpleNamespace(returncode=1, stdout="", + stderr=f"error: target not found: {unknown[0]}") + out = "".join(f"{n}|23.1.3-4|{rows[n]}|{n}.pkg.tar.zst|https://m/{n}|aa\n" + for n in names) + return SimpleNamespace(returncode=0, stdout=out, stderr="") + return probe + + +def test_resolve_includes_an_optional_package_the_repo_carries(monkeypatch): + monkeypatch.setattr(sp.run, "probe", _sync( + {"clang": "extra", "llvm-libs": "extra", "clang-libs": "extra"})) + pkgs = sp.resolve(["clang", "llvm-libs"], optional=["clang-libs"]) + assert [p.name for p in pkgs] == ["clang", "llvm-libs", "clang-libs"] + + +def test_resolve_drops_an_optional_package_the_repo_lacks(monkeypatch): + """A sync DB from before the split has no clang-libs: the required + packages must still resolve, not fail with the optional one.""" + monkeypatch.setattr(sp.run, "probe", _sync({"clang": "extra", "llvm-libs": "extra"})) + pkgs = sp.resolve(["clang", "llvm-libs"], optional=["clang-libs"]) + assert [p.name for p in pkgs] == ["clang", "llvm-libs"] + + +def test_resolve_refuses_an_optional_package_from_the_local_repo(monkeypatch): + monkeypatch.setattr(sp.run, "probe", _sync( + {"clang": "extra", "llvm-libs": "extra", "clang-libs": "sysforge-local"})) + with pytest.raises(sp.StockUnavailable, match="clang-libs resolves to sysforge-local"): + sp.resolve(["clang", "llvm-libs"], optional=["clang-libs"]) + + +def _prefix(tmp_path, *files): + for f in files: + p = tmp_path / f + p.parent.mkdir(parents=True, exist_ok=True) + p.write_bytes(b"") + return tmp_path + + +_COMPLETE = ("usr/bin/clang", "usr/lib/libclang-cpp.so.23.1", "usr/lib/libLLVM.so.23.1", + "usr/lib/clang/23/include/stddef.h") + + +def test_check_complete_accepts_a_full_prefix(tmp_path): + sp.check_complete(_prefix(tmp_path, *_COMPLETE)) + + +@pytest.mark.parametrize("missing, named", [ + ("usr/lib/libclang-cpp.so.23.1", "libclang-cpp"), + ("usr/lib/libLLVM.so.23.1", "libLLVM"), + ("usr/lib/clang/23/include/stddef.h", "builtin headers"), +]) +def test_check_complete_names_what_is_missing(tmp_path, missing, named): + """A part missing from the prefix would be loaded from the host (the + sysforge build) or not found at all: fail before timing anything.""" + prefix = _prefix(tmp_path, *(f for f in _COMPLETE if f != missing)) + with pytest.raises(sp.StockUnavailable, match=named): + sp.check_complete(prefix) From ef2c7d6a5996132f1354582098103340ce39d1e0 Mon Sep 17 00:00:00 2001 From: Keith Raghubar Date: Mon, 26 Oct 2026 11:51:59 -0400 Subject: [PATCH 2/3] Toolchain rollback and hook handle clang-libs, live-mirror test tier (3.5.0-B147, DEV24) B147: Arch's clang 23.1.3-4 split libclang-cpp and the builtin headers into clang-libs, which the first toolchain build after it adds. - gates.install_adds names members the install adds; rollback removes them with pacman -Rdd before reinstalling the snapshot, which would otherwise stop on file conflicts with the old clang - the stored recovery for such a run is the online pacman -S of the suite plus the added members, since one command cannot remove and downgrade - clang-libs joins LLVM_LOCKSTEP_SUITE (Gate 3 skew, preflight, provenance) - the toolchain pacman hook gains clang-libs, polly and openmp; a test holds its targets to the suite - pacman.remove_pkgs takes nodeps DEV24: make test-live runs network-marked tests against the live mirrors (skipped unless SYSFORGE_LIVE_TESTS=1), starting with the bench's stock fetch, unpack and probe; it fails all three ways on the pre-B146 code. Own CI job, in CI_JOBS, exempt from the release preflight. DESIGN.md gains the rubric for which tier a test belongs in; CLAUDE.md points at it. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 19 ++++ CLAUDE.md | 4 + DESIGN.md | 40 +++++++- Makefile | 10 +- docs/RELEASE-CHECKLIST.md | 12 +++ docs/design/07-pipeline-layer.md | 4 +- docs/design/08-cli-verb-framework.md | 2 +- docs/design/09-primitives-layer.md | 2 +- docs/design/17-release-plan.md | 30 ++++++ docs/design/21-standards.md | 2 +- docs/release-notes/unreleased.md | 8 ++ etc/pacman.d/hooks/sysforge-toolchain.hook | 3 + pyproject.toml | 1 + sysforge/pipeline/stages/toolchain/gates.py | 53 ++++++++-- sysforge/pipeline/stages/toolchain/stage.py | 11 +- sysforge/pipeline/stages/toolchain/verify.py | 11 +- sysforge/primitives/pacman.py | 9 +- sysforge/primitives/toolchain_preflight.py | 5 +- tests/conftest.py | 17 ++++ tests/test_live_stock_toolchain.py | 68 +++++++++++++ tests/test_pacman_hooks.py | 13 +++ tests/test_pacman_revert_helpers.py | 11 ++ tests/test_stage_toolchain.py | 46 +++++++++ tests/test_toolchain_gates.py | 101 +++++++++++++++++++ tests/test_toolchain_preflight.py | 7 +- tests/test_toolchain_verify.py | 21 ++++ tools/check_standards.py | 4 +- tools/e2e/README.md | 4 + 28 files changed, 486 insertions(+), 32 deletions(-) create mode 100644 tests/test_live_stock_toolchain.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d23ca39e..bda958c3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -56,3 +56,22 @@ jobs: - name: End-to-end scenarios run: make e2e E2E_HERE=1 + + # Tests against the live Arch mirrors (3.5.0-B146): a canary for a repo + # change the suite's stubs cannot see, such as clang 23.1.3-4 splitting out + # clang-libs. In CI_JOBS, like e2e. llvm-libs brings the stock compiler's + # runtime libraries, and is the host copy the test checks it does not load. + live: + runs-on: ubuntu-latest + container: docker.io/library/archlinux:base-devel + steps: + - name: Install git, make and llvm-libs + run: pacman -Syu --noconfirm --needed git make llvm-libs + + - uses: actions/checkout@v4 + + - name: Dev dependencies + run: make dev-deps-core PACMAN_FLAGS=--noconfirm + + - name: Live mirror tests + run: make test-live diff --git a/CLAUDE.md b/CLAUDE.md index 646d6480..28920d85 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -102,6 +102,10 @@ Shipped-file edits must pass `make check-shipped`; doc/design edits `make check- `verbs/registry.py` `COMMANDS`, directly or via its namespace's `VerbGroup`. §CLI Verb Framework. - **Dual-toolchain test parity**: logic branching on resolved compiler (gcc vs llvm) ships both a gcc-path and an llvm-path test in the same change. +- **A fix whose defect crossed a boundary the unit suite stubs also ships a test in that boundary's + tier**: root/user, cross-command state or a real terminal → `make e2e`; what the live repos hold + (a package split, rename, file move) → `make test-live`. The PR names the tier, or says why unit + tests suffice. Rubric: §Release Process "Which tier a test belongs in". - **Log levels follow the rubric in `docs/design/12-logging.md`** (§Logging; standards row 25): `ui()` is the answer the user asked for, `warn()`/`info()`/`debug()` are narration gated behind `-v`/`-vv`/`-vvv` — `ui()` is not a "always show this" escape hatch. A new pipeline stage requests the diff --git a/DESIGN.md b/DESIGN.md index 06becc44..d8aa9369 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1190,8 +1190,8 @@ When `compiler` is unset (or set to `"gcc"`), the toolchain stage is **register- - **Soname gate — pre-build, between Gate 1 and the build** (`_gate_soname_consumers`, calls `toolchain_safety.assess_libllvm_soname_impact`). PGO/optimization never changes `libLLVM`'s `SOVERSION` or exported symbols, so a same-version rebuild is ABI-identical and silent; but a libLLVM **version bump** changes the soname (`libLLVM.so.22.1` → `libLLVM.so.23.0`), and every installed package that links the old soname (mesa's radeonsi/radv/llvmpipe, and others) is left with a dangling `NEEDED` until rebuilt. The gate derives the target soname from the resolved llvm PKGBUILD pkgver, reads the installed soname authoritatively from `llvm-libs`'s on-disk file list, and — when they differ — enumerates the installed consumers (every package whose recorded `%DEPENDS%` carries a `libLLVM.so=` soname dep, via `dep_analysis._SONAME_RE`), collapsed to pkgbase and **excluding** the LLVM lockstep suite + the packages being built this run (they're rebuilt here already). The `rebuild_soname_consumers` mode (CLI `--rebuild-soname-consumers` > `toolchain.toml` > `"prompt"`) decides: `prompt` (default) warns + lists + asks, approving captures the consumers for rebuild after Gate 3 and declining is a clean abort (non-TTY aborts — never silently break the system, pointing at `=auto`/`=off`); `auto` approves without prompting; `off` builds the toolchain but rebuilds nothing, printing the manual `sysforge build ` command. `--dry-run` previews the impact without prompting or rebuilding. The consumer rebuild runs through the same PKGBUILD review policy as `update` and `build` (`pkgbuild_review.resolve_review_policy` over packages.toml `[build]`, 3.0.0-F3): with no terminal an untrusted changed consumer source is refused, and a refusal fails the stage with the manual rebuild command like any other unfinished consumer. Like `build`, it stamps `source = "repo"` on a consumer whose checkout's `origin` is the official packaging repo for that pkgbase (`aur.repo_checkout_pkgbase`: host `gitlab.archlinux.org`, project `archlinux/packaging/packages/`), so the default trusted list (`["repo"]`) accepts official-repo consumers such as doxygen without a prompt; before `3.5.0-B16` every consumer was unclassified, read as `aur`, and reviewed. Provenance is the origin, never the PKGBUILD's declared names: an AUR PKGBUILD may list any `pkgname`, so a name-based test would let it skip review. The facts function is pure (guarded reads degrade to "no impact"); the stage owns the prompt/abort policy. - **Build (outside the sentinel).** All passes build with `install=False`; the build functions return the built package map. A build-pass failure therefore mutates nothing and leaves **no sentinel** (matches kernel). - **Gate 2 — pre-install ABI audit** (`gate2_audit`, outside the sentinel, both paths). Scans the built `.pkg.tar*` for the `_ZNSt*@LLVM_*` hazard via `toolchain_safety.scan_abi_hazards`, **and** for a **graphics-consumer symbol shortfall** via `toolchain_safety.check_system_consumer_symbols`: it extracts the about-to-be-installed `libLLVM.so.*` from the built packages (lib32 excluded) and, for each installed external consumer that links that exact soname (mesa's `libgallium`/DRI/Vulkan drivers, curated in `_SYSTEM_LIBLLVM_CONSUMER_GLOBS`), checks every `LLVM_x.y`-versioned symbol the consumer imports is exported by the new libLLVM. An import one of the consumer's *other* `NEEDED` libraries exports (resolved through `ldconfig`) is not owed by libLLVM and is left out: libclang, libclang-cpp and liblldb version their own exports `LLVM_*` too, so doxygen's `clang_*@LLVM_13` and lldb's `lldb::SB*@LLVM_` would otherwise read as drift and queue a needless consumer rebuild (`3.5.0-B18`). Neither is any package the batch itself ships: besides the lockstep suite, the merged `llvm` pkgbase builds lldb, libclc and llvm-lit, whose installed copies the same `pacman -U` replaces, so the scan drops every pkgname read from the built archives' `.PKGINFO` (`toolchain_safety.shipped_pkgnames`, `3.5.0-B20`). `_diff_consumers_against_libllvm` splits a miss into two classes (see §`graphics-stack` → *Two stranding classes*): an **unhealable** target-init drop (`LLVMInitialize*` — a reduced `LLVM_TARGETS_TO_BUILD` dropped a backend the consumer needs, e.g. AMDGPU; the symbol exists nowhere) **aborts before any `pacman -U`** — nothing installed, no sentinel; a **healable** `std::` re-export drop (only non-target-init `LLVM_*` symbols — the `-fprofile-use` libLLVM inlined away the weak libstdc++ copies the stock build re-exported at the *same* soname) does **not** abort. For the healable case Gate 2 enumerates the installed libLLVM consumers via `toolchain_safety.libllvm_abi_consumers` (the reverse-dep `%DEPENDS%` walk shared with the soname gate, here un-gated on a soname change), applies the `rebuild_soname_consumers` mode (prompt/auto/off, mirroring the soname gate), and **returns** them for rebuild after Gate 3. This is the gate that `scan_abi_hazards` (intra-build only) never covered: it checks *already-installed* external consumers, the gap that let a target-reduced libLLVM ship. **`std::` export parity (`3.5.0-F4`).** The healable class cannot heal mesa built in the sandbox, which links the repo libLLVM again, so the llvm PKGBUILD's `prepare()` gains a block (`pkgbuild_patcher.patch_llvm_std_exports`, wired as `makepkg_wrapper._maybe_patch_llvm_std_exports` beside the target patch) that adds `primitives/llvm_std_exports.py`'s thunks to libLLVM: each libstdc++ member stock re-exported and the `-fprofile-use` build dropped is exported again as `sym@LLVM_` at a **non-default** version, forwarding to libstdc++'s own `@GLIBCXX_*` export. An already-linked consumer resolves it; no new link binds to it, so the suite built alongside keeps importing `@GLIBCXX_*` and `scan_abi_hazards` stays clean (default-version re-exports, the obvious fix, would trip it). The CMake hook configures itself out unless the build links with lld on x86-64 with a known `LLVM_SHLIB_SYMBOL_VERSION`. Gate 2 then reports, as `-v` narration, any pure-`std` export stock has that the build still lacks and the system `libstdc++.so.6` itself exports (only that extern-template set is imported rather than emitted by a consumer; stock exports other std instantiations only because GCC built it) (`toolchain_safety.check_std_export_parity`, against the signed stock `llvm-libs` of the same version in the pacman cache via `pacman.cached_repo_pkg_file`; skipped when none is cached). The TU goes in `llvm-shlib/sysforge/`, because LLVM's configure (`llvm_check_source_file_list`) rejects any unlisted `*.cpp` in a target's own directory. `build_fingerprint._SCHEMA` 4 keeps `--reuse-built` from reusing a libLLVM built without the thunks, and the rendered block's hash is a Pass-4 fingerprint input, so a later `STD_EXPORTS` edit invalidates reuse on its own. -- **Snapshot.** Right before the install, `cached_pkg_files_for(lockstep suite ∪ built names)` (in `primitives/pacman.py`) locates each currently-installed member's `.pkg.tar*` in the pacman cache. `gates.snapshot_suite` keeps only names installed before the run: a new subpackage or lib32 member has nothing to restore, and its `None` would otherwise read as a missing archive and refuse the whole rollback (`3.5.0-B76`). This is the offline-undo source. Gate 1 warns up front when any member's archive is missing (auto-undo will fall back to `pacman -S`). -- **Gate 3 — post-install verify, inside the sentinel** (`_verify_llvm_install`). Its `expected_targets` (for the `built ⊇ expected` `llvm-config --targets-built` check) is the **actually resolved** target set — `resolve_or_detect_llvm_targets(TOOLCHAIN_PATH, /hardware_profile.toml)`, the same value the build patched in — **not** just `toolchain.toml [llvm] targets`. On an autodetect host that key is unset, so the old form resolved to `None` and skipped the check entirely; sourcing the resolved set makes it actually assert (after the Part-1 baseline, that includes AMDGPU). A **post-install graphics arm** then runs `toolchain_safety.check_installed_consumer_symbols(exclude=…)` (the same symbol diff as Gate 2, but vs the *now-installed* `/usr/lib/libLLVM.so.*`, the numerically newest soname via `toolchain_safety.so_version` so a compat package's older library never stands in for it (`3.5.0-B58`), excluding the same shipped pkgnames, which the install just replaced; the other callers, `doctor` and `build_core`'s post-install backstop, exclude nothing). Only **unhealable** (target-init) findings fold into the verify issues; **healable** `std::` re-export misses are *expected* here — mesa is not rebuilt until *after* Gate 3, so folding them in would auto-rollback the very libLLVM the queued consumer rebuild is about to make coherent — and are logged, not failed. On any (unhealable) failure, the stage **auto-restores** the prior-good toolchain from the snapshot in one `pacman -U` transaction (`rollback_to_snapshot` → `batch_install_pkgs`): if restore succeeds the live `/usr` is whole again, so the sentinel is **cleared** and a `RuntimeError` is raised telling the user to investigate; if restore fails or the snapshot was incomplete, the sentinel is **kept** with `recovery_cmd` set to the snapshot restore (offline `pacman -U `, falling back to `pacman -S `). So a target-reduced libLLVM that somehow reached install is reverted automatically while rollback is still armed, rather than surfacing as a black screen at next login. +- **Snapshot.** Right before the install, `cached_pkg_files_for(lockstep suite ∪ built names)` (in `primitives/pacman.py`) locates each currently-installed member's `.pkg.tar*` in the pacman cache. `gates.snapshot_suite` keeps only names installed before the run: a new subpackage or lib32 member has nothing to restore, and its `None` would otherwise read as a missing archive and refuse the whole rollback (`3.5.0-B76`). This is the offline-undo source. Gate 1 warns up front when any member's archive is missing (auto-undo will fall back to `pacman -S`). **A member the install adds (`3.5.0-B147`).** After Gate 2, `gates.install_adds` names the members the install will put on the system that are not installed now: the same `select_install_members` closure the install uses, so the first build after clang 23.1.3-4 adds `clang-libs`. Such a member can own files an old package's cached archive also carries (`clang-libs` took `libclang-cpp` and the builtin headers from `clang`), so a bare `pacman -U` of the snapshot stops on file conflicts. Rollback therefore removes the added members first (`pacman -Rdd`, since the installed clang still depends on them) and installs nothing if that fails; and because one stored recovery command cannot remove and downgrade together, the sentinel's recovery for such a run is the online `pacman -S `, the added members named explicitly because at the repo's version string a dependency resolve would keep the sysforge copy beside the stock clang. `clang-libs` is in `LLVM_LOCKSTEP_SUITE` too, so Gate 3's skew check, the preflight and the provenance check see a member left behind; a system from before the split simply has none installed, which every reader of the suite allows. +- **Gate 3 — post-install verify, inside the sentinel** (`_verify_llvm_install`). Its `expected_targets` (for the `built ⊇ expected` `llvm-config --targets-built` check) is the **actually resolved** target set — `resolve_or_detect_llvm_targets(TOOLCHAIN_PATH, /hardware_profile.toml)`, the same value the build patched in — **not** just `toolchain.toml [llvm] targets`. On an autodetect host that key is unset, so the old form resolved to `None` and skipped the check entirely; sourcing the resolved set makes it actually assert (after the Part-1 baseline, that includes AMDGPU). A **post-install graphics arm** then runs `toolchain_safety.check_installed_consumer_symbols(exclude=…)` (the same symbol diff as Gate 2, but vs the *now-installed* `/usr/lib/libLLVM.so.*`, the numerically newest soname via `toolchain_safety.so_version` so a compat package's older library never stands in for it (`3.5.0-B58`), excluding the same shipped pkgnames, which the install just replaced; the other callers, `doctor` and `build_core`'s post-install backstop, exclude nothing). Only **unhealable** (target-init) findings fold into the verify issues; **healable** `std::` re-export misses are *expected* here — mesa is not rebuilt until *after* Gate 3, so folding them in would auto-rollback the very libLLVM the queued consumer rebuild is about to make coherent — and are logged, not failed. On any (unhealable) failure, the stage **auto-restores** the prior-good toolchain from the snapshot in one `pacman -U` transaction (`rollback_to_snapshot` → `batch_install_pkgs`): if restore succeeds the live `/usr` is whole again, so the sentinel is **cleared** and a `RuntimeError` is raised telling the user to investigate; if restore fails or the snapshot was incomplete, the sentinel is **kept** with `recovery_cmd` set to the snapshot restore (offline `pacman -U `, falling back to `pacman -S `, which is also the recovery when the install added a member, above). So a target-reduced libLLVM that somehow reached install is reverted automatically while rollback is still armed, rather than surfacing as a black screen at next login. - **Consumer rebuild — after Gate 3, outside the sentinel** (`rebuild_soname_consumers`). The rebuild set is the **union** of the pre-build soname gate's consumers (a soname *bump*, mode `prompt`-approved or `auto`) and Gate 2's healable `std::` re-export drift consumers (a same-soname ABI regression) — deduped. The now-installed-and-verified toolchain is followed by a rebuild of those consumers through the shared `build_core.build_and_install` engine (resolved via `find_pkgbuild` — auto-cloning repo packages like mesa — and the user's normal profile, so they re-link the new libLLVM: a soname-bump consumer picks up the new `NEEDED`, a std::-drift consumer re-links the dropped symbols to libstdc++ `@GLIBCXX_*`). It runs **outside** the toolchain sentinel deliberately: a consumer rebuild failure must **not** roll back the intended toolchain bump — it surfaces as an actionable `RuntimeError` naming the failed packages and the manual `sysforge build` command, with the new toolchain left healthy. The system self-stabilises — once a std::-drift consumer is rebuilt it no longer imports the `…@LLVM_` `std::` symbols, so a later `run toolchain` sees no stranding and queues no rebuild. @@ -1454,7 +1454,7 @@ Read-only lifecycle verb (`requires_sentinel = False`). One term is queried agai Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspect tier, `requires_sentinel = False`). `bench_cmd.py` is only the argparse surface; the engine is `bench_core.run_toolchain_bench`, the one home shared with `state profiles` (GAIN) and the toolchain stage (`3.4.0-F11`), because verb modules are peers and may not import each other (`test_verb_modules_do_not_import_siblings`, which now also catches the `from sysforge import x_cmd` form). It answers the project's central question for the toolchain: is the installed sysforge-built clang actually faster than the stock repo clang? `bench` is a namespace because other targets (mesa via a shader-compile replay, the kernel) are follow-ups. -**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs`, plus `clang-libs` when the sync DB carries it, comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x). **A repo split (`3.5.0-B146`).** `LD_LIBRARY_PATH` only puts the prefix first: a library missing from it is loaded from the host instead. Arch's clang 23.1.3-4 moved the resource dir (builtin headers) and `libclang-cpp` into a new `clang-libs` package, so the stock compiler ran with the sysforge `libclang-cpp` and failed on `#include `. `clang-libs` is passed to `stock_pkg.resolve` as `optional`, looked up on its own because one unknown target fails a whole `pacman -Sp` call, so a sync DB from before the split still benches; an optional package served by the local repo is refused like a required one. Before the stock binary runs, `stock_pkg.check_complete` requires `libclang-cpp`, `libLLVM` and the builtin headers in the prefix and names whatever is missing, so a future split fails the bench instead of borrowing the host's files. The installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock `, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. +**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs`, plus `clang-libs` when the sync DB carries it, comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x). **A repo split (`3.5.0-B146`).** `LD_LIBRARY_PATH` only puts the prefix first: a library missing from it is loaded from the host instead. Arch's clang 23.1.3-4 moved the resource dir (builtin headers) and `libclang-cpp` into a new `clang-libs` package, so the stock compiler ran with the sysforge `libclang-cpp` and failed on `#include `. `clang-libs` is passed to `stock_pkg.resolve` as `optional`, looked up on its own because one unknown target fails a whole `pacman -Sp` call, so a sync DB from before the split still benches; an optional package served by the local repo is refused like a required one. Before the stock binary runs, `stock_pkg.check_complete` requires `libclang-cpp`, `libLLVM` and the builtin headers in the prefix and names whatever is missing, so a future split fails the bench instead of borrowing the host's files. The live-mirror tier (`tests/test_live_stock_toolchain.py`, §Release Process) runs this fetch, unpack and probe against the current repo. The installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock `, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. **Report.** Through `ui()`: the corpus label and file count, each compiler's version, median and min–max, and `stock/sysforge: x`. When the versions differ a line says the ratio includes upstream changes; across a major-version gap there is no ratio, only the reason. There is **no verdict and no threshold**: a cutoff tuned on one host means nothing on another. @@ -2059,7 +2059,7 @@ The five read-only queries above (`get_installed_version`, `get_all_installed_pa Constants: `BATCH_STRIP_FLAGS` (flags removed from per-build makepkg calls during batch install — composed as `SYNC_FLAGS | INSTALL_FLAGS`, the two flag families defined in `makepkg_flags.py`), `BATCH_EXTRA_FLAGS`. `SYNC_FLAGS` (`{--syncdeps, -s}`) is the single source of truth for the dep-sync strip — reused by the toolchain stage's staged-deps passes (§`run toolchain` Dep resolution) so the two sites that suppress makepkg's `pacman -S` share one name. Both `INSTALL_FLAGS` and `SYNC_FLAGS` live in `makepkg_flags.py` (their natural home — flag-family constants); `pacman`/`toolchain` import them from there, and `makepkg_wrapper` re-exports `INSTALL_FLAGS` for back-compat. -**Pacman PostTransaction hooks.** Three libalpm hooks under `/usr/share/libalpm/hooks/sysforge-{kernel,toolchain,buildstate}.hook` invoke `/usr/lib/sysforge/pacman-hook-helper.sh` to drop a sentinel into `/var/lib/sysforge/sentinels/`. Targets: kernel hook fires on `linux*` (inclusive of `linux-firmware` and `linux-headers`); toolchain hook fires on `llvm*`, `clang`, `lld`, `compiler-rt`, `gcc`, `gcc-libs` and the lib32 variants; buildstate hook fires on `*`. The helper is failsafe — every error path exits 0 so it cannot break a pacman transaction — and never follows a link: the sentinel directory's parent is group-writable, so it `cd`s in, acts only when the physical path is `sentinels` and owned by root, and appends through `dd oflag=append,nofollow,nonblock` (`3.5.0-B37`). `cmd_update` calls `_consume_pacman_hook_sentinels()` at entry: kernel/toolchain sentinels become `_log.warn` reminders, then unlink; buildstate is unlinked silently because the existing `BuildState.sync_with_installed()` already runs. The sentinel directory is created via tmpfiles.d and pre-created during the bootstrap configure stage; the consumer skips silently when the directory is absent so older installs that predate the hooks still work. +**Pacman PostTransaction hooks.** Four libalpm hooks under `/usr/share/libalpm/hooks/sysforge-{kernel,toolchain,buildstate,artifacts}.hook` invoke `/usr/lib/sysforge/pacman-hook-helper.sh` to drop a sentinel into `/var/lib/sysforge/sentinels/`. Targets: kernel hook fires on `linux*` (inclusive of `linux-firmware` and `linux-headers`); toolchain hook fires on every `LLVM_LOCKSTEP_SUITE` member (`llvm`, `llvm-libs`, `clang`, `clang-libs`, `lld`, `compiler-rt`, `polly`, `openmp`), `gcc`, `gcc-libs` and the lib32 variants, and `tests/test_pacman_hooks.py` holds its `Target` lines to the suite, so a member split out upstream cannot be missed again (`3.5.0-B147`: `clang-libs`, `polly` and `openmp` were); buildstate and artifacts hooks fire on `*`. The helper is failsafe — every error path exits 0 so it cannot break a pacman transaction — and never follows a link: the sentinel directory's parent is group-writable, so it `cd`s in, acts only when the physical path is `sentinels` and owned by root, and appends through `dd oflag=append,nofollow,nonblock` (`3.5.0-B37`). `cmd_update` calls `_consume_pacman_hook_sentinels()` at entry: kernel/toolchain sentinels become `_log.warn` reminders, then unlink; buildstate is unlinked silently because the existing `BuildState.sync_with_installed()` already runs. The sentinel directory is created via tmpfiles.d and pre-created during the bootstrap configure stage; the consumer skips silently when the directory is absent so older installs that predate the hooks still work. ### `change_report.py` @@ -4122,6 +4122,36 @@ listed, with its reason, in `tools/e2e/uncovered.toml` (`3.5.0-DEV23`); `tools/e does the comparison, `tests/test_e2e_coverage.py` fails the suite on a command in neither set or a listing a scenario has made redundant, and `make e2e-coverage` prints both. +The **live-mirror tier** (`make test-live`, `tests/test_live_stock_toolchain.py`, `3.5.0-DEV24`) +is the part of the suite that reads the Arch repos as they are today. Its tests carry the `network` +marker and are skipped unless `SYSFORGE_LIVE_TESTS=1`, which only `test-live` sets: an environment +switch, because `tools/ci.sh` runs the suite with `-o addopts=` and would drop a marker filter. It +is a canary for an upstream change the stubs cannot see. Arch's clang 23.1.3-4 moved +`libclang-cpp` and the builtin headers into a new `clang-libs` package, and every unit test still +passed (`3.5.0-B146`): the stubs held the old layout. The bench's stock toolchain is the first +subject. Its tests run the real fetch, unpack and probe, check that `libLLVM` and `libclang-cpp` +resolve inside the stock prefix rather than from the host, and compile against the builtin headers. +It is in `CI_JOBS` as a `live` job of its own (with `llvm-libs` installed for the stock compiler's +runtime libraries, the host copy the library check must not resolve to), so `pre-release` runs it; +the release preflight does not (`_RELEASE_GATE_EXEMPT`). + +**Which tier a test belongs in.** Every fix and feature ships unit tests. A fix also ships a +test in a higher tier when the defect crossed a boundary the unit suite stubs, since a stub encodes +the same assumption the bug broke: + +| The defect crossed | Tier | +|--------------------|------| +| root versus the user, group membership, sudo | `make e2e` scenario | +| state one command leaves for the next (sentinels, `build_state.toml`, the local repo) | `make e2e` scenario | +| a real terminal (prompts, progress, line discipline) | `make e2e` (prompts answered by `tools/e2e/answer.py`) or the terminal-replay tests (`3.5.0-DEV22`) | +| what the live repos or mirrors hold (a package split, rename or file move; a stale sync DB) | `make test-live` | +| nothing: logic inside a seam the unit tests exercise for real | unit tests only | + +When a boundary applies but a higher-tier test is impractical (a precondition that costs an +hours-long build, as `bench toolchain` itself has), test the boundary at the narrowest point that +still crosses it, as the live tier tests the bench's stock fetch without the bench. The pull +request names the tier, or says why only unit tests apply. + **CI** (`.github/workflows/ci.yml`, `3.4.0-DEV1`) runs every `pre-release` gate on each pull request (a newer push to it cancels the superseded run) and on each push to `main`, in a fresh `archlinux:base-devel` container, as an unprivileged user, with only the core dev tier (`make dev-deps-core`) installed. Its job is @@ -4384,7 +4414,7 @@ subsection is only presentation. | 24 | Deprecation registry (declared removal version + warn-on-use) | Every config key, state token, CLI flag, or path sysforge still honours only for backwards compatibility | enforced | `primitives/deprecations.py` is the single home: each record carries `surface`/`kind`/`function`/`deprecated_in`/`removed_in`/`replacement`, and each compat read path calls `warn_used(surface)` so the warning text is built from the record and cannot drift from the version the gate enforces. Two `function` values, because removal is not uniformly breaking — a `compat` surface still works (removing it is breaking, so `removed_in` must be `X.0.0`, and presence is proven by its `warn_used` call sites), while a `shim` already fails and is kept only to name its replacement (removal is not breaking, `removed_in` may be `X.Y.0`, presence proven by an exact repo-relative `anchor`). `check_standards` `deprecations` group: registry↔call-site bijection both ways, major-only removal for `compat`, and an error when the release target is at or past a declared `removed_in` with the surface still present (`--target-version`); a registry the gate cannot read is itself an error, because a check that cannot fail is worse than no check; one written as an empty tuple (`_REGISTRY = ()`) is accepted, since that is what removing the last record leaves (`3.5.0-F16`). The release-note removal parity check matches the registry's literal surface string against the `## Removed` section body, so a removal note must spell the canonical identifier (e.g. `build_state.build_mode=profiled`), not just prose describing it. `tests/test_deprecations.py` (behavioural, incl. once-per-run dedup and per-surface resolution) + `tests/test_standards_compliance.py`. **First application:** the 2.6.1-F5 sweep removed all five `compat` surfaces the registry catalogued at `2.6.1-STD2` shipping time, proving the major-only removal gate end to end. The `shim` half was proven the same way at `3.0.0-STD3`: `doctor.flat_flags` declared `removed_in = 3.1.0`, and the release-target gate refused the 3.1.0 release until the hint table, its `cli.py` hook and the record itself were deleted — a scheduled removal the gate enforced rather than a reviewer remembering. The registry is empty as of 4.0.0: its last record, the `compat` `profiles.build_mode=patched_pkgbuild` (`2.6.1-STD3`), was removed on schedule (`3.5.0-F16`), and `normalize_build_mode` now refuses the token. That record predated the registry and `2.6.1-STD2` missed it, because the bijection walks registry→call-site and never code→registry: **an unregistered compat surface is invisible to the gate by construction.** A catalogued-empty `compat` half is therefore never proof that none exist, and finding the next one is a review obligation, not a tooling guarantee — a code→registry direction is unimplemented (no reliable static signal distinguishes a compat alias table from any other lookup dict). Its `removed_in` was `4.0.0` rather than `3.0.0` because 3.0.0 shipped with the surface present, and the gate refuses a release at or past a declared removal with the surface still live. | | 25 | Log-level rubric (UI / ERROR / WARN / INFO / DEBUG) | Every call site that emits user-visible output | followed | `docs/design/12-logging.md` is the single home: the five-level table (fn, gate, reserved-for), the decision test that resolves most call sites (*is this the answer, or narration about producing the answer?*), and the explicit anti-pattern that `ui()` is not a "make this always show up" escape hatch — `ui()` is the answer the user ran the command to see, `info()`/`warn()` are narration and are gated behind `-vv`/`-v`. Enforcement is deliberately partial and the row does not claim otherwise. The one mechanical guard is the `quiet_at_default` fixture (`tests/conftest.py`), a golden-output regression that runs a stage at the shipped default (`verbosity=0`) and again at `-vv`, and fails if the default run emitted an `[INFO]` or `[WARN]` line. That catches the failure mode that matters — narration leaking into always-visible output — but nothing checks WARN-vs-INFO or INFO-vs-DEBUG classification, which stays a review obligation. Coverage is per-stage opt-in: the fixture is anchored on `packages` (`_load_packages`, `tests/test_stage_packages.py`) and `configure`'s dry run (`tests/test_stage_bootstrap.py`). A stage added without requesting the fixture is unguarded by construction, so **coverage thins with every stage that does not opt in** — requesting it is part of adding a stage, not a later cleanup. | | 26 | Lint scope (every Python tree in the repo) | `sysforge/`, `tests/`, and `tools/` — all Python the repo owns, not only the shipped package | enforced | `make lint-py` runs `ruff check sysforge/ tests/ tools/`, and `.claude/hooks/ruff-on-edit.sh` blocks on the same three trees. The two must name the identical set: before 2.6.1-STD8 the gate covered `sysforge/` alone while the hook already blocked on `tests/`, so 189 violations accumulated in a tree nothing reported on and the hook fired on pre-existing debt in files an author had merely touched — enforcement strict enough to interrupt, scoped so it could only ever surface someone else's backlog. Rule selection is uniform (`[tool.ruff.lint] select`); the only per-tree relaxation is `[tool.ruff.lint.per-file-ignores]` `"tests/**" = ["S", "PTH", "SIM117"]`, each with a stated reason — `S` because test code is not production attack surface (2.3.0-F1), `PTH` because it fires on the `sys.path` import bootstrap repeated across ~25 test files, `SIM117` because collapsing stacked monkeypatch/mock context managers costs a line-length violation per test and reads worse. A new relaxation is a per-file-ignores entry with a comment, never a directory dropped from the gate. Scope note: `tools/` is deliberately **not** relaxed — it holds `check_standards.py`, `check_shipped.py`, and `release.sh`'s Python siblings, which the release process depends on. | -| 27 | Release-gate completeness | Static-analysis gates of `make pre-release` vs the `tools/release.sh` preflight | enforced | Every `pre-release` prerequisite except the full `test` suite and the `e2e` tier (the heavier tiers, by design; `e2e` also needs podman and the network, so it could not be deterministic) is also run by the release preflight, which fails the release on it. `pre-release` is advisory and the script is the gate a release cannot skip, so a check only the former runs is one a tag can ship without: `v3.2.0` shipped four pyright errors — one a live crash (`3.2.0-B30`) the type-checker had named weeks earlier — and the 3.3.0 cycle reached eighteen (`3.2.0-STD1`). `check_standards` `release_gates` group diffs the `pre-release:` prerequisite list against the targets `release.sh` invokes, so `docs/RELEASE-CHECKLIST.md`'s gate table cannot silently diverge. A new `pre-release` gate is added to the script in the same change, or to `_RELEASE_GATE_EXEMPT` with its reason. Because the script *blocks*, each gate must also be deterministic: a `uv run --with` overlay in a gate's recipe is pinned (`PYRIGHT_VERSION`, `REUSE_VERSION` in the Makefile header), since an unpinned overlay resolves the newest release on every run and a tool update alone could turn a release red; the same group flags an unpinned `--with` in any gate recipe. Pins are bumped deliberately in their own commit, and the header keeps the no-`[dependency-groups] dev` rule (the overlays stay the one record). Known residue: `lint` runs the pacman-provided `ruff`, which is kept on `PATH` for editor/hook integration and so follows system upgrades. | +| 27 | Release-gate completeness | Static-analysis gates of `make pre-release` vs the `tools/release.sh` preflight | enforced | Every `pre-release` prerequisite except the full `test` suite, the `e2e` tier and the live-mirror tier `test-live` (the heavier tiers, by design; `e2e` also needs podman and the network, and `test-live` reads the repos as they are today, so neither could be deterministic) is also run by the release preflight, which fails the release on it. `pre-release` is advisory and the script is the gate a release cannot skip, so a check only the former runs is one a tag can ship without: `v3.2.0` shipped four pyright errors — one a live crash (`3.2.0-B30`) the type-checker had named weeks earlier — and the 3.3.0 cycle reached eighteen (`3.2.0-STD1`). `check_standards` `release_gates` group diffs the `pre-release:` prerequisite list against the targets `release.sh` invokes, so `docs/RELEASE-CHECKLIST.md`'s gate table cannot silently diverge. A new `pre-release` gate is added to the script in the same change, or to `_RELEASE_GATE_EXEMPT` with its reason. Because the script *blocks*, each gate must also be deterministic: a `uv run --with` overlay in a gate's recipe is pinned (`PYRIGHT_VERSION`, `REUSE_VERSION` in the Makefile header), since an unpinned overlay resolves the newest release on every run and a tool update alone could turn a release red; the same group flags an unpinned `--with` in any gate recipe. Pins are bumped deliberately in their own commit, and the header keeps the no-`[dependency-groups] dev` rule (the overlays stay the one record). Known residue: `lint` runs the pacman-provided `ruff`, which is kept on `PATH` for editor/hook integration and so follows system upgrades. | | 30 | No em-dashes in user-facing prose | `README.md`, `docs/guides/`, `docs/concepts/` (`3.5.0-DOC3`), the man page (`man/sysforge.1.scd.in` and the generated `man/sysforge.1`), `completions/`, and every non-docstring string literal under `sysforge/`: `--help` text, log and prompt messages, exception text, and the headers of files sysforge generates | enforced | Rewrite with a comma, colon, semicolon, period or parentheses as the sentence needs, never a blind substitution (`3.4.0-DOC2`). `check_standards` `em_dash` group: a line scan of the files and an AST scan of every string constant under `sysforge/`, so implicit concatenation and f-string pieces are covered without importing `sysforge`. Exempt: docstrings (developer prose), a literal that is only the glyph (an empty-cell placeholder, or a key in `log`'s ASCII glyph-fallback table), ROADMAP and release-note entries, whose ``- **`` — .**`` lead is a lint-enforced format (changing it is its own `STD`), internal docs, shipped-config and hook comments, and code comments. `tests/test_standards_compliance.py` (`test_em_dash_*`). | | 31 | Subprocess goes through `primitives/run` | Every external command `sysforge/` runs | enforced | Numbered 28 until `3.4.0-STD2`, when it was found sharing that number with the `pam.d(5)` row, which took 28 first and keeps it. ruff `banned-api` (TID251, `pyproject.toml`) bans `subprocess.run`/`Popen`/`call`/`check_call`/`check_output` outside the seam modules `primitives/run.py`, `pty_runner.py`, `privilege.py` and `sudo_session.py` (`3.2.0-F13`). Probes use `run.capture` (missing binary → `None`) or `run.probe` (missing binary → status 127), raise-on-failure calls use `run_or_raise`, and escalation uses the §22 privilege seam. A call that legitimately stays raw (inherits the TTY, streams, needs its own `preexec_fn`, or is a privileged command whose status or `CalledProcessError` its caller inspects) carries `# noqa: TID251 — <why>`, and the reason is mandatory. Enforced by the ruff rule plus `tests/test_standards_compliance.py` (`test_subprocess_fence_*`), which pins the banned set, the exemption list and the per-call reasons. Complements row 17, which governs argv form; this row governs *where* the call is made. | | 32 | Machine-readable output contract (`--json`) | `state list --json`, `doctor` (`system`, `pkg`, bare) `--json`, `update --dry-run --json`, `config check --json` (`3.5.0-F18`) | enforced | `primitives/json_out.py` is the one home (`3.4.0-F7`). Stdout carries exactly one JSON document ([RFC 8259](https://www.rfc-editor.org/rfc/rfc8259)) and nothing else: `cli.main` enters `json_out.reserved_stdout()` right after parsing and `verbs.runner.run_verb` enters it again (re-entrant), so every `ui()`/`print()`, startup logs and progress phases included, goes to stderr; the document goes out through `json_out.emit`. Every document has a top-level `"schema"` integer (`SCHEMA_VERSION`, now 1) and a `"kind"` (`state-list`, `doctor`, `update-dry-run`). **Adding a field is a minor change; renaming or removing one, or changing its type, is a major change and bumps `SCHEMA_VERSION`.** Each document is built from the result objects the human renderer reads (`AxisResult` via `diagnostics.axis_document`, `run.results` and the drift lists via `update._dry_run_document`, build_state rows through an explicit field list in `state_cmd._JSON_ENTRY_FIELDS`), so the two outputs cannot disagree. Exit codes are unchanged by `--json`. Mutating runs are out of scope: `update --json` without `--dry-run` and `doctor pkg --apply --json` are refused (exit 2). The schemas are committed as JSON Schema draft 2020-12 under `tests/data/json-schema/`; `tests/test_json_output.py` validates each verb's document against its schema (a subset validator, so the suite needs no new dependency), checks that stdout parses as one document with `log` in dry-run mode at `-vv`, and that a doctor skip appears under `roster.skipped`, never as a finding. A new field updates its schema in the same change. | diff --git a/Makefile b/Makefile index 5b026aed..279f13d4 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: all help surface check-surface install-hooks dev venv build install dev-install dev-uninstall clean distclean test test-x lint typecheck coverage coverage-ratchet coverage-ratchet-update man demo-gif readme-preview \ +.PHONY: all help surface check-surface install-hooks dev venv build install dev-install dev-uninstall clean distclean test test-x test-live lint typecheck coverage coverage-ratchet coverage-ratchet-update man demo-gif readme-preview \ dev-deps dev-deps-core dev-deps-pkg dev-deps-container dev-deps-release dev-deps-docs dev-deps-list \ lint-py lint-sh \ bench-completions check-shipped check-personal check-standards check-bump check-standards-at next-id next-bump design check-design pre-release print-gates audit \ @@ -179,6 +179,12 @@ test: ## Run the full suite (ARGS= extra pytest args) test-x: ## Run the suite, stop on first failure uv run --no-sync --with pytest --with tomlkit pytest -x $(ARGS) +# Tests that read the live Arch mirrors (3.5.0-B146), marked `network` and +# skipped by `make test`: a canary for repo changes the suite's stubs cannot +# see, such as a package split. CI runs it as a job of its own (CI_JOBS). +test-live: ## Run the live-mirror tests (downloads the stock toolchain packages) + SYSFORGE_LIVE_TESTS=1 uv run --no-sync --with pytest --with tomlkit pytest -m network $(ARGS) + # Both linters. Shell is not a second-class language here: 12 of the repo's # scripts are the test/release tooling, where a quoting bug fails a release or # silently tests the wrong tree. @@ -378,7 +384,7 @@ sync-config: ## Merge new shipped defaults into the live config dir # runs each in a job of its own (`make <job>` in .github/workflows/ci.yml), not # as a gate of tools/ci.sh (3.5.0-DEV21). GATES := lint typecheck test check-shipped check-personal check-design check-roadmap-table check-standards check-surface -CI_JOBS := e2e +CI_JOBS := e2e test-live pre-release: $(GATES) $(CI_JOBS) ## Composite gate: lint, types, tests, all checks, end-to-end tier diff --git a/docs/RELEASE-CHECKLIST.md b/docs/RELEASE-CHECKLIST.md index bc23c242..6016a526 100644 --- a/docs/RELEASE-CHECKLIST.md +++ b/docs/RELEASE-CHECKLIST.md @@ -16,6 +16,7 @@ make design roadmap-table # 2. regenerate, then commit anything that moved make ci-local # 3. every static gate + full suite, clean container make coverage-ratchet audit # 3. the two checks no gate runs make e2e # 4a +make test-live # 4a make vm-pkg-stable && make container-smoke && make container-smoke-cachyos # 4a make vm-snapshot && make vm-test && make vm-stop # 4b (+ git flavor) # 5. /release-notes skill @@ -72,6 +73,7 @@ In more detail: | `typecheck` (pyright) | yes | yes | no | yes | | full test suite | yes (`test`) | yes (`test`) | yes (`coverage`) | no | | end-to-end tier (`e2e`) | yes | no (its own CI job) | no | no | +| live-mirror tier (`test-live`) | yes | no (its own CI job) | no | no | | `check-shipped` (includes version markers, completions) | yes | yes | subset | yes | | `check-personal`, `check-design`, `check-roadmap-table`, `check-standards` | yes | yes | no | yes | | clean environment (not your workstation) | no | yes | no | no | @@ -157,6 +159,16 @@ make e2e # tools/e2e/README.md - [ ] `make e2e` exits 0 +Then the live-mirror tier: the stock toolchain the bench compares against, fetched from the repos +as they are today, so a repo-side change such as a package split shows up before users hit it. +CI runs it in a `live` job; run it here for the release commit. + +```bash +make test-live # tests/test_live_stock_toolchain.py +``` + +- [ ] `make test-live` exits 0 + The smokes then check the packaging itself. Seconds rather than a boot. The CachyOS arm is the one that actually exercises portability (repo/AUR diff --git a/docs/design/07-pipeline-layer.md b/docs/design/07-pipeline-layer.md index ecbdfac3..124314a7 100644 --- a/docs/design/07-pipeline-layer.md +++ b/docs/design/07-pipeline-layer.md @@ -566,8 +566,8 @@ When `compiler` is unset (or set to `"gcc"`), the toolchain stage is **register- - **Soname gate — pre-build, between Gate 1 and the build** (`_gate_soname_consumers`, calls `toolchain_safety.assess_libllvm_soname_impact`). PGO/optimization never changes `libLLVM`'s `SOVERSION` or exported symbols, so a same-version rebuild is ABI-identical and silent; but a libLLVM **version bump** changes the soname (`libLLVM.so.22.1` → `libLLVM.so.23.0`), and every installed package that links the old soname (mesa's radeonsi/radv/llvmpipe, and others) is left with a dangling `NEEDED` until rebuilt. The gate derives the target soname from the resolved llvm PKGBUILD pkgver, reads the installed soname authoritatively from `llvm-libs`'s on-disk file list, and — when they differ — enumerates the installed consumers (every package whose recorded `%DEPENDS%` carries a `libLLVM.so=<old major.minor>` soname dep, via `dep_analysis._SONAME_RE`), collapsed to pkgbase and **excluding** the LLVM lockstep suite + the packages being built this run (they're rebuilt here already). The `rebuild_soname_consumers` mode (CLI `--rebuild-soname-consumers` > `toolchain.toml` > `"prompt"`) decides: `prompt` (default) warns + lists + asks, approving captures the consumers for rebuild after Gate 3 and declining is a clean abort (non-TTY aborts — never silently break the system, pointing at `=auto`/`=off`); `auto` approves without prompting; `off` builds the toolchain but rebuilds nothing, printing the manual `sysforge build <consumers>` command. `--dry-run` previews the impact without prompting or rebuilding. The consumer rebuild runs through the same PKGBUILD review policy as `update` and `build` (`pkgbuild_review.resolve_review_policy` over packages.toml `[build]`, 3.0.0-F3): with no terminal an untrusted changed consumer source is refused, and a refusal fails the stage with the manual rebuild command like any other unfinished consumer. Like `build`, it stamps `source = "repo"` on a consumer whose checkout's `origin` is the official packaging repo for that pkgbase (`aur.repo_checkout_pkgbase`: host `gitlab.archlinux.org`, project `archlinux/packaging/packages/<pkgbase>`), so the default trusted list (`["repo"]`) accepts official-repo consumers such as doxygen without a prompt; before `3.5.0-B16` every consumer was unclassified, read as `aur`, and reviewed. Provenance is the origin, never the PKGBUILD's declared names: an AUR PKGBUILD may list any `pkgname`, so a name-based test would let it skip review. The facts function is pure (guarded reads degrade to "no impact"); the stage owns the prompt/abort policy. - **Build (outside the sentinel).** All passes build with `install=False`; the build functions return the built package map. A build-pass failure therefore mutates nothing and leaves **no sentinel** (matches kernel). - **Gate 2 — pre-install ABI audit** (`gate2_audit`, outside the sentinel, both paths). Scans the built `.pkg.tar*` for the `_ZNSt*@LLVM_*` hazard via `toolchain_safety.scan_abi_hazards`, **and** for a **graphics-consumer symbol shortfall** via `toolchain_safety.check_system_consumer_symbols`: it extracts the about-to-be-installed `libLLVM.so.*` from the built packages (lib32 excluded) and, for each installed external consumer that links that exact soname (mesa's `libgallium`/DRI/Vulkan drivers, curated in `_SYSTEM_LIBLLVM_CONSUMER_GLOBS`), checks every `LLVM_x.y`-versioned symbol the consumer imports is exported by the new libLLVM. An import one of the consumer's *other* `NEEDED` libraries exports (resolved through `ldconfig`) is not owed by libLLVM and is left out: libclang, libclang-cpp and liblldb version their own exports `LLVM_*` too, so doxygen's `clang_*@LLVM_13` and lldb's `lldb::SB*@LLVM_<ver>` would otherwise read as drift and queue a needless consumer rebuild (`3.5.0-B18`). Neither is any package the batch itself ships: besides the lockstep suite, the merged `llvm` pkgbase builds lldb, libclc and llvm-lit, whose installed copies the same `pacman -U` replaces, so the scan drops every pkgname read from the built archives' `.PKGINFO` (`toolchain_safety.shipped_pkgnames`, `3.5.0-B20`). `_diff_consumers_against_libllvm` splits a miss into two classes (see §`graphics-stack` → *Two stranding classes*): an **unhealable** target-init drop (`LLVMInitialize*` — a reduced `LLVM_TARGETS_TO_BUILD` dropped a backend the consumer needs, e.g. AMDGPU; the symbol exists nowhere) **aborts before any `pacman -U`** — nothing installed, no sentinel; a **healable** `std::` re-export drop (only non-target-init `LLVM_*` symbols — the `-fprofile-use` libLLVM inlined away the weak libstdc++ copies the stock build re-exported at the *same* soname) does **not** abort. For the healable case Gate 2 enumerates the installed libLLVM consumers via `toolchain_safety.libllvm_abi_consumers` (the reverse-dep `%DEPENDS%` walk shared with the soname gate, here un-gated on a soname change), applies the `rebuild_soname_consumers` mode (prompt/auto/off, mirroring the soname gate), and **returns** them for rebuild after Gate 3. This is the gate that `scan_abi_hazards` (intra-build only) never covered: it checks *already-installed* external consumers, the gap that let a target-reduced libLLVM ship. **`std::` export parity (`3.5.0-F4`).** The healable class cannot heal mesa built in the sandbox, which links the repo libLLVM again, so the llvm PKGBUILD's `prepare()` gains a block (`pkgbuild_patcher.patch_llvm_std_exports`, wired as `makepkg_wrapper._maybe_patch_llvm_std_exports` beside the target patch) that adds `primitives/llvm_std_exports.py`'s thunks to libLLVM: each libstdc++ member stock re-exported and the `-fprofile-use` build dropped is exported again as `sym@LLVM_<mm>` at a **non-default** version, forwarding to libstdc++'s own `@GLIBCXX_*` export. An already-linked consumer resolves it; no new link binds to it, so the suite built alongside keeps importing `@GLIBCXX_*` and `scan_abi_hazards` stays clean (default-version re-exports, the obvious fix, would trip it). The CMake hook configures itself out unless the build links with lld on x86-64 with a known `LLVM_SHLIB_SYMBOL_VERSION`. Gate 2 then reports, as `-v` narration, any pure-`std` export stock has that the build still lacks and the system `libstdc++.so.6` itself exports (only that extern-template set is imported rather than emitted by a consumer; stock exports other std instantiations only because GCC built it) (`toolchain_safety.check_std_export_parity`, against the signed stock `llvm-libs` of the same version in the pacman cache via `pacman.cached_repo_pkg_file`; skipped when none is cached). The TU goes in `llvm-shlib/sysforge/`, because LLVM's configure (`llvm_check_source_file_list`) rejects any unlisted `*.cpp` in a target's own directory. `build_fingerprint._SCHEMA` 4 keeps `--reuse-built` from reusing a libLLVM built without the thunks, and the rendered block's hash is a Pass-4 fingerprint input, so a later `STD_EXPORTS` edit invalidates reuse on its own. -- **Snapshot.** Right before the install, `cached_pkg_files_for(lockstep suite ∪ built names)` (in `primitives/pacman.py`) locates each currently-installed member's `.pkg.tar*` in the pacman cache. `gates.snapshot_suite` keeps only names installed before the run: a new subpackage or lib32 member has nothing to restore, and its `None` would otherwise read as a missing archive and refuse the whole rollback (`3.5.0-B76`). This is the offline-undo source. Gate 1 warns up front when any member's archive is missing (auto-undo will fall back to `pacman -S`). -- **Gate 3 — post-install verify, inside the sentinel** (`_verify_llvm_install`). Its `expected_targets` (for the `built ⊇ expected` `llvm-config --targets-built` check) is the **actually resolved** target set — `resolve_or_detect_llvm_targets(TOOLCHAIN_PATH, <state_dir>/hardware_profile.toml)`, the same value the build patched in — **not** just `toolchain.toml [llvm] targets`. On an autodetect host that key is unset, so the old form resolved to `None` and skipped the check entirely; sourcing the resolved set makes it actually assert (after the Part-1 baseline, that includes AMDGPU). A **post-install graphics arm** then runs `toolchain_safety.check_installed_consumer_symbols(exclude=…)` (the same symbol diff as Gate 2, but vs the *now-installed* `/usr/lib/libLLVM.so.*`, the numerically newest soname via `toolchain_safety.so_version` so a compat package's older library never stands in for it (`3.5.0-B58`), excluding the same shipped pkgnames, which the install just replaced; the other callers, `doctor` and `build_core`'s post-install backstop, exclude nothing). Only **unhealable** (target-init) findings fold into the verify issues; **healable** `std::` re-export misses are *expected* here — mesa is not rebuilt until *after* Gate 3, so folding them in would auto-rollback the very libLLVM the queued consumer rebuild is about to make coherent — and are logged, not failed. On any (unhealable) failure, the stage **auto-restores** the prior-good toolchain from the snapshot in one `pacman -U` transaction (`rollback_to_snapshot` → `batch_install_pkgs`): if restore succeeds the live `/usr` is whole again, so the sentinel is **cleared** and a `RuntimeError` is raised telling the user to investigate; if restore fails or the snapshot was incomplete, the sentinel is **kept** with `recovery_cmd` set to the snapshot restore (offline `pacman -U <cached>`, falling back to `pacman -S <suite>`). So a target-reduced libLLVM that somehow reached install is reverted automatically while rollback is still armed, rather than surfacing as a black screen at next login. +- **Snapshot.** Right before the install, `cached_pkg_files_for(lockstep suite ∪ built names)` (in `primitives/pacman.py`) locates each currently-installed member's `.pkg.tar*` in the pacman cache. `gates.snapshot_suite` keeps only names installed before the run: a new subpackage or lib32 member has nothing to restore, and its `None` would otherwise read as a missing archive and refuse the whole rollback (`3.5.0-B76`). This is the offline-undo source. Gate 1 warns up front when any member's archive is missing (auto-undo will fall back to `pacman -S`). **A member the install adds (`3.5.0-B147`).** After Gate 2, `gates.install_adds` names the members the install will put on the system that are not installed now: the same `select_install_members` closure the install uses, so the first build after clang 23.1.3-4 adds `clang-libs`. Such a member can own files an old package's cached archive also carries (`clang-libs` took `libclang-cpp` and the builtin headers from `clang`), so a bare `pacman -U` of the snapshot stops on file conflicts. Rollback therefore removes the added members first (`pacman -Rdd`, since the installed clang still depends on them) and installs nothing if that fails; and because one stored recovery command cannot remove and downgrade together, the sentinel's recovery for such a run is the online `pacman -S <suite> <added>`, the added members named explicitly because at the repo's version string a dependency resolve would keep the sysforge copy beside the stock clang. `clang-libs` is in `LLVM_LOCKSTEP_SUITE` too, so Gate 3's skew check, the preflight and the provenance check see a member left behind; a system from before the split simply has none installed, which every reader of the suite allows. +- **Gate 3 — post-install verify, inside the sentinel** (`_verify_llvm_install`). Its `expected_targets` (for the `built ⊇ expected` `llvm-config --targets-built` check) is the **actually resolved** target set — `resolve_or_detect_llvm_targets(TOOLCHAIN_PATH, <state_dir>/hardware_profile.toml)`, the same value the build patched in — **not** just `toolchain.toml [llvm] targets`. On an autodetect host that key is unset, so the old form resolved to `None` and skipped the check entirely; sourcing the resolved set makes it actually assert (after the Part-1 baseline, that includes AMDGPU). A **post-install graphics arm** then runs `toolchain_safety.check_installed_consumer_symbols(exclude=…)` (the same symbol diff as Gate 2, but vs the *now-installed* `/usr/lib/libLLVM.so.*`, the numerically newest soname via `toolchain_safety.so_version` so a compat package's older library never stands in for it (`3.5.0-B58`), excluding the same shipped pkgnames, which the install just replaced; the other callers, `doctor` and `build_core`'s post-install backstop, exclude nothing). Only **unhealable** (target-init) findings fold into the verify issues; **healable** `std::` re-export misses are *expected* here — mesa is not rebuilt until *after* Gate 3, so folding them in would auto-rollback the very libLLVM the queued consumer rebuild is about to make coherent — and are logged, not failed. On any (unhealable) failure, the stage **auto-restores** the prior-good toolchain from the snapshot in one `pacman -U` transaction (`rollback_to_snapshot` → `batch_install_pkgs`): if restore succeeds the live `/usr` is whole again, so the sentinel is **cleared** and a `RuntimeError` is raised telling the user to investigate; if restore fails or the snapshot was incomplete, the sentinel is **kept** with `recovery_cmd` set to the snapshot restore (offline `pacman -U <cached>`, falling back to `pacman -S <suite>`, which is also the recovery when the install added a member, above). So a target-reduced libLLVM that somehow reached install is reverted automatically while rollback is still armed, rather than surfacing as a black screen at next login. - **Consumer rebuild — after Gate 3, outside the sentinel** (`rebuild_soname_consumers`). The rebuild set is the **union** of the pre-build soname gate's consumers (a soname *bump*, mode `prompt`-approved or `auto`) and Gate 2's healable `std::` re-export drift consumers (a same-soname ABI regression) — deduped. The now-installed-and-verified toolchain is followed by a rebuild of those consumers through the shared `build_core.build_and_install` engine (resolved via `find_pkgbuild` — auto-cloning repo packages like mesa — and the user's normal profile, so they re-link the new libLLVM: a soname-bump consumer picks up the new `NEEDED`, a std::-drift consumer re-links the dropped symbols to libstdc++ `@GLIBCXX_*`). It runs **outside** the toolchain sentinel deliberately: a consumer rebuild failure must **not** roll back the intended toolchain bump — it surfaces as an actionable `RuntimeError` naming the failed packages and the manual `sysforge build` command, with the new toolchain left healthy. The system self-stabilises — once a std::-drift consumer is rebuilt it no longer imports the `…@LLVM_<ver>` `std::` symbols, so a later `run toolchain` sees no stranding and queues no rebuild. diff --git a/docs/design/08-cli-verb-framework.md b/docs/design/08-cli-verb-framework.md index 4baba011..81cff658 100644 --- a/docs/design/08-cli-verb-framework.md +++ b/docs/design/08-cli-verb-framework.md @@ -141,7 +141,7 @@ Read-only lifecycle verb (`requires_sentinel = False`). One term is queried agai Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspect tier, `requires_sentinel = False`). `bench_cmd.py` is only the argparse surface; the engine is `bench_core.run_toolchain_bench`, the one home shared with `state profiles` (GAIN) and the toolchain stage (`3.4.0-F11`), because verb modules are peers and may not import each other (`test_verb_modules_do_not_import_siblings`, which now also catches the `from sysforge import x_cmd` form). It answers the project's central question for the toolchain: is the installed sysforge-built clang actually faster than the stock repo clang? `bench` is a namespace because other targets (mesa via a shader-compile replay, the kernel) are follow-ups. -**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs`, plus `clang-libs` when the sync DB carries it, comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=<prefix>/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x). **A repo split (`3.5.0-B146`).** `LD_LIBRARY_PATH` only puts the prefix first: a library missing from it is loaded from the host instead. Arch's clang 23.1.3-4 moved the resource dir (builtin headers) and `libclang-cpp` into a new `clang-libs` package, so the stock compiler ran with the sysforge `libclang-cpp` and failed on `#include <stddef.h>`. `clang-libs` is passed to `stock_pkg.resolve` as `optional`, looked up on its own because one unknown target fails a whole `pacman -Sp` call, so a sync DB from before the split still benches; an optional package served by the local repo is refused like a required one. Before the stock binary runs, `stock_pkg.check_complete` requires `libclang-cpp`, `libLLVM` and the builtin headers in the prefix and names whatever is missing, so a future split fails the bench instead of borrowing the host's files. The installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock <pkg> <ver>`, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. +**Flow.** Preconditions first (exit 2, nothing measured): `/usr/bin/clang{,++}` exist, `BuildState.get("clang")` is a non-`pacman` build, and `pacman -Q clang` gives exactly the `[epoch:]pkgver-pkgrel` that entry recorded. Stage-owned entries are never auto-demoted, so without that match a stock clang reinstalled by `pacman -S`/`-Syu` would be benched as the sysforge build and its ~1.00x recorded as the gain. The workload is `ToolchainConfig.bench_corpus` (`toolchain.toml [bench] corpus`, files or directories of C/C++ sources) or, unset, the bench's own built-in TUs (`primitives/bench_workload.py`); each file is compiled standalone (`-O2 -c`, no include paths or defines), so a configured corpus must be self-contained or preprocessed sources. `ToolchainConfig.from_toml` validates the `[bench]` table (`_bench_table`): a non-table `[bench]` or a `corpus` that is not a path or a list of paths warns and falls back to the built-in workload rather than raising, because every verb that loads toolchain.toml parses the table, including `run toolchain`, which `[bench]` never affects. The stock build of `clang` + `llvm-libs`, plus `clang-libs` when the sync DB carries it, comes from `primitives/stock_pkg.py`: resolved with `pacman -Sp`, taken from the pacman cache or downloaded **as the user** (never `pacman -Sw`, which needs root), checked against the sync database's SHA256 (falling back across the repo's mirrors, and past a corrupt cached copy), and the compiler's own files unpacked with `bsdtar` into a temp prefix, the downloaded archive deleted once unpacked. `--offline` forbids the download. **A stale sync DB (`3.5.0-F1`).** When every mirror fails (`StockDownloadFailed`, a `StockUnavailable` subclass that an offline miss never raises), the system sync DB most likely names a build the repo has since replaced: `bench_core._fetch_stock` drops what that attempt downloaded, syncs a private copy with `stock_pkg.private_sync` into the bench's temp dir and resolves again from it, once. The system DB is never synced: that needs root, and a bare `-Sy` would leave the host partially upgraded. The fallback, not an always-private DB, keeps the usual comparison against the build the host's own DB names and costs nothing when that DB is current. A failed refresh reports both errors and points at `pacman -Syu`. The stock compiler runs from that prefix with `LD_LIBRARY_PATH=<prefix>/usr/lib`, so it loads the stock `libLLVM`/`libclang-cpp` rather than the host's sysforge build (otherwise the ratio would silently read about 1.00x). **A repo split (`3.5.0-B146`).** `LD_LIBRARY_PATH` only puts the prefix first: a library missing from it is loaded from the host instead. Arch's clang 23.1.3-4 moved the resource dir (builtin headers) and `libclang-cpp` into a new `clang-libs` package, so the stock compiler ran with the sysforge `libclang-cpp` and failed on `#include <stddef.h>`. `clang-libs` is passed to `stock_pkg.resolve` as `optional`, looked up on its own because one unknown target fails a whole `pacman -Sp` call, so a sync DB from before the split still benches; an optional package served by the local repo is refused like a required one. Before the stock binary runs, `stock_pkg.check_complete` requires `libclang-cpp`, `libLLVM` and the builtin headers in the prefix and names whatever is missing, so a future split fails the bench instead of borrowing the host's files. The live-mirror tier (`tests/test_live_stock_toolchain.py`, §Release Process) runs this fetch, unpack and probe against the current repo. The installed one runs as `/usr/bin/clang`. Both use absolute paths, so the ccache `PATH` wrappers never intervene, and every compile is `-O2 -c`, so link time and compiler-rt stay out of the measurement. `primitives/bench_harness.py` runs one untimed warm-up per compiler, then `--runs N` (default 5) rounds alternating stock and sysforge, so drift lands on both, with the order inside a round flipped every round (ABBA, `3.4.0-F10`) so a within-round order effect, such as caches the first compile warms or a turbo budget it spends, does not always favour the same side. **Progress (`3.4.0-F9`).** Each step sets the progress phase (`bench toolchain: fetching stock <pkg> <ver>`, `unpacking stock clang`, `warm-up`, `round i/N`): `bench_core._phase` drives `ui.progress.phase`, and the rounds reach it through `run_interleaved`'s `on_round` callback, so the harness primitive never imports `ui/`. `run_toolchain_bench` clears the phase in a `finally` around the measurement, so the report, or a failure, replaces it on every exit path. The `info()` narration is unchanged; the phase line is the display, not the log, and in plain (non-TTY) mode it prints as `[PROGRESS]` lines. **Report.** Through `ui()`: the corpus label and file count, each compiler's version, median and min–max, and `stock/sysforge: <r>x`. When the versions differ a line says the ratio includes upstream changes; across a major-version gap there is no ratio, only the reason. There is **no verdict and no threshold**: a cutoff tuned on one host means nothing on another. diff --git a/docs/design/09-primitives-layer.md b/docs/design/09-primitives-layer.md index 513dfc01..16862f47 100644 --- a/docs/design/09-primitives-layer.md +++ b/docs/design/09-primitives-layer.md @@ -164,7 +164,7 @@ The five read-only queries above (`get_installed_version`, `get_all_installed_pa Constants: `BATCH_STRIP_FLAGS` (flags removed from per-build makepkg calls during batch install — composed as `SYNC_FLAGS | INSTALL_FLAGS`, the two flag families defined in `makepkg_flags.py`), `BATCH_EXTRA_FLAGS`. `SYNC_FLAGS` (`{--syncdeps, -s}`) is the single source of truth for the dep-sync strip — reused by the toolchain stage's staged-deps passes (§`run toolchain` Dep resolution) so the two sites that suppress makepkg's `pacman -S` share one name. Both `INSTALL_FLAGS` and `SYNC_FLAGS` live in `makepkg_flags.py` (their natural home — flag-family constants); `pacman`/`toolchain` import them from there, and `makepkg_wrapper` re-exports `INSTALL_FLAGS` for back-compat. -**Pacman PostTransaction hooks.** Three libalpm hooks under `/usr/share/libalpm/hooks/sysforge-{kernel,toolchain,buildstate}.hook` invoke `/usr/lib/sysforge/pacman-hook-helper.sh` to drop a sentinel into `/var/lib/sysforge/sentinels/`. Targets: kernel hook fires on `linux*` (inclusive of `linux-firmware` and `linux-headers`); toolchain hook fires on `llvm*`, `clang`, `lld`, `compiler-rt`, `gcc`, `gcc-libs` and the lib32 variants; buildstate hook fires on `*`. The helper is failsafe — every error path exits 0 so it cannot break a pacman transaction — and never follows a link: the sentinel directory's parent is group-writable, so it `cd`s in, acts only when the physical path is `sentinels` and owned by root, and appends through `dd oflag=append,nofollow,nonblock` (`3.5.0-B37`). `cmd_update` calls `_consume_pacman_hook_sentinels()` at entry: kernel/toolchain sentinels become `_log.warn` reminders, then unlink; buildstate is unlinked silently because the existing `BuildState.sync_with_installed()` already runs. The sentinel directory is created via tmpfiles.d and pre-created during the bootstrap configure stage; the consumer skips silently when the directory is absent so older installs that predate the hooks still work. +**Pacman PostTransaction hooks.** Four libalpm hooks under `/usr/share/libalpm/hooks/sysforge-{kernel,toolchain,buildstate,artifacts}.hook` invoke `/usr/lib/sysforge/pacman-hook-helper.sh` to drop a sentinel into `/var/lib/sysforge/sentinels/`. Targets: kernel hook fires on `linux*` (inclusive of `linux-firmware` and `linux-headers`); toolchain hook fires on every `LLVM_LOCKSTEP_SUITE` member (`llvm`, `llvm-libs`, `clang`, `clang-libs`, `lld`, `compiler-rt`, `polly`, `openmp`), `gcc`, `gcc-libs` and the lib32 variants, and `tests/test_pacman_hooks.py` holds its `Target` lines to the suite, so a member split out upstream cannot be missed again (`3.5.0-B147`: `clang-libs`, `polly` and `openmp` were); buildstate and artifacts hooks fire on `*`. The helper is failsafe — every error path exits 0 so it cannot break a pacman transaction — and never follows a link: the sentinel directory's parent is group-writable, so it `cd`s in, acts only when the physical path is `sentinels` and owned by root, and appends through `dd oflag=append,nofollow,nonblock` (`3.5.0-B37`). `cmd_update` calls `_consume_pacman_hook_sentinels()` at entry: kernel/toolchain sentinels become `_log.warn` reminders, then unlink; buildstate is unlinked silently because the existing `BuildState.sync_with_installed()` already runs. The sentinel directory is created via tmpfiles.d and pre-created during the bootstrap configure stage; the consumer skips silently when the directory is absent so older installs that predate the hooks still work. ### `change_report.py` diff --git a/docs/design/17-release-plan.md b/docs/design/17-release-plan.md index bffd4e0f..5c2f5cba 100644 --- a/docs/design/17-release-plan.md +++ b/docs/design/17-release-plan.md @@ -52,6 +52,36 @@ listed, with its reason, in `tools/e2e/uncovered.toml` (`3.5.0-DEV23`); `tools/e does the comparison, `tests/test_e2e_coverage.py` fails the suite on a command in neither set or a listing a scenario has made redundant, and `make e2e-coverage` prints both. +The **live-mirror tier** (`make test-live`, `tests/test_live_stock_toolchain.py`, `3.5.0-DEV24`) +is the part of the suite that reads the Arch repos as they are today. Its tests carry the `network` +marker and are skipped unless `SYSFORGE_LIVE_TESTS=1`, which only `test-live` sets: an environment +switch, because `tools/ci.sh` runs the suite with `-o addopts=` and would drop a marker filter. It +is a canary for an upstream change the stubs cannot see. Arch's clang 23.1.3-4 moved +`libclang-cpp` and the builtin headers into a new `clang-libs` package, and every unit test still +passed (`3.5.0-B146`): the stubs held the old layout. The bench's stock toolchain is the first +subject. Its tests run the real fetch, unpack and probe, check that `libLLVM` and `libclang-cpp` +resolve inside the stock prefix rather than from the host, and compile against the builtin headers. +It is in `CI_JOBS` as a `live` job of its own (with `llvm-libs` installed for the stock compiler's +runtime libraries, the host copy the library check must not resolve to), so `pre-release` runs it; +the release preflight does not (`_RELEASE_GATE_EXEMPT`). + +**Which tier a test belongs in.** Every fix and feature ships unit tests. A fix also ships a +test in a higher tier when the defect crossed a boundary the unit suite stubs, since a stub encodes +the same assumption the bug broke: + +| The defect crossed | Tier | +|--------------------|------| +| root versus the user, group membership, sudo | `make e2e` scenario | +| state one command leaves for the next (sentinels, `build_state.toml`, the local repo) | `make e2e` scenario | +| a real terminal (prompts, progress, line discipline) | `make e2e` (prompts answered by `tools/e2e/answer.py`) or the terminal-replay tests (`3.5.0-DEV22`) | +| what the live repos or mirrors hold (a package split, rename or file move; a stale sync DB) | `make test-live` | +| nothing: logic inside a seam the unit tests exercise for real | unit tests only | + +When a boundary applies but a higher-tier test is impractical (a precondition that costs an +hours-long build, as `bench toolchain` itself has), test the boundary at the narrowest point that +still crosses it, as the live tier tests the bench's stock fetch without the bench. The pull +request names the tier, or says why only unit tests apply. + **CI** (`.github/workflows/ci.yml`, `3.4.0-DEV1`) runs every `pre-release` gate on each pull request (a newer push to it cancels the superseded run) and on each push to `main`, in a fresh `archlinux:base-devel` container, as an unprivileged user, with only the core dev tier (`make dev-deps-core`) installed. Its job is diff --git a/docs/design/21-standards.md b/docs/design/21-standards.md index 46b16434..947c13d3 100644 --- a/docs/design/21-standards.md +++ b/docs/design/21-standards.md @@ -60,7 +60,7 @@ subsection is only presentation. | 24 | Deprecation registry (declared removal version + warn-on-use) | Every config key, state token, CLI flag, or path sysforge still honours only for backwards compatibility | enforced | `primitives/deprecations.py` is the single home: each record carries `surface`/`kind`/`function`/`deprecated_in`/`removed_in`/`replacement`, and each compat read path calls `warn_used(surface)` so the warning text is built from the record and cannot drift from the version the gate enforces. Two `function` values, because removal is not uniformly breaking — a `compat` surface still works (removing it is breaking, so `removed_in` must be `X.0.0`, and presence is proven by its `warn_used` call sites), while a `shim` already fails and is kept only to name its replacement (removal is not breaking, `removed_in` may be `X.Y.0`, presence proven by an exact repo-relative `anchor`). `check_standards` `deprecations` group: registry↔call-site bijection both ways, major-only removal for `compat`, and an error when the release target is at or past a declared `removed_in` with the surface still present (`--target-version`); a registry the gate cannot read is itself an error, because a check that cannot fail is worse than no check; one written as an empty tuple (`_REGISTRY = ()`) is accepted, since that is what removing the last record leaves (`3.5.0-F16`). The release-note removal parity check matches the registry's literal surface string against the `## Removed` section body, so a removal note must spell the canonical identifier (e.g. `build_state.build_mode=profiled`), not just prose describing it. `tests/test_deprecations.py` (behavioural, incl. once-per-run dedup and per-surface resolution) + `tests/test_standards_compliance.py`. **First application:** the 2.6.1-F5 sweep removed all five `compat` surfaces the registry catalogued at `2.6.1-STD2` shipping time, proving the major-only removal gate end to end. The `shim` half was proven the same way at `3.0.0-STD3`: `doctor.flat_flags` declared `removed_in = 3.1.0`, and the release-target gate refused the 3.1.0 release until the hint table, its `cli.py` hook and the record itself were deleted — a scheduled removal the gate enforced rather than a reviewer remembering. The registry is empty as of 4.0.0: its last record, the `compat` `profiles.build_mode=patched_pkgbuild` (`2.6.1-STD3`), was removed on schedule (`3.5.0-F16`), and `normalize_build_mode` now refuses the token. That record predated the registry and `2.6.1-STD2` missed it, because the bijection walks registry→call-site and never code→registry: **an unregistered compat surface is invisible to the gate by construction.** A catalogued-empty `compat` half is therefore never proof that none exist, and finding the next one is a review obligation, not a tooling guarantee — a code→registry direction is unimplemented (no reliable static signal distinguishes a compat alias table from any other lookup dict). Its `removed_in` was `4.0.0` rather than `3.0.0` because 3.0.0 shipped with the surface present, and the gate refuses a release at or past a declared removal with the surface still live. | | 25 | Log-level rubric (UI / ERROR / WARN / INFO / DEBUG) | Every call site that emits user-visible output | followed | `docs/design/12-logging.md` is the single home: the five-level table (fn, gate, reserved-for), the decision test that resolves most call sites (*is this the answer, or narration about producing the answer?*), and the explicit anti-pattern that `ui()` is not a "make this always show up" escape hatch — `ui()` is the answer the user ran the command to see, `info()`/`warn()` are narration and are gated behind `-vv`/`-v`. Enforcement is deliberately partial and the row does not claim otherwise. The one mechanical guard is the `quiet_at_default` fixture (`tests/conftest.py`), a golden-output regression that runs a stage at the shipped default (`verbosity=0`) and again at `-vv`, and fails if the default run emitted an `[INFO]` or `[WARN]` line. That catches the failure mode that matters — narration leaking into always-visible output — but nothing checks WARN-vs-INFO or INFO-vs-DEBUG classification, which stays a review obligation. Coverage is per-stage opt-in: the fixture is anchored on `packages` (`_load_packages`, `tests/test_stage_packages.py`) and `configure`'s dry run (`tests/test_stage_bootstrap.py`). A stage added without requesting the fixture is unguarded by construction, so **coverage thins with every stage that does not opt in** — requesting it is part of adding a stage, not a later cleanup. | | 26 | Lint scope (every Python tree in the repo) | `sysforge/`, `tests/`, and `tools/` — all Python the repo owns, not only the shipped package | enforced | `make lint-py` runs `ruff check sysforge/ tests/ tools/`, and `.claude/hooks/ruff-on-edit.sh` blocks on the same three trees. The two must name the identical set: before 2.6.1-STD8 the gate covered `sysforge/` alone while the hook already blocked on `tests/`, so 189 violations accumulated in a tree nothing reported on and the hook fired on pre-existing debt in files an author had merely touched — enforcement strict enough to interrupt, scoped so it could only ever surface someone else's backlog. Rule selection is uniform (`[tool.ruff.lint] select`); the only per-tree relaxation is `[tool.ruff.lint.per-file-ignores]` `"tests/**" = ["S", "PTH", "SIM117"]`, each with a stated reason — `S` because test code is not production attack surface (2.3.0-F1), `PTH` because it fires on the `sys.path` import bootstrap repeated across ~25 test files, `SIM117` because collapsing stacked monkeypatch/mock context managers costs a line-length violation per test and reads worse. A new relaxation is a per-file-ignores entry with a comment, never a directory dropped from the gate. Scope note: `tools/` is deliberately **not** relaxed — it holds `check_standards.py`, `check_shipped.py`, and `release.sh`'s Python siblings, which the release process depends on. | -| 27 | Release-gate completeness | Static-analysis gates of `make pre-release` vs the `tools/release.sh` preflight | enforced | Every `pre-release` prerequisite except the full `test` suite and the `e2e` tier (the heavier tiers, by design; `e2e` also needs podman and the network, so it could not be deterministic) is also run by the release preflight, which fails the release on it. `pre-release` is advisory and the script is the gate a release cannot skip, so a check only the former runs is one a tag can ship without: `v3.2.0` shipped four pyright errors — one a live crash (`3.2.0-B30`) the type-checker had named weeks earlier — and the 3.3.0 cycle reached eighteen (`3.2.0-STD1`). `check_standards` `release_gates` group diffs the `pre-release:` prerequisite list against the targets `release.sh` invokes, so `docs/RELEASE-CHECKLIST.md`'s gate table cannot silently diverge. A new `pre-release` gate is added to the script in the same change, or to `_RELEASE_GATE_EXEMPT` with its reason. Because the script *blocks*, each gate must also be deterministic: a `uv run --with` overlay in a gate's recipe is pinned (`PYRIGHT_VERSION`, `REUSE_VERSION` in the Makefile header), since an unpinned overlay resolves the newest release on every run and a tool update alone could turn a release red; the same group flags an unpinned `--with` in any gate recipe. Pins are bumped deliberately in their own commit, and the header keeps the no-`[dependency-groups] dev` rule (the overlays stay the one record). Known residue: `lint` runs the pacman-provided `ruff`, which is kept on `PATH` for editor/hook integration and so follows system upgrades. | +| 27 | Release-gate completeness | Static-analysis gates of `make pre-release` vs the `tools/release.sh` preflight | enforced | Every `pre-release` prerequisite except the full `test` suite, the `e2e` tier and the live-mirror tier `test-live` (the heavier tiers, by design; `e2e` also needs podman and the network, and `test-live` reads the repos as they are today, so neither could be deterministic) is also run by the release preflight, which fails the release on it. `pre-release` is advisory and the script is the gate a release cannot skip, so a check only the former runs is one a tag can ship without: `v3.2.0` shipped four pyright errors — one a live crash (`3.2.0-B30`) the type-checker had named weeks earlier — and the 3.3.0 cycle reached eighteen (`3.2.0-STD1`). `check_standards` `release_gates` group diffs the `pre-release:` prerequisite list against the targets `release.sh` invokes, so `docs/RELEASE-CHECKLIST.md`'s gate table cannot silently diverge. A new `pre-release` gate is added to the script in the same change, or to `_RELEASE_GATE_EXEMPT` with its reason. Because the script *blocks*, each gate must also be deterministic: a `uv run --with` overlay in a gate's recipe is pinned (`PYRIGHT_VERSION`, `REUSE_VERSION` in the Makefile header), since an unpinned overlay resolves the newest release on every run and a tool update alone could turn a release red; the same group flags an unpinned `--with` in any gate recipe. Pins are bumped deliberately in their own commit, and the header keeps the no-`[dependency-groups] dev` rule (the overlays stay the one record). Known residue: `lint` runs the pacman-provided `ruff`, which is kept on `PATH` for editor/hook integration and so follows system upgrades. | | 30 | No em-dashes in user-facing prose | `README.md`, `docs/guides/`, `docs/concepts/` (`3.5.0-DOC3`), the man page (`man/sysforge.1.scd.in` and the generated `man/sysforge.1`), `completions/`, and every non-docstring string literal under `sysforge/`: `--help` text, log and prompt messages, exception text, and the headers of files sysforge generates | enforced | Rewrite with a comma, colon, semicolon, period or parentheses as the sentence needs, never a blind substitution (`3.4.0-DOC2`). `check_standards` `em_dash` group: a line scan of the files and an AST scan of every string constant under `sysforge/`, so implicit concatenation and f-string pieces are covered without importing `sysforge`. Exempt: docstrings (developer prose), a literal that is only the glyph (an empty-cell placeholder, or a key in `log`'s ASCII glyph-fallback table), ROADMAP and release-note entries, whose ``- **`<ID>` — <title>.**`` lead is a lint-enforced format (changing it is its own `STD`), internal docs, shipped-config and hook comments, and code comments. `tests/test_standards_compliance.py` (`test_em_dash_*`). | | 31 | Subprocess goes through `primitives/run` | Every external command `sysforge/` runs | enforced | Numbered 28 until `3.4.0-STD2`, when it was found sharing that number with the `pam.d(5)` row, which took 28 first and keeps it. ruff `banned-api` (TID251, `pyproject.toml`) bans `subprocess.run`/`Popen`/`call`/`check_call`/`check_output` outside the seam modules `primitives/run.py`, `pty_runner.py`, `privilege.py` and `sudo_session.py` (`3.2.0-F13`). Probes use `run.capture` (missing binary → `None`) or `run.probe` (missing binary → status 127), raise-on-failure calls use `run_or_raise`, and escalation uses the §22 privilege seam. A call that legitimately stays raw (inherits the TTY, streams, needs its own `preexec_fn`, or is a privileged command whose status or `CalledProcessError` its caller inspects) carries `# noqa: TID251 — <why>`, and the reason is mandatory. Enforced by the ruff rule plus `tests/test_standards_compliance.py` (`test_subprocess_fence_*`), which pins the banned set, the exemption list and the per-call reasons. Complements row 17, which governs argv form; this row governs *where* the call is made. | | 32 | Machine-readable output contract (`--json`) | `state list --json`, `doctor` (`system`, `pkg`, bare) `--json`, `update --dry-run --json`, `config check --json` (`3.5.0-F18`) | enforced | `primitives/json_out.py` is the one home (`3.4.0-F7`). Stdout carries exactly one JSON document ([RFC 8259](https://www.rfc-editor.org/rfc/rfc8259)) and nothing else: `cli.main` enters `json_out.reserved_stdout()` right after parsing and `verbs.runner.run_verb` enters it again (re-entrant), so every `ui()`/`print()`, startup logs and progress phases included, goes to stderr; the document goes out through `json_out.emit`. Every document has a top-level `"schema"` integer (`SCHEMA_VERSION`, now 1) and a `"kind"` (`state-list`, `doctor`, `update-dry-run`). **Adding a field is a minor change; renaming or removing one, or changing its type, is a major change and bumps `SCHEMA_VERSION`.** Each document is built from the result objects the human renderer reads (`AxisResult` via `diagnostics.axis_document`, `run.results` and the drift lists via `update._dry_run_document`, build_state rows through an explicit field list in `state_cmd._JSON_ENTRY_FIELDS`), so the two outputs cannot disagree. Exit codes are unchanged by `--json`. Mutating runs are out of scope: `update --json` without `--dry-run` and `doctor pkg --apply --json` are refused (exit 2). The schemas are committed as JSON Schema draft 2020-12 under `tests/data/json-schema/`; `tests/test_json_output.py` validates each verb's document against its schema (a subset validator, so the suite needs no new dependency), checks that stdout parses as one document with `log` in dry-run mode at `-vv`, and that a doctor skip appears under `roster.skipped`, never as a finding. A new field updates its schema in the same change. | diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index ed4a256c..6e37155b 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -234,6 +234,10 @@ https://keepachangelog.com/en/1.1.0/ --- - **`3.5.0-DEV23` — the end-to-end tier runs 35 of the 42 commands, and a new one cannot be missed.** Six scenarios join the four under `tools/e2e/`: maintaining a package and letting it go (`state list`, `resolve`, `log`, the `packages` commands, `state forget`, `uninstall`), `revert-to-stock`, keeping the config healthy (`config check`, `clean` and `reset`, and a `setup --preset` undone by `config reset`), the read-only reports changing nothing (`state`, `doctor`, `env`, `fetch`), a user script through the `artifact` commands, and `[aur] enabled = false` and `freeze_sources` refusing before any checkout, with `search` still answering from the repos. A seventh previews `run toolchain` and `run kernel` with non-default `toolchain.toml` and `kernel.toml`; it is marked to fail until their dry run shows something, a bug it found. `make e2e-coverage` lists which scenario runs each command, and `tests/test_e2e_coverage.py` fails the suite on a command that no scenario runs and `tools/e2e/uncovered.toml` does not excuse with a reason; seven are excused, such as `run pipeline`, which installs Arch from the live ISO. The `artifact` scenario also found that those commands crash in a user's first session when nothing has prepared the state directory yet; both bugs are on the roadmap. + +--- + +- **`3.5.0-DEV24` — `make test-live` checks the stock toolchain against the live repos, and a rubric says which tier a test belongs in.** The unit tests stub the repos, so they kept passing when Arch moved files into `clang-libs` (`3.5.0-B146`). `tests/test_live_stock_toolchain.py` runs the bench's real stock fetch, unpack and probe, checks that `libLLVM` and `libclang-cpp` load from the stock prefix and not the host, and compiles against the builtin headers; against the code before `3.5.0-B146` it fails all three ways. Its tests carry a new `network` marker and are skipped unless `SYSFORGE_LIVE_TESTS=1`, which `make test-live` sets, so `make test` and the CI gates stay offline. It runs in a `live` CI job, is in `CI_JOBS` so `make pre-release` runs it, and is exempt from the release preflight like `e2e`. DESIGN.md §Release Process now maps each boundary the unit suite stubs (root and the user, state between commands, a real terminal, the live repos) to the tier whose test a fix crossing it must add. --- - **`3.5.0-DOC1` — the package description says what sysforge optimizes.** The `pkgdesc` shown by `pacman -Si` and on the AUR now names what people search for in one short line: "AUR helper that builds and tracks packages tuned to your CPU with PGO, BOLT and LTO", instead of a generic "build and maintenance suite" line. The README's opening paragraph says the same. @@ -919,6 +923,10 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-B146` — `bench toolchain` works with Arch's new `clang-libs` package, and never times part of your build as stock.** From clang 23.1.3-4, Arch ships the compiler's builtin headers and `libclang-cpp` in a separate `clang-libs` package. The bench fetched only `clang` and `llvm-libs`, so its stock compiler stopped with `stddef.h` not found; before failing, it had also loaded your own `libclang-cpp` in place of the stock one. The bench now fetches `clang-libs` too, and still works against a repo from before the split. It also checks that the stock files it needs are all there before timing anything, and stops with a message naming what is missing, rather than measuring a stock compiler that is partly your own build. +--- + +- **`3.5.0-B147` — `run toolchain` can roll back the first build after Arch split out `clang-libs`.** From 23.1.3-4, Arch's LLVM build also produces `clang-libs`, which holds `libclang-cpp` and the compiler's builtin headers, and `clang` requires it at the same version. sysforge already built and installed it with `clang`. But if that first build then failed its post-install checks, putting the previous toolchain back failed: the old `clang` carries those same files, which the new `clang-libs` now owns, so pacman refused with file conflicts and left the half-replaced toolchain and a recovery command that hit the same wall. The rollback now removes `clang-libs` first, and the recovery command saved for an interrupted run reinstalls the toolchain from the repos, `clang-libs` included, so it never pairs your build with the stock `clang`. The post-install checks now also catch a `clang-libs` whose version lags the rest of the toolchain. The reminder `sysforge update` gives after a pacman transaction changes your toolchain now also fires for `clang-libs`, `polly` and `openmp`, which the hook did not watch. Upgrading the package installs the new hook; where it was installed some other way, `sysforge doctor` reports it as stale and `sysforge setup` refreshes it. + ## Security - **`3.5.0-B37` — the root pacman hook no longer follows links in the state directory.** The hook helper runs as root in every pacman transaction and wrote into `/var/lib/sysforge/sentinels`, whose parent any `sysforge` group member can write to. A member could swap `sentinels` for a symlink to another directory, which the next transaction made group-writable, or symlink one of its files so the hook appended to any file as root. The helper now enters the directory and acts only if it is the real `sentinels`, owned by root. It appends only to a regular file, opened so that a symlink is refused and a planted FIFO cannot stall the transaction. As before, it never fails a transaction. diff --git a/etc/pacman.d/hooks/sysforge-toolchain.hook b/etc/pacman.d/hooks/sysforge-toolchain.hook index 4ae3d59e..b4327edd 100644 --- a/etc/pacman.d/hooks/sysforge-toolchain.hook +++ b/etc/pacman.d/hooks/sysforge-toolchain.hook @@ -13,8 +13,11 @@ Type = Package Target = llvm Target = llvm-libs Target = clang +Target = clang-libs Target = lld Target = compiler-rt +Target = polly +Target = openmp Target = gcc Target = gcc-libs Target = lib32-llvm diff --git a/pyproject.toml b/pyproject.toml index 1dc1f339..5d9f86d8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,6 +47,7 @@ pythonpath = ["."] addopts = "-v" markers = [ "host_pacman: may run the real pacman (tests of the query functions); exempts the 3.5.0-DEV4 fence", + "network: reads the live Arch mirrors; skipped unless SYSFORGE_LIVE_TESTS=1 (`make test-live`)", ] [tool.pyright] diff --git a/sysforge/pipeline/stages/toolchain/gates.py b/sysforge/pipeline/stages/toolchain/gates.py index e7551a85..03a202e8 100644 --- a/sysforge/pipeline/stages/toolchain/gates.py +++ b/sysforge/pipeline/stages/toolchain/gates.py @@ -17,6 +17,7 @@ half-installed LLVM is worse than no new LLVM. """ from pathlib import Path +import subprocess from sysforge.primitives import toolchain_safety from sysforge.primitives.stage_sentinel import Recovery, sentinel_scope @@ -641,13 +642,34 @@ def snapshot_suite( return pacman.cached_pkg_files_for(installed) -def rollback_to_snapshot(snapshot: dict[str, "Path | None"]) -> bool: +def install_adds(built_map: dict[str, Path], install_set: "set[str]") -> tuple[str, ...]: + """The members the install puts on the system that are not installed now. + + The same selection ``profdata.pgo_install`` makes (3.5.0-F2), so a member a + selected package depends on counts: the first build after clang 23.1.3-4 + adds ``clang-libs``, which takes over files the installed clang owns + (3.5.0-B146). Sorted; empty when the build left no archives (a dry run). + """ + chosen = profdata.select_install_members( + profdata.collect_pgo_packages(built_map), set(install_set)) + names = {n for p in chosen if (n := pacman.read_pkgname_from_file(p))} + return tuple(sorted(n for n in names if pacman.get_installed_version(n) is None)) + + +def rollback_to_snapshot( + snapshot: dict[str, "Path | None"], *, added: "tuple[str, ...]" = (), +) -> bool: """Reinstall the snapshot's cached packages in one ``pacman -U``. Returns True when every captured package was reinstalled (live ``/usr`` is back to the prior-good set), False when the snapshot is incomplete (a member had no cached file) or the ``pacman -U`` itself failed — in which case the caller keeps the sentinel set with a recovery command. + + ``added`` (:func:`install_adds`) is removed first, with ``-Rdd`` since the + installed clang still depends on it: a member split out of an old package + owns files the old package's archive also carries, so ``pacman -U`` alone + stops on file conflicts (3.5.0-B146). A failed removal installs nothing. """ files = [p for p in snapshot.values() if p is not None] missing = [n for n, p in snapshot.items() if p is None] @@ -668,26 +690,45 @@ def rollback_to_snapshot(snapshot: dict[str, "Path | None"]) -> bool: "prior version to reinstall." ) return False + present = [n for n in added if pacman.get_installed_version(n) is not None] + if present: + _log.warn(f"Removing {', '.join(present)}, which this run added and the " + "prior toolchain does not have") + try: + pacman.remove_pkgs(present, nodeps=True) + except subprocess.CalledProcessError as e: + _log.error(f"Could not remove {', '.join(present)} ({e}); " + "not reinstalling the prior toolchain over it.") + return False _log.warn(f"Rolling back to {len(files)} cached package(s) via pacman -U") return pacman.batch_install_pkgs(files) -def snapshot_recovery(snapshot: dict[str, "Path | None"]) -> "Recovery": +def snapshot_recovery( + snapshot: dict[str, "Path | None"], *, added: "tuple[str, ...]" = (), +) -> "Recovery": """Recovery stored in the sentinel when rollback can't run cleanly. Prefers an offline ``pacman -U <cached files>`` when every member is cached; otherwise falls back to the online ``pacman -S <suite>``. Stored in the sentinel so the next-run recovery prompt has a restore to offer. + + When the install adds members (``added``), one stored command cannot remove + them before the downgrade, so it is the online reinstall, naming them too: + at the repo's version string a dependency resolve would keep the sysforge + copy beside the stock clang (3.5.0-B146). """ files = [p for p in snapshot.values() if p is not None] - if files and all(p is not None for p in snapshot.values()): + if not added and files and all(p is not None for p in snapshot.values()): return Recovery.pacman_upgrade(files) - return verify.llvm_recovery() + return verify.llvm_recovery(extra=added) -def snapshot_recovery_cmd(snapshot: dict[str, "Path | None"]) -> str: +def snapshot_recovery_cmd( + snapshot: dict[str, "Path | None"], *, added: "tuple[str, ...]" = (), +) -> str: """:func:`snapshot_recovery` as a shell line, for messages.""" - return snapshot_recovery(snapshot).display() + return snapshot_recovery(snapshot, added=added).display() def log_toolchain_resolution_summary( diff --git a/sysforge/pipeline/stages/toolchain/stage.py b/sysforge/pipeline/stages/toolchain/stage.py index ae973a81..0662c622 100644 --- a/sysforge/pipeline/stages/toolchain/stage.py +++ b/sysforge/pipeline/stages/toolchain/stage.py @@ -451,6 +451,11 @@ def _build_from_source(self, sysforge_config, state, options, tcfg, no_build): built_map, all_names, options, tcfg, dry_run=options.dry_run, ) + # Members this install adds (clang-libs, the first run after clang + # 23.1.3-4): rollback removes them first, and the stored recovery + # cannot, so it reinstalls from the repos (3.5.0-B146). + added = () if options.dry_run else gates.install_adds(built_map, install_set) + # sudo must be usable before the sentinel goes down: an unanswered # prompt means nothing ran, and a sentinel left for it would offer # a recovery that replaces the live toolchain (3.5.0-B30). @@ -477,7 +482,7 @@ def _build_from_source(self, sysforge_config, state, options, tcfg, no_build): else sentinel_scope( options.state_dir, "toolchain", - recovery=gates.snapshot_recovery(snapshot), + recovery=gates.snapshot_recovery(snapshot, added=added), retry_cmd=retry, compiler=compiler, pgo=pgo_enabled, @@ -568,7 +573,7 @@ def _build_from_source(self, sysforge_config, state, options, tcfg, no_build): _log.warn(f"Diagnostic evidence written to: {evidence_path}") _log.warn("Attempting automatic rollback to the prior-good toolchain…") progress.phase("rolling back to the prior toolchain") - if gates.rollback_to_snapshot(snapshot): + if gates.rollback_to_snapshot(snapshot, added=added): # Live /usr restored — the system is whole, so clear # the sentinel and raise (nothing to recover at next run). if sentinel is not None: # None only under --dry-run @@ -584,7 +589,7 @@ def _build_from_source(self, sysforge_config, state, options, tcfg, no_build): "[TOOLCHAIN] Built toolchain failed Gate-3 verification AND " "automatic rollback could not complete. The live toolchain may " "be inconsistent. Stage sentinel left in place; restore with: " - f"{gates.snapshot_recovery_cmd(snapshot)}" + f"{gates.snapshot_recovery_cmd(snapshot, added=added)}" ) # Verify passed — safe to wipe the stage2/stage3 prefixes # (PGO only). stage3 held the optimized libLLVM that the diff --git a/sysforge/pipeline/stages/toolchain/verify.py b/sysforge/pipeline/stages/toolchain/verify.py index ddb87e7a..915413c7 100644 --- a/sysforge/pipeline/stages/toolchain/verify.py +++ b/sysforge/pipeline/stages/toolchain/verify.py @@ -339,9 +339,14 @@ def verify_llvm_install( return issues -def llvm_recovery() -> "stage_sentinel.Recovery": - """The canonical pacman reinstall that restores a consistent LLVM set.""" - return stage_sentinel.Recovery.pacman_sync(_LLVM_VERSION_MATCH_SET) +def llvm_recovery(extra: "tuple[str, ...]" = ()) -> "stage_sentinel.Recovery": + """The canonical pacman reinstall that restores a consistent LLVM set. + + ``extra`` names more members to reinstall from the repos with it, in order + after the suite and once each. + """ + return stage_sentinel.Recovery.pacman_sync( + dict.fromkeys((*_LLVM_VERSION_MATCH_SET, *extra))) def llvm_recovery_command() -> str: diff --git a/sysforge/primitives/pacman.py b/sysforge/primitives/pacman.py index 8f3b7d24..1169a39f 100644 --- a/sysforge/primitives/pacman.py +++ b/sysforge/primitives/pacman.py @@ -889,17 +889,20 @@ def install_repo_pkgs(names: list) -> None: raise RuntimeError(f"repo install failed (exit {result.returncode})") -def remove_pkgs(names: list) -> None: +def remove_pkgs(names: list, *, nodeps: bool = False) -> None: """Remove packages via ``sudo pacman -R --noconfirm``. Used by ``revert-to-stock`` to drop a renamed optimized build (e.g. ``mesa-sysforge``) before reinstalling the stock package. No-op - on an empty list so callers need not guard. + on an empty list so callers need not guard. ``nodeps`` (``-Rdd``) removes a + package something installed still depends on, for a caller that reinstalls + the dependent's replacement right after (toolchain rollback, 3.5.0-B146). """ if not names: return + flag = "-Rdd" if nodeps else "-R" subprocess.run( # noqa: TID251 — privileged; CalledProcessError is this helper's contract - privileged_argv(["pacman", "-R", "--noconfirm", "--", *names]), + privileged_argv(["pacman", flag, "--noconfirm", "--", *names]), check=True, preexec_fn=lift_for_child, ) diff --git a/sysforge/primitives/toolchain_preflight.py b/sysforge/primitives/toolchain_preflight.py index 406b7bec..7a6668f9 100644 --- a/sysforge/primitives/toolchain_preflight.py +++ b/sysforge/primitives/toolchain_preflight.py @@ -296,8 +296,11 @@ def _probe_meson() -> ToolchainCheck: # (_LLVM_VERSION_MATCH_SET) imports it too. ``spirv-llvm-translator`` (its own # version scheme) and ``lib32-*`` (separate multilib lineage, may carry an # epoch) are deliberately excluded: they are not pkgver-locked to this set. +# ``clang-libs`` (libclang-cpp and the builtin headers) was split out of clang in +# 23.1.3-4, which depends on it at its exact version (3.5.0-B146); on a system +# from before the split it is simply not installed, which every reader allows. LLVM_LOCKSTEP_SUITE: tuple[str, ...] = ( - "llvm", "llvm-libs", "clang", "lld", "compiler-rt", "polly", "openmp", + "llvm", "llvm-libs", "clang", "clang-libs", "lld", "compiler-rt", "polly", "openmp", ) diff --git a/tests/conftest.py b/tests/conftest.py index 6de99e59..aff38501 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -40,9 +40,26 @@ # Show all log messages in tests so assertions against log output work. import sysforge.log as _sf_log + _sf_log.set_verbosity(2) +def pytest_collection_modifyitems(config, items): + """Skip ``network`` tests unless ``SYSFORGE_LIVE_TESTS=1`` (3.5.0-B146). + + They read the live Arch mirrors, so ``make test`` and the CI gates stay + offline; ``make test-live`` (its own CI job) sets the variable. An + environment switch rather than ``-m`` in ``addopts``: tools/ci.sh runs the + suite with ``-o addopts=``, which would drop a marker filter. + """ + if os.environ.get("SYSFORGE_LIVE_TESTS") == "1": + return + skip = pytest.mark.skip(reason="live mirror test: run with `make test-live`") + for item in items: + if item.get_closest_marker("network"): + item.add_marker(skip) + + REPO_ROOT = TESTS_DIR.parent diff --git a/tests/test_live_stock_toolchain.py b/tests/test_live_stock_toolchain.py new file mode 100644 index 00000000..1e9588c2 --- /dev/null +++ b/tests/test_live_stock_toolchain.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: 2026 Keith Raghubar +# +# SPDX-License-Identifier: MIT + +"""The stock toolchain ``bench toolchain`` compares against, from the live +mirrors (3.5.0-B146). + +The unit tests stub the repo, so they only check sysforge against what it +assumes the stock packages contain. Arch's clang 23.1.3-4 broke that +assumption (it moved libclang-cpp and the builtin headers to ``clang-libs``) +and every unit test still passed. This canary runs the bench's own fetch, +unpack and probe against the current repo, then checks the two things a +repackaging breaks: that the stock compiler loads its libraries from the +stock prefix, never the host's (which would time the sysforge build as +stock), and that it compiles with its builtin headers. + +Skipped by ``make test``; ``make test-live`` (its own CI job) runs it. It +downloads the stock packages unless the pacman cache already holds them. The +stock compiler needs llvm-libs' runtime dependencies on the host; CI installs +``llvm-libs`` for them, which is also the host copy the library check below +must not resolve to. +""" +from pathlib import Path + +import pytest + +from sysforge import bench_core +from sysforge.primitives import run, stock_pkg + +pytestmark = [pytest.mark.network, pytest.mark.host_pacman] + + +@pytest.fixture(scope="module") +def stock_prefix(tmp_path_factory) -> Path: + tmp = tmp_path_factory.mktemp("stock-toolchain") + try: + _pkgs, archives = bench_core._fetch_stock(tmp, offline=False) + except stock_pkg.StockUnavailable as exc: + pytest.fail(f"could not obtain the stock packages: {exc}") + prefix = tmp / "stock" + stock_pkg.extract(archives, prefix, downloaded_in=tmp) + return prefix + + +def test_the_stock_prefix_holds_everything_the_compiler_needs(stock_prefix): + stock_pkg.check_complete(stock_prefix) + bench_core._probe_stock(stock_prefix) + + +@pytest.mark.parametrize("lib", ["libLLVM", "libclang-cpp"]) +def test_the_stock_compiler_loads_its_libraries_from_the_prefix(stock_prefix, lib): + res = run.probe(["ldd", str(stock_prefix / "usr/bin/clang")], + env=bench_core._stock_env(stock_prefix)) + assert res.returncode == 0, res.stderr + lines = [ln for ln in res.stdout.splitlines() if f"{lib}.so" in ln] + assert lines, f"stock clang does not link {lib}:\n{res.stdout}" + assert all(str(stock_prefix) in ln for ln in lines), \ + f"{lib} resolves outside the stock prefix:\n" + "\n".join(lines) + + +def test_the_stock_compiler_compiles_with_its_builtin_headers(stock_prefix, tmp_path): + src = tmp_path / "builtins.c" + src.write_text("#include <stddef.h>\n#include <stdint.h>\n#include <stdarg.h>\n" + "size_t width(void) { return sizeof(uint64_t); }\n") + res = run.probe([str(stock_prefix / "usr/bin/clang"), "-O2", "-c", str(src), + "-o", str(tmp_path / "builtins.o")], + env=bench_core._stock_env(stock_prefix)) + assert res.returncode == 0, res.stderr diff --git a/tests/test_pacman_hooks.py b/tests/test_pacman_hooks.py index 61f2fa73..442e6b61 100644 --- a/tests/test_pacman_hooks.py +++ b/tests/test_pacman_hooks.py @@ -161,3 +161,16 @@ def test_helper_refuses_path_kind(tmp_path): d = tmp_path / "sentinels" assert _run_helper(d, kind="../escape").returncode == 0 assert not (tmp_path / "escape").exists() + + +def test_toolchain_hook_targets_every_lockstep_suite_member(): + """3.5.0-B147: a hand-kept Target list missed clang-libs when clang 23.1.3-4 + split it out. The hook watches every member of the suite that moves + together, so a new member joins it from the one list.""" + from sysforge.primitives.toolchain_preflight import LLVM_LOCKSTEP_SUITE + hook = next(a for a in pacman_hooks.shipped_sources() + if a.name == "sysforge-toolchain.hook") + text = hook.content.decode() if isinstance(hook.content, bytes) else hook.content + targets = {line.split("=", 1)[1].strip() for line in text.splitlines() + if line.startswith("Target")} + assert set(LLVM_LOCKSTEP_SUITE) <= targets, set(LLVM_LOCKSTEP_SUITE) - targets diff --git a/tests/test_pacman_revert_helpers.py b/tests/test_pacman_revert_helpers.py index d076c944..fdf9478e 100644 --- a/tests/test_pacman_revert_helpers.py +++ b/tests/test_pacman_revert_helpers.py @@ -60,3 +60,14 @@ def test_reinstall_repo_pkgs_plain_keeps_safe_conflict_default(): run.return_value.returncode = 0 pacman.reinstall_repo_pkgs(["mesa"]) assert "--ask=4" not in run.call_args[0][0] + + +def test_remove_pkgs_nodeps_skips_dependency_checks(): + """3.5.0-B146: toolchain rollback drops a member the installed clang still + depends on, just before the old clang takes its files back.""" + with patch("sysforge.primitives.pacman.subprocess.run") as run: + run.return_value.returncode = 0 + pacman.remove_pkgs(["clang-libs"], nodeps=True) + argv = run.call_args[0][0] + assert argv[:3] == ["sudo", "pacman", "-Rdd"] + assert argv[-2:] == ["--", "clang-libs"] diff --git a/tests/test_stage_toolchain.py b/tests/test_stage_toolchain.py index bafcc79f..9ee7fab4 100644 --- a/tests/test_stage_toolchain.py +++ b/tests/test_stage_toolchain.py @@ -1543,3 +1543,49 @@ def test_healable_drift_line_names_the_manual_rebuild_when_nothing_is_queued(): assert "queued consumer rebuild" in healable_drift_line("m", queued=True) off = healable_drift_line("m", queued=False) assert "no consumer rebuild is queued" in off and "sysforge build" in off + + +def _gate3_fails_with_added_member(tmp_path, monkeypatch, *, restore_ok): + """Gate 3 fails after an install that added clang-libs (3.5.0-B146).""" + toml_path, state, config, options = _single_pass_setup(tmp_path) + cached = tmp_path / "cache" / "clang-23.1.3-3-x86_64.pkg.tar.zst" + cached.parent.mkdir(parents=True) + cached.touch() + monkeypatch.setattr("sysforge.primitives.pacman.cached_pkg_files_for", + lambda names: {n: cached for n in names}) + monkeypatch.setattr("sysforge.primitives.pacman.get_installed_version", + lambda name: "1") + monkeypatch.setattr("sysforge.pipeline.stages.toolchain.gates.install_adds", + lambda built_map, install_set: ("clang-libs",)) + calls = [] + with patch("sysforge.pipeline.stages.toolchain.config.TOOLCHAIN_PATH", toml_path), \ + patch("sysforge.pipeline.stages.toolchain.passes.makepkg_run"), \ + patch("sysforge.pipeline.stages.toolchain.pkgbuilds.sync_pkgbuild_dirs"), \ + patch("sysforge.pipeline.stages.toolchain.profdata.pgo_install"), \ + patch("sysforge.pipeline.stages.toolchain.verify.verify_llvm_install", + return_value=["clang --version: exit 127"]), \ + patch("sysforge.primitives.pacman.remove_pkgs", + side_effect=lambda names, nodeps=False: calls.append(("remove", names, nodeps))), \ + patch("sysforge.primitives.pacman.batch_install_pkgs", + side_effect=lambda files: calls.append(("install",)) or restore_ok), \ + patch("sys.stdin.isatty", return_value=False): + with pytest.raises(RuntimeError) as exc: + ToolchainStage().run(config, state, options) + return calls, str(exc.value) + + +def test_gate3_rollback_removes_a_member_the_install_added(tmp_path, monkeypatch): + calls, msg = _gate3_fails_with_added_member(tmp_path, monkeypatch, restore_ok=True) + assert "prior toolchain was restored" in msg + assert calls == [("remove", ["clang-libs"], True), ("install",)] + assert not _sentinel_exists(tmp_path / "state") + + +def test_failed_rollback_with_an_added_member_stores_the_online_recovery( + tmp_path, monkeypatch): + from sysforge.primitives.stage_sentinel import StageSentinel + _, msg = _gate3_fails_with_added_member(tmp_path, monkeypatch, restore_ok=False) + record = StageSentinel(tmp_path / "state").get_active() + assert record["recovery_kind"] == "pacman_sync" + assert "clang-libs" in record["recovery_args"] + assert "pacman -S" in msg and "clang-libs" in msg diff --git a/tests/test_toolchain_gates.py b/tests/test_toolchain_gates.py index 63d317bd..2e59dae0 100644 --- a/tests/test_toolchain_gates.py +++ b/tests/test_toolchain_gates.py @@ -236,3 +236,104 @@ def test_empty_snapshot_rollback_says_why(monkeypatch, capsys): err = capsys.readouterr().err assert "snapshot is empty" in err assert "none of the toolchain packages was installed before this run" in err + + +# --- a member the install adds (clang-libs, 3.5.0-B146) --------------------- + + +def _added_setup(tmp_path, monkeypatch, installed): + """llvm's build produced clang, clang-libs and llvm-libs; clang depends on + clang-libs. ``installed`` is what pacman reports now.""" + from sysforge.pipeline.stages.toolchain import gates, profdata + from sysforge.primitives import pacman + files = {n: tmp_path / f"{n}-23.1.3-4-x86_64.pkg.tar.zst" + for n in ("clang", "clang-libs", "llvm-libs", "wasi-libc++")} + monkeypatch.setattr(profdata, "collect_pgo_packages", lambda m: list(files.values())) + monkeypatch.setattr(pacman, "read_pkgname_from_file", + lambda p: next(n for n, f in files.items() if f == p)) + deps = {"clang": {"clang-libs", "llvm-libs"}} + monkeypatch.setattr(pacman, "read_pkginfo_relations", lambda p: { + "depend": deps.get(pacman.read_pkgname_from_file(p), set()), "provides": set()}) + monkeypatch.setattr(pacman, "get_installed_version", installed.get) + return gates + + +def test_install_adds_names_the_members_not_installed_yet(tmp_path, monkeypatch): + gates = _added_setup(tmp_path, monkeypatch, + {"clang": "23.1.3-3", "llvm-libs": "23.1.3-3"}) + built = {"clang": tmp_path / "llvm" / "PKGBUILD"} + assert gates.install_adds(built, {"clang", "llvm-libs"}) == ("clang-libs",) + + +def test_install_adds_nothing_once_the_member_is_installed(tmp_path, monkeypatch): + gates = _added_setup(tmp_path, monkeypatch, { + "clang": "23.1.3-4", "clang-libs": "23.1.3-4", "llvm-libs": "23.1.3-4"}) + built = {"clang": tmp_path / "llvm" / "PKGBUILD"} + assert gates.install_adds(built, {"clang", "clang-libs", "llvm-libs"}) == () + + +def test_rollback_removes_an_added_member_before_reinstalling(tmp_path, monkeypatch): + """The cached clang 23.1.3-3 owns libclang-cpp and the builtin headers, + which the installed clang-libs now owns: pacman -U alone hits file + conflicts, so the member the install added goes first.""" + from sysforge.pipeline.stages.toolchain import gates + from sysforge.primitives import pacman + calls = [] + monkeypatch.setattr(pacman, "get_installed_version", {"clang-libs": "23.1.3-4"}.get) + monkeypatch.setattr(pacman, "remove_pkgs", + lambda names, nodeps=False: calls.append(("remove", names, nodeps))) + monkeypatch.setattr(pacman, "batch_install_pkgs", + lambda files: calls.append(("install", files)) or True) + snap = {"clang": tmp_path / "clang-23.1.3-3.pkg.tar.zst"} + assert gates.rollback_to_snapshot(snap, added=("clang-libs",)) is True + assert calls == [("remove", ["clang-libs"], True), + ("install", [tmp_path / "clang-23.1.3-3.pkg.tar.zst"])] + + +def test_rollback_skips_an_added_member_that_is_not_installed(tmp_path, monkeypatch): + from sysforge.pipeline.stages.toolchain import gates + from sysforge.primitives import pacman + removed = [] + monkeypatch.setattr(pacman, "get_installed_version", {}.get) + monkeypatch.setattr(pacman, "remove_pkgs", + lambda names, nodeps=False: removed.append(names)) + monkeypatch.setattr(pacman, "batch_install_pkgs", lambda files: True) + assert gates.rollback_to_snapshot({"clang": tmp_path / "c.pkg.tar.zst"}, + added=("clang-libs",)) is True + assert removed == [] + + +def test_rollback_stops_when_the_added_member_cannot_be_removed(tmp_path, monkeypatch): + import subprocess + from sysforge.pipeline.stages.toolchain import gates + from sysforge.primitives import pacman + installs = [] + + def remove(names, nodeps=False): + raise subprocess.CalledProcessError(1, ["pacman", "-Rdd"]) + + monkeypatch.setattr(pacman, "get_installed_version", {"clang-libs": "23.1.3-4"}.get) + monkeypatch.setattr(pacman, "remove_pkgs", remove) + monkeypatch.setattr(pacman, "batch_install_pkgs", lambda files: installs.append(files)) + assert gates.rollback_to_snapshot({"clang": tmp_path / "c.pkg.tar.zst"}, + added=("clang-libs",)) is False + assert installs == [] + + +def test_recovery_goes_online_when_the_install_adds_a_member(tmp_path): + """One stored command cannot remove clang-libs and downgrade clang, so the + recovery reinstalls the repo suite, clang-libs named explicitly: at the + same version string a dependency resolve would keep the sysforge copy.""" + from sysforge.pipeline.stages.toolchain import gates + from sysforge.primitives import stage_sentinel + snap = {"clang": tmp_path / "clang-23.1.3-3.pkg.tar.zst"} + rec = gates.snapshot_recovery(snap, added=("clang-libs",)) + assert rec.kind == stage_sentinel.RECOVERY_PACMAN_SYNC + assert {"clang", "clang-libs", "llvm-libs"} <= set(rec.args) + + +def test_recovery_stays_offline_when_nothing_is_added(tmp_path): + from sysforge.pipeline.stages.toolchain import gates + from sysforge.primitives import stage_sentinel + snap = {"clang": tmp_path / "clang-23.1.3-3.pkg.tar.zst"} + assert gates.snapshot_recovery(snap).kind == stage_sentinel.RECOVERY_PACMAN_UPGRADE diff --git a/tests/test_toolchain_preflight.py b/tests/test_toolchain_preflight.py index 4d630779..e27b3e8a 100644 --- a/tests/test_toolchain_preflight.py +++ b/tests/test_toolchain_preflight.py @@ -281,7 +281,8 @@ def test_skew_check_carries_structured_version_rows(monkeypatch): """_probe_cc populates `versions` so the renderer, not the probe, formats. clang/lld lag behind llvm/llvm-libs; every lagging group targets the newest - installed pkgver. + installed pkgver. clang-libs is pinned to clang's exact version, so it lags + with clang (3.5.0-B147). """ monkeypatch.setattr("shutil.which", lambda c: f"/usr/bin/{c}") @@ -289,14 +290,14 @@ def fake_run(cmd, *a, **kw): if cmd[:2] == ["clang", "--version"]: return _FakeResult(0, "clang version 22.1.5\n", "") if cmd[:2] == ["pacman", "-Q"]: - ver = "22.1.5-1" if cmd[2] in ("clang", "lld") else "22.1.6-1" + ver = "22.1.5-1" if cmd[2] in ("clang", "clang-libs", "lld") else "22.1.6-1" return _FakeResult(0, f"{cmd[2]} {ver}\n", "") return _FakeResult(0, "", "") monkeypatch.setattr(subprocess, "run", fake_run) c = run_preflight(frozenset({"cc:clang"})).failed[0] rows = dict((label, (old, new)) for label, old, new in c.versions) - assert rows["clang/lld"] == ("22.1.5", "22.1.6") + assert rows["clang/clang-libs/lld"] == ("22.1.5", "22.1.6") # The newest group is its own target. assert rows["compiler-rt/llvm/llvm-libs/openmp/polly"] == ("22.1.6", "22.1.6") diff --git a/tests/test_toolchain_verify.py b/tests/test_toolchain_verify.py index d63b4512..874ca0b8 100644 --- a/tests/test_toolchain_verify.py +++ b/tests/test_toolchain_verify.py @@ -411,3 +411,24 @@ def fake_run(cmd, **kwargs): issues = verify_llvm_install() assert any("-fuse-ld=lld" in c for c in seen) assert any("clang compile and lld link" in i for i in issues) + + +def test_verify_llvm_install_flags_a_lagging_clang_libs(): + """3.5.0-B146: clang depends on clang-libs at its exact version, and + clang-libs holds libclang-cpp: a clang-libs left behind is suite skew.""" + from sysforge.pipeline.stages.toolchain import verify_llvm_install + + versions = {"llvm": "23.1.4-1", "llvm-libs": "23.1.4-1", "clang": "23.1.4-1", + "clang-libs": "23.1.3-4", "lld": "23.1.4-1", "compiler-rt": "23.1.4-1"} + pacman_output = "\n".join(f"{n} {v}" for n, v in versions.items()) + + def fake_run(cmd, **kwargs): + if cmd[0] == "pacman": + return MagicMock(returncode=0, stdout=pacman_output, stderr="") + if cmd[0] == "llvm-config": + return MagicMock(returncode=0, stdout="X86 AMDGPU NVPTX\n", stderr="") + return MagicMock(returncode=0, stdout="ok", stderr="") + + with patch("sysforge.primitives.run.capture", side_effect=fake_run): + issues = verify_llvm_install(expected_targets=["X86"]) + assert any("clang-libs" in i for i in issues), issues diff --git a/tools/check_standards.py b/tools/check_standards.py index ac64eb9e..1bd9fbad 100644 --- a/tools/check_standards.py +++ b/tools/check_standards.py @@ -1803,7 +1803,9 @@ def check_semver_bump(repo: Path, # suite is minutes, not seconds (docs/RELEASE-CHECKLIST.md "Which command # does what"), and the end-to-end tier (3.5.0-DEV21) is minutes too, needs # podman, and fetches from the AUR, so it could not be a deterministic gate. -_RELEASE_GATE_EXEMPT = {"test", "e2e"} +# The live-mirror tier (`test-live`, 3.5.0-DEV24) reads whatever the repos +# hold today, which is its point, and so is not deterministic either. +_RELEASE_GATE_EXEMPT = {"test", "e2e", "test-live"} _PRE_RELEASE_RE = re.compile(r"^pre-release:([^#\n]*)", re.MULTILINE) _RELEASE_MAKE_RE = re.compile(r"\bmake\s+(?:--no-print-directory\s+)?([a-z][\w-]*)") _MAKE_VAR_REF_RE = re.compile(r"\$\((\w+)\)") diff --git a/tools/e2e/README.md b/tools/e2e/README.md index 5bafc121..562bbcc3 100644 --- a/tools/e2e/README.md +++ b/tools/e2e/README.md @@ -59,6 +59,10 @@ clone). `pkgs/tree` is a stand-in named after a repo package, which puts `build` on its repo opt-in path without fetching Arch's packaging repo. +A scenario can't catch a change in what the live repos contain, because `pkgs/` stands in for +them. That is the live-mirror tier's job (`make test-live`, `tests/test_live_stock_toolchain.py`). +DESIGN.md §Release Process has the rubric for which tier a test belongs in. + ## Coverage Every command in `tests/data/surface.json` is either run by a scenario or listed in From 9e8f20b13e0fab341ae464850ca69298295a3ba8 Mon Sep 17 00:00:00 2001 From: Keith Raghubar <KeithRaghubar@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:51:59 -0400 Subject: [PATCH 3/3] File e2e coverage gaps from the usage and bug-history audit (3.5.0-DEV25 to DEV34, Q1) Toolchain to kernel handoff and the real chain (DEV25, DEV26), whether to measure a sysforge-toolchain kernel against stock (Q1), and eight scenarios the audit found uncovered: update --devel, stock install over a sysforge build, the build sandbox in the VM tier, config merge, --interactive, package PGO, unattended runs and the --frozen/--thaw flags (DEV27 to DEV34). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- ROADMAP.md | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index fdacb539..dae97d59 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -112,6 +112,17 @@ canonical ordering. <!-- BEGIN roadmap-table (generated by tools/gen_roadmap_table.py — do not edit by hand; run `make roadmap-table`) --> | ID | Item | Priority | Effort | Bump | |----|------|----------|--------|------| +| `3.5.0-DEV27` | an end-to-end scenario for update --devel on a -git package | high | medium | patch | +| `3.5.0-DEV25` | an end-to-end scenario for the toolchain → kernel handoff | med | small | patch | +| `3.5.0-DEV28` | an end-to-end scenario for a stock package installed over a sysforge build | med | small | patch | +| `3.5.0-DEV30` | an end-to-end scenario for config merge | med | small | patch | +| `3.5.0-DEV31` | an end-to-end scenario for build --interactive and update --interactive | med | small | patch | +| `3.5.0-DEV29` | the build sandbox end to end, in the VM tier | med | large | patch | +| `3.5.0-DEV33` | an end-to-end scenario for unattended runs | low | small | patch | +| `3.5.0-DEV34` | an end-to-end scenario for the --frozen and --thaw flags | low | small | patch | +| `3.5.0-Q1` | should sysforge measure a kernel built with its toolchain against one built with stock clang? | low | small | patch | +| `3.5.0-DEV26` | a release-time run of the real toolchain → kernel chain on real hardware | low | medium | patch | +| `3.5.0-DEV32` | an end-to-end scenario for package PGO (build --pgo) | low | medium | patch | <!-- END roadmap-table --> ### Features @@ -120,4 +131,145 @@ canonical ordering. ### Developer tooling +- **`3.5.0-DEV25` — an end-to-end scenario for the toolchain → kernel handoff.** `95-stage-dry-run` + previews `run toolchain` and `run kernel` separately, so nothing checks what one leaves for the + other: the compiler, its paths and its variant in `pipeline_state.toml`, which the kernel stage + (and `packages`) read. A scenario under `tools/e2e/scenarios/` runs a real `run toolchain` with + `toolchain.toml compiler = "llvm"` and `skip_build = true` (it registers the container's clang + without building, in seconds), asserts the stage result it saved, then asserts `run kernel + --dry-run` plans with that compiler (`compiler: llvm (from …)`, `cc=/usr/bin/clang`, + `variant: stock_llvm`), and again after switching back to `compiler = "gcc"`. A real kernel build + in the container is a follow-up only if a spike shows it fits: a `tinyconfig` build is minutes, + but the stage's boot-readiness check expects mkinitcpio and a bootloader the container lacks + (`state boot-entries` is already excused in `tools/e2e/uncovered.toml` for that reason). Tier + rubric row "state one command leaves for the next" (§Release Process). + *Priority: med · Effort: small · Bump: patch* — the cross-stage contract behind every LLVM kernel + build is untested end to end; the scenario is seconds long. + +--- + +- **`3.5.0-DEV26` — a release-time run of the real toolchain → kernel chain on real hardware.** + `DEV25` checks the handoff with a registered clang; what only a real build shows is that the kernel + is compiled by the toolchain sysforge just built and is recorded that way. A script under `tools/` + (and a Make target, with a step in `docs/RELEASE-CHECKLIST.md` for minor and major releases) runs + `run toolchain` then `run kernel` on the workstation, and asserts on provenance, never on speed: + the built `vmlinux`'s `.comment` section names the clang version `build_state.toml` recorded for + the toolchain; the kernel's `build_state` entry carries that toolchain's variant (`pgo_llvm`); and + `update --dry-run` reports no toolchain drift for it. Then the drift leg: with a stock clang + installed from the pacman cache, `update --dry-run` reports the kernel as drifted, and the script + puts the sysforge toolchain back from its cached archives (`run toolchain --install-only`) before + it exits, on success or failure. Not CI and not the VM tier: the last PGO toolchain build here took + about 3.2 hours, and the VM (4 cores, 4 GiB, `VM_DISK_SIZE` 40G) cannot link libLLVM, though its + real bootloader would suit a kernel-only leg if one is wanted later. Kernel half of the rubric row + "state one command leaves for the next" (§Release Process). + *Priority: low · Effort: medium · Bump: patch* — hours long and run by hand, so a release + checklist item; it catches a chain the CI scenario can only stand in for. + +--- + +- **`3.5.0-DEV27` — an end-to-end scenario for `update --devel` on a `-git` package.** `update --devel` is + the most-run flag in the maintainer's own use and no scenario passes it; about ten 3.5.0 fixes were + VCS-package bugs (`3.5.0-B144` among the latest). A stand-in under `tools/e2e/pkgs/` named `*-git`, + whose `source=()` is a `git+file://` repository the scenario creates, keeps it off the network: + build it, then `update --devel` with no new commit reports it current and rebuilds nothing; after a + commit to that repository it reports the new `pkgver` and rebuilds; `build_state.toml` records the + commit it built. The AUR-repo half of `3.5.0-B144` (an AUR packaging repo fetched only when the AUR + says it changed) needs a real AUR package, so it belongs in `10-quickstart`'s networked run if it is + covered at all. + *Priority: high · Effort: medium · Bump: patch* — the maintainer runs this daily and it has the densest recent bug history of any untested path. + +--- + +- **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** A + user's own `pacman -S` of a package sysforge built from source should demote its record to a plain + `pacman` marker on the next `update` (`primitives/install_reconcile.py`, through the + `sysforge-buildstate.hook` sentinel), and a toolchain package changed outside sysforge should leave + the toolchain reminder (`sysforge-toolchain.hook`). The hooks run inside every scenario, but nothing + asserts their effect, and the ordering between the hook's events and sysforge's own installs has + needed two fixes (`3.5.0-B64`, `3.5.0-B120`). The scenario builds `pkgs/tree` from source, runs + `sudo pacman -S tree`, then asserts `update` demotes it and does not rebuild it; then `sudo pacman -S + lld` and asserts the reminder appears once and the sentinel is consumed. Needs the mirrors, as + `60-revert-to-stock` already does. + *Priority: med · Effort: small · Bump: patch* — a documented behaviour with a bug history and no end-to-end check; one short scenario. + +--- + +- **`3.5.0-DEV29` — the build sandbox end to end, in the VM tier.** `[security] sandbox_builds = true` + (clean chroot, local repo) is how every build on the maintainer's system runs, about ten 3.5.0 + fixes were sandbox bugs, and no scenario enables it. It cannot run in the container tier: + `makechrootpkg` needs `systemd-nspawn`, which an unprivileged podman or Actions container cannot + start. The VM has full systemd, so the e2e scenarios gain a sandbox mode run there + (`tools/e2e/run.sh --here` inside the VM, with `sandbox_builds` on): at least an AUR build, a build + whose dependency is a sysforge build served from the local repo, and a `-git` build, asserting the + package came out of the chroot and that `build_state.toml` records the sandbox toolchain. A + release-checklist step, beside `make vm-test`. + *Priority: med · Effort: large · Bump: patch* — the default build path on the maintainer's system is untested end to end, but it needs VM plumbing. + +--- + +- **`3.5.0-DEV30` — an end-to-end scenario for `config merge`.** `config merge` is excused in + `tools/e2e/uncovered.toml` because it "needs a .sfnew and an interactive merge tool", but the tool is + `sysforge.toml [ui] merge`, which a scenario can set to a script. The scenario leaves a `.sfnew` + beside an edited config (as a package upgrade would), sets `merge` to a script that takes the new + file's added key and keeps the user's edit, runs `config merge`, and asserts the merged file, that + the `.sfnew` is gone and that `config check` is clean. The excusal comes out of `uncovered.toml` in + the same change. + *Priority: med · Effort: small · Bump: patch* — a command the maintainer runs after upgrades, excused for a reason that does not hold. + +--- + +- **`3.5.0-DEV31` — an end-to-end scenario for `build --interactive` and `update --interactive`.** `--interactive` + runs makepkg without `--noconfirm` and hands it the terminal, so pacman's own questions (a package + conflict, a provider choice) reach the user instead of failing under `--noconfirm` (`3.3.0-B6`). No + scenario passes it, and prompts and terminal handling account for about seventeen 3.5.0 fixes. A + stand-in package that conflicts with an installed one, built with `--interactive` and the conflict + question answered through `tools/e2e/answer.py`, asserts the swap happened; the same without the + flag asserts the build stops and installs nothing. + *Priority: med · Effort: small · Bump: patch* — a frequently used mode whose bug class is the largest after the two stages. + +--- + +- **`3.5.0-DEV32` — an end-to-end scenario for package PGO (`build --pgo`).** `docs/guides/pgo.md` + documents an instrumented build, a training run and a rebuild with the profile; the e2e tier runs + none of it. A stand-in C package whose program has a hot path the training run exercises: `build + --pgo` (instrument), run it, rebuild with the profile, then assert the profile store holds its data, + `state profiles` lists it, and the second build used it (the recorded flags). Asserts on what was + recorded and used, not on speed. + *Priority: low · Effort: medium · Bump: patch* — a documented guide path, used occasionally; the stand-in package is the work. + +--- + +- **`3.5.0-DEV33` — an end-to-end scenario for unattended runs.** `--unattended` and `update --yes` + (`3.5.0-F20`) are for timers and scripts: they must never block on a prompt and must refuse what + needs consent instead of assuming it. The scenario runs `update --unattended` with stdin closed and + no terminal, against a state that would prompt (a repo package awaiting opt-in, a PKGBUILD change + awaiting review), and asserts it exits within a timeout, installs nothing it needed consent for, and + says why; then `uninstall --yes` completes without a terminal. + *Priority: low · Effort: small · Bump: patch* — new and documented in the kernel guide; the maintainer runs no timers, so lower value here. + +--- + +- **`3.5.0-DEV34` — an end-to-end scenario for the `--frozen` and `--thaw` flags.** `90-source-policy` + covers `freeze_sources` in config but not the flags `docs/guides/freeze.md` documents for a single + run. Add to it: `--frozen` refuses a clone with the config off, `--no-frozen` lets one through with + it on, and `--thaw` behaves as the guide says, each asserting no checkout appears when refused. + *Priority: low · Effort: small · Bump: patch* — documented flags on an already-covered command; a few lines in an existing scenario. + +--- + ### Open questions + +- **`3.5.0-Q1` — should sysforge measure a kernel built with its toolchain against one built with + stock clang?** `bench toolchain` times the compiler itself; a user who builds the toolchain to get + a faster kernel has no sysforge answer to whether the kernel got faster, or whether its build did. + The two readings are different features. Build time (the kernel compiled by each clang, interleaved + as the toolchain bench does) fits `bench`'s model: read-only, the stock clang unpacked to a temp + dir. Runtime (the kernel *running* faster) needs a reboot into each kernel and a workload, which is + outside anything sysforge does unattended. Either way it is a measurement with no pass/fail + threshold (a cutoff tuned on one machine means nothing on another), so it is not a test; the + provenance checks belong to `DEV26`. Resolve by choosing build time, runtime, or neither + (a measurement left to the user, as `kernel-bench.sh` is today). + *Priority: low · Effort: small · Bump: patch* — a decision, not an implementation; the effort is the + spike that informs it. + +---