diff --git a/ROADMAP.md b/ROADMAP.md index dae97d59..274776c5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -112,7 +112,6 @@ canonical ordering. | ID | Item | Priority | Effort | Bump | |----|------|----------|--------|------| -| `3.5.0-DEV27` | an end-to-end scenario for update --devel on a -git package | high | medium | patch | | `3.5.0-DEV25` | an end-to-end scenario for the toolchain → kernel handoff | med | small | patch | | `3.5.0-DEV28` | an end-to-end scenario for a stock package installed over a sysforge build | med | small | patch | | `3.5.0-DEV30` | an end-to-end scenario for config merge | med | small | patch | @@ -120,13 +119,41 @@ canonical ordering. | `3.5.0-DEV29` | the build sandbox end to end, in the VM tier | med | large | patch | | `3.5.0-DEV33` | an end-to-end scenario for unattended runs | low | small | patch | | `3.5.0-DEV34` | an end-to-end scenario for the --frozen and --thaw flags | low | small | patch | -| `3.5.0-Q1` | should sysforge measure a kernel built with its toolchain against one built with stock clang? | low | small | patch | | `3.5.0-DEV26` | a release-time run of the real toolchain → kernel chain on real hardware | low | medium | patch | | `3.5.0-DEV32` | an end-to-end scenario for package PGO (build --pgo) | low | medium | patch | +| `3.5.0-F24` | bench package : time an installed package's build with the stock and the sysforge clang | low | medium | minor | ### Features +- **`3.5.0-F24` — `bench package `: time an installed package's build with the stock and the + sysforge clang.** `bench toolchain` times the compiler on a fixed workload; a user who builds the + toolchain to get faster package builds has no answer for the package they care about (the + kernel, mesa). A new `bench` target builds one installed package's `build()` with its **resolved + profile**, twice per round, changing only the compiler: the stock `clang` + `llvm-libs` + (+ `clang-libs`) from `bench_core._fetch_stock`, against the installed sysforge build, interleaved + ABBA through `run_interleaved`, recorded in `bench.toml` keyed by package. Same preconditions as + `bench toolchain` (a sysforge-built clang whose `pkgver-pkgrel` matches the stock one, so every + profile flag, `-fprofile-use` included, means the same to both); a package whose profile resolves + to gcc exits 2, since there is no clang to compare. Design points the spec has to settle: the + source is extracted and `prepare()`d once into a pristine copy that each timed run starts from + (untimed copy), since a second `build()` in a built tree is incremental; only `build()` is timed + (`--noarchive --nocheck`, no download or compression); ccache and `RUSTC_WRAPPER` are off for both + sides; the stock compiler gets its `LD_LIBRARY_PATH` through a per-compiler wrapper, not the build + env, so cmake, meson and rustc keep loading the host's libraries; under `sandbox_builds` the stock + prefix is bind-mounted read-only into the chroot (or the target refuses, in a first cut); the + default `--runs` is lower than `bench toolchain`'s, since each run is a full build. Unlike + `bench toolchain` it runs the PKGBUILD's `build()`, so it carries `sysforge build`'s trust, though + it installs nothing. Kernel-mode packages (`build_mode = "kernel"`) are in scope and build with + the kernel stage's flags. A package where clang is not the hot path (Rust, Go) reports its ~1.00x + as measured. Out of scope: whether the built package *runs* faster, which needs a reboot or a + workload of the user's choosing (as `kernel-bench.sh` is today). No pass/fail threshold, since a + cutoff tuned on one machine means nothing on another. Promoted from `3.5.0-Q1`. + *Priority: low · Effort: medium · Bump: minor* — a new verb that reuses the stock fetch, harness + and record; the pristine-tree reset and wrapper are the new work. + +--- + ### Bugs ### Developer tooling @@ -167,19 +194,6 @@ canonical ordering. --- -- **`3.5.0-DEV27` — an end-to-end scenario for `update --devel` on a `-git` package.** `update --devel` is - the most-run flag in the maintainer's own use and no scenario passes it; about ten 3.5.0 fixes were - VCS-package bugs (`3.5.0-B144` among the latest). A stand-in under `tools/e2e/pkgs/` named `*-git`, - whose `source=()` is a `git+file://` repository the scenario creates, keeps it off the network: - build it, then `update --devel` with no new commit reports it current and rebuilds nothing; after a - commit to that repository it reports the new `pkgver` and rebuilds; `build_state.toml` records the - commit it built. The AUR-repo half of `3.5.0-B144` (an AUR packaging repo fetched only when the AUR - says it changed) needs a real AUR package, so it belongs in `10-quickstart`'s networked run if it is - covered at all. - *Priority: high · Effort: medium · Bump: patch* — the maintainer runs this daily and it has the densest recent bug history of any untested path. - ---- - - **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** A user's own `pacman -S` of a package sysforge built from source should demote its record to a plain `pacman` marker on the next `update` (`primitives/install_reconcile.py`, through the @@ -258,18 +272,3 @@ canonical ordering. --- ### Open questions - -- **`3.5.0-Q1` — should sysforge measure a kernel built with its toolchain against one built with - stock clang?** `bench toolchain` times the compiler itself; a user who builds the toolchain to get - a faster kernel has no sysforge answer to whether the kernel got faster, or whether its build did. - The two readings are different features. Build time (the kernel compiled by each clang, interleaved - as the toolchain bench does) fits `bench`'s model: read-only, the stock clang unpacked to a temp - dir. Runtime (the kernel *running* faster) needs a reboot into each kernel and a workload, which is - outside anything sysforge does unattended. Either way it is a measurement with no pass/fail - threshold (a cutoff tuned on one machine means nothing on another), so it is not a test; the - provenance checks belong to `DEV26`. Resolve by choosing build time, runtime, or neither - (a measurement left to the user, as `kernel-bench.sh` is today). - *Priority: low · Effort: small · Bump: patch* — a decision, not an implementation; the effort is the - spike that informs it. - ---- diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index 6e37155b..58bcb3d2 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -238,6 +238,10 @@ https://keepachangelog.com/en/1.1.0/ --- - **`3.5.0-DEV24` — `make test-live` checks the stock toolchain against the live repos, and a rubric says which tier a test belongs in.** The unit tests stub the repos, so they kept passing when Arch moved files into `clang-libs` (`3.5.0-B146`). `tests/test_live_stock_toolchain.py` runs the bench's real stock fetch, unpack and probe, checks that `libLLVM` and `libclang-cpp` load from the stock prefix and not the host, and compiles against the builtin headers; against the code before `3.5.0-B146` it fails all three ways. Its tests carry a new `network` marker and are skipped unless `SYSFORGE_LIVE_TESTS=1`, which `make test-live` sets, so `make test` and the CI gates stay offline. It runs in a `live` CI job, is in `CI_JOBS` so `make pre-release` runs it, and is exempt from the release preflight like `e2e`. DESIGN.md §Release Process now maps each boundary the unit suite stubs (root and the user, state between commands, a real terminal, the live repos) to the tier whose test a fix crossing it must add. + +--- + +- **`3.5.0-DEV27` — the end-to-end tier runs `update --devel` on a `-git` package.** `update --devel` is the most-run path in daily use and no scenario passed it, while about ten of this release's fixes were VCS-package bugs. A new scenario, `35-update-devel`, builds a `-git` stand-in (`tools/e2e/pkgs/e2e-hello-git`) whose one source is a git repository the scenario creates, so it needs no network: with no new commit, `update --devel` plans `UP_TO_DATE` and rebuilds nothing; after a commit it plans the new `pkgver()` and installs it; and `build_state.toml` records the commit each build used, which the next run's `git ls-remote` short-circuit reads. The AUR half of `3.5.0-B144` (the packaging repo fetched only when the AUR says it changed) needs a real AUR package and is not covered. --- - **`3.5.0-DOC1` — the package description says what sysforge optimizes.** The `pkgdesc` shown by `pacman -Si` and on the AUR now names what people search for in one short line: "AUR helper that builds and tracks packages tuned to your CPU with PGO, BOLT and LTO", instead of a generic "build and maintenance suite" line. The README's opening paragraph says the same. diff --git a/tools/e2e/README.md b/tools/e2e/README.md index 562bbcc3..b8a465cb 100644 --- a/tools/e2e/README.md +++ b/tools/e2e/README.md @@ -41,6 +41,7 @@ sentinels (`lib.sh`), not on rendered output. Prompts are answered on a pseudo-t | `10-quickstart` | the README Quick start in one login: setup, an AUR build through the review gate, `update` | | `20-repo-optin` | a repo package refused without a terminal, the opt-in prompt, the saved answer | | `30-flag-drift` | a default-profile change reported by `update --dry-run` and rebuilt by `--rebuild-on-flag-drift` | +| `35-update-devel` | `update --devel` on a `-git` package: nothing rebuilt with no new commit, the new `pkgver()` built after one, the built commit recorded | | `40-pre-install-failure` | a build failing before any install leaves no sentinel | | `50-maintain` | `state list`, `resolve`, `log`, `packages add`/`list`/`remove`/`add-group`, `state forget` and `uninstall` on a source-built package | | `60-revert-to-stock` | a source-built repo package replaced by the stock one, its record dropped | @@ -57,7 +58,8 @@ pacman for build dependencies), `60-revert-to-stock` and `99-self-uninstall` (th from the mirrors) and `90-source-policy` (the AUR lookup a frozen build makes before refusing the clone). `pkgs/tree` is a stand-in named after a repo package, which puts `build` on its repo opt-in path -without fetching Arch's packaging repo. +without fetching Arch's packaging repo. `pkgs/e2e-hello-git` is a VCS package whose source is a git +repository its scenario creates, so `update --devel` runs `git ls-remote` and `pkgver()` offline. A scenario can't catch a change in what the live repos contain, because `pkgs/` stands in for them. That is the live-mirror tier's job (`make test-live`, `tests/test_live_stock_toolchain.py`). diff --git a/tools/e2e/pkgs/e2e-hello-git/PKGBUILD b/tools/e2e/pkgs/e2e-hello-git/PKGBUILD new file mode 100644 index 00000000..7cdb1241 --- /dev/null +++ b/tools/e2e/pkgs/e2e-hello-git/PKGBUILD @@ -0,0 +1,28 @@ +# A VCS package for the end-to-end tier (3.5.0-DEV27): its one source is a +# local git repository the scenario creates, so `update --devel` runs its +# real path, git ls-remote and pkgver(), with no network. The scenario +# replaces @UPSTREAM@ with that repository's path: a literal URL, which the +# --devel short-circuit and the built-commit record both need. +pkgname=e2e-hello-git +pkgver=r0.0000000 +pkgrel=1 +pkgdesc="sysforge end-to-end test package (VCS)" +arch=('x86_64') +license=('MIT') +makedepends=('git') +source=('e2e-hello::git+file://@UPSTREAM@') +sha256sums=('SKIP') + +pkgver() { + cd e2e-hello + printf 'r%s.%s' "$(git rev-list --count HEAD)" "$(git rev-parse --short=7 HEAD)" +} + +build() { + cd e2e-hello + ${CC:-cc} $CPPFLAGS $CFLAGS $LDFLAGS -o e2e-hello-git hello.c +} + +package() { + install -Dm755 e2e-hello/e2e-hello-git "$pkgdir/usr/bin/e2e-hello-git" +} diff --git a/tools/e2e/scenarios/35-update-devel.sh b/tools/e2e/scenarios/35-update-devel.sh new file mode 100644 index 00000000..41444d34 --- /dev/null +++ b/tools/e2e/scenarios/35-update-devel.sh @@ -0,0 +1,70 @@ +# shellcheck shell=bash +# `update --devel` on a -git package: with no new upstream commit it reports +# the package current and rebuilds nothing; after a commit it reports the new +# pkgver and rebuilds; build_state.toml records the commit each build used. +# The upstream is a local git repository, so this needs no network. +# start: ready + +upstream=~/e2e/upstream/e2e-hello +rm -rf "$upstream" +mkdir -p "$upstream" +cp "$E2E_DIR/pkgs/e2e-hello/hello.c" "$upstream/" +git -C "$upstream" init -q -b main +git -C "$upstream" -c user.name=e2e -c user.email=e2e@localhost add hello.c +git -C "$upstream" -c user.name=e2e -c user.email=e2e@localhost commit -q -m one + +dir=$(stage_pkg e2e-hello-git) +sed -i "s|@UPSTREAM@|$upstream|" "$dir/PKGBUILD" + +# installed_ver PKG: the installed version, as pacman reports it. +installed_ver() { + pacman -Q "$1" | cut -d' ' -f2 +} + +# devel_plan: what `update --devel --dry-run` plans for e2e-hello-git, as +# "ACTION AVAILABLE" from its --json document. +devel_plan() { + local out + out=$(sysforge update --devel --dry-run --json >"$upstream/hello.c" +git -C "$upstream" -c user.name=e2e -c user.email=e2e@localhost commit -q -am two +second=$(git -C "$upstream" rev-parse HEAD) +want="r2.${second:0:7}" +plan=$(devel_plan) +[[ $plan == "NEEDS_REBUILD $want-1" ]] || fail "update --devel --dry-run planned \`$plan\`, expected NEEDS_REBUILD $want-1" +output_has "$want" sysforge update --devel --dry-run