End-to-end scenario for update --devel on a -git package (3.5.0-DEV27), F24 filed

ROADMAP.md

@@ -112,21 +112,48 @@ canonical ordering.
 <!-- BEGIN roadmap-table (generated by tools/gen_roadmap_table.py — do not edit by hand; run `make roadmap-table`) -->
 | ID | Item | Priority | Effort | Bump |
 |----|------|----------|--------|------|
-| `3.5.0-DEV27` | an end-to-end scenario for update --devel on a -git package | high | medium | patch |
 | `3.5.0-DEV25` | an end-to-end scenario for the toolchain → kernel handoff | med | small | patch |
 | `3.5.0-DEV28` | an end-to-end scenario for a stock package installed over a sysforge build | med | small | patch |
 | `3.5.0-DEV30` | an end-to-end scenario for config merge | med | small | patch |
 | `3.5.0-DEV31` | an end-to-end scenario for build --interactive and update --interactive | med | small | patch |
 | `3.5.0-DEV29` | the build sandbox end to end, in the VM tier | med | large | patch |
 | `3.5.0-DEV33` | an end-to-end scenario for unattended runs | low | small | patch |
 | `3.5.0-DEV34` | an end-to-end scenario for the --frozen and --thaw flags | low | small | patch |
-| `3.5.0-Q1` | should sysforge measure a kernel built with its toolchain against one built with stock clang? | low | small | patch |
 | `3.5.0-DEV26` | a release-time run of the real toolchain → kernel chain on real hardware | low | medium | patch |
 | `3.5.0-DEV32` | an end-to-end scenario for package PGO (build --pgo) | low | medium | patch |
+| `3.5.0-F24` | bench package <pkg>: time an installed package's build with the stock and the sysforge clang | low | medium | minor |
 <!-- END roadmap-table -->
 
 ### Features
 
+- **`3.5.0-F24` — `bench package <pkg>`: time an installed package's build with the stock and the
+  sysforge clang.** `bench toolchain` times the compiler on a fixed workload; a user who builds the
+  toolchain to get faster package builds has no answer for the package they care about (the
+  kernel, mesa). A new `bench` target builds one installed package's `build()` with its **resolved
+  profile**, twice per round, changing only the compiler: the stock `clang` + `llvm-libs`
+  (+ `clang-libs`) from `bench_core._fetch_stock`, against the installed sysforge build, interleaved
+  ABBA through `run_interleaved`, recorded in `bench.toml` keyed by package. Same preconditions as
+  `bench toolchain` (a sysforge-built clang whose `pkgver-pkgrel` matches the stock one, so every
+  profile flag, `-fprofile-use` included, means the same to both); a package whose profile resolves
+  to gcc exits 2, since there is no clang to compare. Design points the spec has to settle: the
+  source is extracted and `prepare()`d once into a pristine copy that each timed run starts from
+  (untimed copy), since a second `build()` in a built tree is incremental; only `build()` is timed
+  (`--noarchive --nocheck`, no download or compression); ccache and `RUSTC_WRAPPER` are off for both
+  sides; the stock compiler gets its `LD_LIBRARY_PATH` through a per-compiler wrapper, not the build
+  env, so cmake, meson and rustc keep loading the host's libraries; under `sandbox_builds` the stock
+  prefix is bind-mounted read-only into the chroot (or the target refuses, in a first cut); the
+  default `--runs` is lower than `bench toolchain`'s, since each run is a full build. Unlike
+  `bench toolchain` it runs the PKGBUILD's `build()`, so it carries `sysforge build`'s trust, though
+  it installs nothing. Kernel-mode packages (`build_mode = "kernel"`) are in scope and build with
+  the kernel stage's flags. A package where clang is not the hot path (Rust, Go) reports its ~1.00x
+  as measured. Out of scope: whether the built package *runs* faster, which needs a reboot or a
+  workload of the user's choosing (as `kernel-bench.sh` is today). No pass/fail threshold, since a
+  cutoff tuned on one machine means nothing on another. Promoted from `3.5.0-Q1`.
+  *Priority: low · Effort: medium · Bump: minor* — a new verb that reuses the stock fetch, harness
+  and record; the pristine-tree reset and wrapper are the new work.
+
+---
+
 ### Bugs
 
 ### Developer tooling
@@ -167,19 +194,6 @@ canonical ordering.
 
 ---
 
-- **`3.5.0-DEV27` — an end-to-end scenario for `update --devel` on a `-git` package.** `update --devel` is
-  the most-run flag in the maintainer's own use and no scenario passes it; about ten 3.5.0 fixes were
-  VCS-package bugs (`3.5.0-B144` among the latest). A stand-in under `tools/e2e/pkgs/` named `*-git`,
-  whose `source=()` is a `git+file://` repository the scenario creates, keeps it off the network:
-  build it, then `update --devel` with no new commit reports it current and rebuilds nothing; after a
-  commit to that repository it reports the new `pkgver` and rebuilds; `build_state.toml` records the
-  commit it built. The AUR-repo half of `3.5.0-B144` (an AUR packaging repo fetched only when the AUR
-  says it changed) needs a real AUR package, so it belongs in `10-quickstart`'s networked run if it is
-  covered at all.
-  *Priority: high · Effort: medium · Bump: patch* — the maintainer runs this daily and it has the densest recent bug history of any untested path.
-
----
-
 - **`3.5.0-DEV28` — an end-to-end scenario for a stock package installed over a sysforge build.** A
   user's own `pacman -S` of a package sysforge built from source should demote its record to a plain
   `pacman` marker on the next `update` (`primitives/install_reconcile.py`, through the
@@ -258,18 +272,3 @@ canonical ordering.
 ---
 
 ### Open questions
-
-- **`3.5.0-Q1` — should sysforge measure a kernel built with its toolchain against one built with
-  stock clang?** `bench toolchain` times the compiler itself; a user who builds the toolchain to get
-  a faster kernel has no sysforge answer to whether the kernel got faster, or whether its build did.
-  The two readings are different features. Build time (the kernel compiled by each clang, interleaved
-  as the toolchain bench does) fits `bench`'s model: read-only, the stock clang unpacked to a temp
-  dir. Runtime (the kernel *running* faster) needs a reboot into each kernel and a workload, which is
-  outside anything sysforge does unattended. Either way it is a measurement with no pass/fail
-  threshold (a cutoff tuned on one machine means nothing on another), so it is not a test; the
-  provenance checks belong to `DEV26`. Resolve by choosing build time, runtime, or neither
-  (a measurement left to the user, as `kernel-bench.sh` is today).
-  *Priority: low · Effort: small · Bump: patch* — a decision, not an implementation; the effort is the
-  spike that informs it.
-
----

docs/release-notes/unreleased.md

@@ -238,6 +238,10 @@ https://keepachangelog.com/en/1.1.0/
 ---
 
 - **`3.5.0-DEV24` — `make test-live` checks the stock toolchain against the live repos, and a rubric says which tier a test belongs in.** The unit tests stub the repos, so they kept passing when Arch moved files into `clang-libs` (`3.5.0-B146`). `tests/test_live_stock_toolchain.py` runs the bench's real stock fetch, unpack and probe, checks that `libLLVM` and `libclang-cpp` load from the stock prefix and not the host, and compiles against the builtin headers; against the code before `3.5.0-B146` it fails all three ways. Its tests carry a new `network` marker and are skipped unless `SYSFORGE_LIVE_TESTS=1`, which `make test-live` sets, so `make test` and the CI gates stay offline. It runs in a `live` CI job, is in `CI_JOBS` so `make pre-release` runs it, and is exempt from the release preflight like `e2e`. DESIGN.md §Release Process now maps each boundary the unit suite stubs (root and the user, state between commands, a real terminal, the live repos) to the tier whose test a fix crossing it must add.
+
+---
+
+- **`3.5.0-DEV27` — the end-to-end tier runs `update --devel` on a `-git` package.** `update --devel` is the most-run path in daily use and no scenario passed it, while about ten of this release's fixes were VCS-package bugs. A new scenario, `35-update-devel`, builds a `-git` stand-in (`tools/e2e/pkgs/e2e-hello-git`) whose one source is a git repository the scenario creates, so it needs no network: with no new commit, `update --devel` plans `UP_TO_DATE` and rebuilds nothing; after a commit it plans the new `pkgver()` and installs it; and `build_state.toml` records the commit each build used, which the next run's `git ls-remote` short-circuit reads. The AUR half of `3.5.0-B144` (the packaging repo fetched only when the AUR says it changed) needs a real AUR package and is not covered.
 ---
 
 - **`3.5.0-DOC1` — the package description says what sysforge optimizes.** The `pkgdesc` shown by `pacman -Si` and on the AUR now names what people search for in one short line: "AUR helper that builds and tracks packages tuned to your CPU with PGO, BOLT and LTO", instead of a generic "build and maintenance suite" line. The README's opening paragraph says the same.

tools/e2e/README.md

@@ -41,6 +41,7 @@ sentinels (`lib.sh`), not on rendered output. Prompts are answered on a pseudo-t
 | `10-quickstart` | the README Quick start in one login: setup, an AUR build through the review gate, `update` |
 | `20-repo-optin` | a repo package refused without a terminal, the opt-in prompt, the saved answer |
 | `30-flag-drift` | a default-profile change reported by `update --dry-run` and rebuilt by `--rebuild-on-flag-drift` |
+| `35-update-devel` | `update --devel` on a `-git` package: nothing rebuilt with no new commit, the new `pkgver()` built after one, the built commit recorded |
 | `40-pre-install-failure` | a build failing before any install leaves no sentinel |
 | `50-maintain` | `state list`, `resolve`, `log`, `packages add`/`list`/`remove`/`add-group`, `state forget` and `uninstall` on a source-built package |
 | `60-revert-to-stock` | a source-built repo package replaced by the stock one, its record dropped |
@@ -57,7 +58,8 @@ pacman for build dependencies), `60-revert-to-stock` and `99-self-uninstall` (th
 from the mirrors) and `90-source-policy` (the AUR lookup a frozen build makes before refusing the
 clone).
 `pkgs/tree` is a stand-in named after a repo package, which puts `build` on its repo opt-in path
-without fetching Arch's packaging repo.
+without fetching Arch's packaging repo. `pkgs/e2e-hello-git` is a VCS package whose source is a git
+repository its scenario creates, so `update --devel` runs `git ls-remote` and `pkgver()` offline.
 
 A scenario can't catch a change in what the live repos contain, because `pkgs/` stands in for
 them. That is the live-mirror tier's job (`make test-live`, `tests/test_live_stock_toolchain.py`).

tools/e2e/pkgs/e2e-hello-git/PKGBUILD

@@ -0,0 +1,28 @@
+# A VCS package for the end-to-end tier (3.5.0-DEV27): its one source is a
+# local git repository the scenario creates, so `update --devel` runs its
+# real path, git ls-remote and pkgver(), with no network. The scenario
+# replaces @UPSTREAM@ with that repository's path: a literal URL, which the
+# --devel short-circuit and the built-commit record both need.
+pkgname=e2e-hello-git
+pkgver=r0.0000000
+pkgrel=1
+pkgdesc="sysforge end-to-end test package (VCS)"
+arch=('x86_64')
+license=('MIT')
+makedepends=('git')
+source=('e2e-hello::git+file://@UPSTREAM@')
+sha256sums=('SKIP')
+
+pkgver() {
+    cd e2e-hello
+    printf 'r%s.%s' "$(git rev-list --count HEAD)" "$(git rev-parse --short=7 HEAD)"
+}
+
+build() {
+    cd e2e-hello
+    ${CC:-cc} $CPPFLAGS $CFLAGS $LDFLAGS -o e2e-hello-git hello.c
+}
+
+package() {
+    install -Dm755 e2e-hello/e2e-hello-git "$pkgdir/usr/bin/e2e-hello-git"
+}

tools/e2e/scenarios/35-update-devel.sh

@@ -0,0 +1,70 @@
+# shellcheck shell=bash
+# `update --devel` on a -git package: with no new upstream commit it reports
+# the package current and rebuilds nothing; after a commit it reports the new
+# pkgver and rebuilds; build_state.toml records the commit each build used.
+# The upstream is a local git repository, so this needs no network.
+# start: ready
+
+upstream=~/e2e/upstream/e2e-hello
+rm -rf "$upstream"
+mkdir -p "$upstream"
+cp "$E2E_DIR/pkgs/e2e-hello/hello.c" "$upstream/"
+git -C "$upstream" init -q -b main
+git -C "$upstream" -c user.name=e2e -c user.email=e2e@localhost add hello.c
+git -C "$upstream" -c user.name=e2e -c user.email=e2e@localhost commit -q -m one
+
+dir=$(stage_pkg e2e-hello-git)
+sed -i "s|@UPSTREAM@|$upstream|" "$dir/PKGBUILD"
+
+# installed_ver PKG: the installed version, as pacman reports it.
+installed_ver() {
+    pacman -Q "$1" | cut -d' ' -f2
+}
+
+# devel_plan: what `update --devel --dry-run` plans for e2e-hello-git, as
+# "ACTION AVAILABLE" from its --json document.
+devel_plan() {
+    local out
+    out=$(sysforge update --devel --dry-run --json </dev/null) || fail "update --devel --dry-run --json failed"
+    python3 -c '
+import json, sys
+for p in json.load(sys.stdin)["packages"]:
+    if p["pkgbase"] == "e2e-hello-git":
+        print(p["action"], p["available"])
+' <<<"$out"
+}
+
+step "build it"
+sysforge build "$dir" -si </dev/null
+assert_installed e2e-hello-git
+assert_recorded e2e-hello-git
+first=$(git -C "$upstream" rev-parse HEAD)
+[[ $(state_get e2e-hello-git built_upstream_commit) == "$first" ]] \
+    || fail "build_state.toml did not record the commit built ($first)"
+ver1=$(installed_ver e2e-hello-git)
+[[ $ver1 == r1.* ]] || fail "pkgver() did not run: installed $ver1"
+built1=$(state_get e2e-hello-git built_at)
+
+step "no new commit: reported current, nothing rebuilt"
+plan=$(devel_plan)
+[[ $plan == UP_TO_DATE* ]] || fail "update --devel --dry-run planned \`$plan\` with no new commit"
+sysforge update --devel --yes </dev/null
+[[ $(installed_ver e2e-hello-git) == "$ver1" ]] || fail "update --devel changed the version with no new commit"
+[[ $(state_get e2e-hello-git built_at) == "$built1" ]] || fail "update --devel rebuilt e2e-hello-git with no new commit"
+
+step "a new commit: the new pkgver reported and built"
+echo '/* two */' >>"$upstream/hello.c"
+git -C "$upstream" -c user.name=e2e -c user.email=e2e@localhost commit -q -am two
+second=$(git -C "$upstream" rev-parse HEAD)
+want="r2.${second:0:7}"
+plan=$(devel_plan)
+[[ $plan == "NEEDS_REBUILD $want-1" ]] || fail "update --devel --dry-run planned \`$plan\`, expected NEEDS_REBUILD $want-1"
+output_has "$want" sysforge update --devel --dry-run </dev/null \
+    || fail "update --devel --dry-run did not show $want"
+sysforge update --devel --yes </dev/null
+[[ $(installed_ver e2e-hello-git) == "$want-1" ]] \
+    || fail "update --devel installed $(installed_ver e2e-hello-git), expected $want-1"
+[[ $(state_get e2e-hello-git built_upstream_commit) == "$second" ]] \
+    || fail "build_state.toml did not record the new commit ($second)"
+assert_built_here e2e-hello-git
+assert_no_sentinel