diff --git a/DESIGN.md b/DESIGN.md index 36528a37..38f814bc 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -3410,7 +3410,7 @@ Why the argv *shape* branches rather than composing another `build_throttle.wrap **The final conf is audited before the build (`3.4.0-F2`).** Every correction above was found the slow way: a build error several layers away from a host path or binary that crossed into the container. `audit_chroot_conf` runs on `chroot_conf_text`'s output *after* `provision_toolchain` and `provision_profile_data`, so it judges the real `root/` tree rather than predicting it, and refuses (`SandboxUnavailable`) on anything the correction sets missed. It folds the text into its **effective** assignments (`config.parse_makepkg_conf_text`, last assignment wins, so the copied host `PKGDEST` is superseded by `/pkgdest`) and scans only what a tool reads at build time: keys ending in `FLAGS`, exported keys, `BUILDENV` and the dest keys. Package-internal locations (`DBGSRCDIR`, `MAN_DIRS`, `PURGE_TARGETS`, …) are not filesystem inputs and are not scanned, and a toolchain key counts only when it is **exported**, because makepkg exports the `*FLAGS` keys, `CHOST` and `MAKEFLAGS` and nothing else, so a plain `AR=llvm-ar` in a host conf is inert. An absolute path passes when it is under a container prefix (the dest keys, `/startdir`, `/tmp`), under `/usr` or `/opt` (dependencies install into the working copy after the root is cloned), present in `root/`, or part of a `-f{file,debug,macro}-prefix-map=` rewrite rule; a binary (an exported toolchain key, a `-fuse-ld=` linker, an exported `*_WRAPPER`, an enabled `ccache`/`distcc`) must resolve in `root/usr/bin`. The refusal names the key, the value and the correction set that should have covered it. Every member of the bug class so far (`3.2.0-B2`, `B4`, `B6`, `B7`, `B10`, `B12`) would have been one such refusal; `3.2.0-B5`, a stale chroot package database, is outside it. -**Provenance comes from the build that ran (`3.4.0-B3`).** `toolchain_variant`/`toolchain_fingerprint` are resolved once per run from the host, but a sandboxed build compiles with the container's compiler, so stamping the host values made toolchain drift compare the host against itself. `build_argv` now passes `-l sysforge` (`COPY_NAME`), which fixes the working copy at `/sysforge`. After a successful sandboxed build the seam parks a `SandboxProvenance(cc_name, copy_identity)` on the module session, where `copy_identity` is `build_fingerprint.static_identity` of the copy's compiler: size and mtime of the driver and its `libLLVM.so*`, never executed (it would load host libraries). `_record_build_state` takes it before anything can return; `recorded_toolchain` keeps the host's variant and fingerprint when the host's `/usr/bin/` has the same static identity (pacman restores mtimes from the archive, so the same package file matches), and otherwise records `stock_llvm`/`gcc` with a `sandbox|…` fingerprint that can never equal a host one. `invoke_makepkg` clears any pending provenance on entry, so a failed build never inherits an earlier success's. `build_state` gains `sandboxed`, and flag drift a matching sub-axis (§Drift detection). The same holds for a host build on the other compiler family (`3.5.0-B125`): `_own_compiler_variant` records `gcc`/`stock_llvm` and no fingerprint when the build's `CC` (the raw `--cc`, else the resolved profile's) is not the active variant's family. The decision is `pipeline_state.variant_built_with`. The toolchain stage's builds (which record their own variants) and an unrecognised compiler name keep the run-wide values; a kernel built with the other family records it through its `cc_override` (`3.5.0-B126`). +**Provenance comes from the build that ran (`3.4.0-B3`).** `toolchain_variant`/`toolchain_fingerprint` are resolved once per run from the host, but a sandboxed build compiles with the container's compiler, so stamping the host values made toolchain drift compare the host against itself. `build_argv` now passes `-l sysforge` (`COPY_NAME`), which fixes the working copy at `/sysforge`. After a successful sandboxed build the seam parks a `SandboxProvenance(cc_name, copy_identity)` on the module session, where `copy_identity` is `build_fingerprint.static_identity` of the copy's compiler: size and mtime of the driver and its `libLLVM.so*`, never executed (it would load host libraries). `_record_build_state` takes it before anything can return; `recorded_toolchain` keeps the host's variant and fingerprint when the host's `/usr/bin/` has the same static identity (pacman restores mtimes from the archive, so the same package file matches), and otherwise records `stock_llvm`/`gcc` with a `sandbox|…` fingerprint that can never equal a host one. Toolchain drift does not report that stamp while the next build would record it again (`3.5.0-B145`, `flag_drift.builds_with_repo_compiler`; §Drift detection). `invoke_makepkg` clears any pending provenance on entry, so a failed build never inherits an earlier success's. `build_state` gains `sandboxed`, and flag drift a matching sub-axis (§Drift detection). The same holds for a host build on the other compiler family (`3.5.0-B125`): `_own_compiler_variant` records `gcc`/`stock_llvm` and no fingerprint when the build's `CC` (the raw `--cc`, else the resolved profile's) is not the active variant's family. The decision is `pipeline_state.variant_built_with`. The toolchain stage's builds (which record their own variants) and an unrecognised compiler name keep the run-wide values; a kernel built with the other family records it through its `cc_override` (`3.5.0-B126`). **The heartbeat sets detail; the pty loop paints.** `makepkg_invoke._on_idle` fires every `MAKEPKG_HEARTBEAT_S` of child silence, logs a `[heartbeat]` line (the latest `\r`-redrawn ninja status, or "no output") to the per-package log, and hands the same text to `progress_hooks.hooks().heartbeat()`, which only attaches it to the bar's line. Putting it on screen is the scheduled painter's job (`3.3.0-F2`): at default verbosity `run_with_pty` forwards makepkg's raw bytes, so it enters `forwarding_output()` (the bar's ticker stands down) and calls `refresh()` at most once a second, only when `_StreamBoundary` says the forwarded stream sits between escape sequences and UTF-8 characters. At `-vvv` output arrives as whole log lines and the ticker stays in charge. @@ -4232,7 +4232,7 @@ Escape hatches: *Flag drift.* Same package version but a different resolved compiler configuration — e.g. profile changed, new flag added, or build mode switched. At build time the resolved flags string is recorded per package in `build_state.toml`; Phase 4.3 re-resolves the current profile for each source-built package and diffs the result against the stored string (via `primitives/flag_drift.resolve_flag_drift`). Both the recorded and the re-resolved string are the **effective** flags — the resolved profile after the `[preserved_system_tokens]` pass, produced on both sides by `makepkg_conf.serialize_effective_flags` (3.1.0-B11). Consequently the axis also covers changes to `[preserved_system_tokens]`, `[append_conflict_groups]`, and `/etc/makepkg.conf`'s own hardening baseline, none of which were visible to it before. Flag drift is **reported by default** and is network-free (`--offline --dry-run` gives a read-only report); `--rebuild-on-flag-drift` rebuilds the drifted packages in this run's walk — its `ui()` notice is built from that queue (`update._partition_flag_drift`, the promotion loop's own predicate) and names separately what drifted but was not queued, with the command that would rebuild it (`sysforge run ` for a stage-owned kernel/toolchain package, else `sysforge build `; 3.2.0-B40) — and `--rebuild-on-drift` is the umbrella over both the flag and toolchain-variant axes. See §`update.py` → *Phase 4.3 — Flag drift*. The axis has one sub-axis that is not a flag: **`sandboxed`** (3.4.0-B3). A record whose `sandboxed` differs from what a fresh build would do (the process-wide sandbox policy through `for_profile`, always `False` for a stage-owned package) gets a ` sandboxed: → ` line in the same diff vocabulary and is promoted by `--rebuild-on-flag-drift` like any other. It is judged before the `!buildflags` and no-recorded-flags returns, because the sandbox changes the built artifact (dependency versions, the compiler, automatically detected optional features) even where makepkg ignores conf flags. A record without the field never drifts. **Build inputs** (3.2.0-F18) are the second non-flag sub-axis: settings that change the artifact through a seam the profile never sees. `primitives/build_inputs.resolve_build_inputs` returns the ones that apply to a package — `python` (the `major.minor` of the interpreter `makepkg_env.resolve_build_python` pins, for a package whose runtime `depends` names `python`/`python-*`; `major.minor` rather than the path, so a patch-level update never drifts, and runtime `depends` rather than `makedepends`, so a package that only runs python while building is not tied to its version), `mesa_gallium`/`mesa_vulkan` (`resolve_or_detect_mesa_drivers`, only with `[mesa] filter_drivers` on), `llvm_targets` (`resolve_or_detect_llvm_targets`, never for `lib32-*`), and `pgo_profile` (`mesa_pgo.reuse_profdata` — the merged profile a rebuild's reuse branch would splice in through `compiler_flags_extra` — only for an LLVM compiler and a non-stage-owned build, mirroring that branch's gates). `_record_build_state` stores the result as `build_inputs`, serialized in `flags_string`'s own `KEY=value` line form, and `flag_drift._inputs_diff` re-derives it through the **same function** with the compiler resolved the same way, then renders it with `diff_flags`, so drift lines read ` python: '3.13' → '3.14'`, ` +pgo_profile: …`. Recording the resolved value rather than what a patcher reported applying is deliberate: a patch whose anchor was missing would otherwise record "no drivers" against a resolver that says "drivers", and the drift would never clear however often the package rebuilt (the B11 rule). The sub-axis is enabled by passing `state_dir` (which locates `hardware_profile.toml`), is judged before the `!buildflags` and no-recorded-flags returns like the sandbox line, and reads an absent field as unknown, so an upgrade flags nothing. It is also the **only** axis judged for a PGO-mode entry (`build_mode` `pgo`): `flag_drift.drift_candidate` admits such an entry once it has recorded `build_inputs`, and its profile flags stay unjudged, as they were before the field existed. Out of scope: `kernel.toml` kconfig inputs (stage-owned, diffed request-vs-result by the kernel stage's Gate 2) and the toolchain stage's own PGO/BOLT settings; an LLVM package's `llvm_targets` drift lands in the build-state-wide fold with the owning-stage hint like any other stage-owned entry. -*Toolchain drift.* Same package version, built by a different compiler than a fresh build would use. Every source build records `toolchain_variant` (`gcc`/`stock_llvm`/`pgo_llvm`) and `toolchain_fingerprint`; Phase 4.25 (`update._phase_toolchain_drift`) compares them with the active toolchain (`get_toolchain_variant`/`get_toolchain_fingerprint`). A different variant drifts, and so does the same variant with a different fingerprint (a toolchain rebuild with unchanged soname, compared only when both sides have one). The recorded variant names the compiler the build actually used (`3.5.0-B125`): a package swapped to the other family by `build --toolchain`, a `[package_compiler_overrides]` row or a profile's own `toolchain` records that family's variant (`stock_llvm` for clang, as a sandboxed build's repo clang does) and no fingerprint (`makepkg_wrapper._own_compiler_variant`, via `pipeline_state.variant_for_compiler`); the toolchain stage's builds keep the run-wide values, and a kernel records its own compiler like any other build (`3.5.0-B126`). Phase 4.25 then skips a record in the other family only while a fresh build would still use that family (`flag_drift.current_compiler`, the same profile resolution flag drift and the sandbox sub-axis use), so a package kept on gcc on purpose is never rebuilt toward clang, and a one-off `--toolchain` build still drifts back. No toolchain stage (`system`) means no toolchain drift. Reported by default; `--rebuild-on-toolchain-drift` rebuilds. +*Toolchain drift.* Same package version, built by a different compiler than a fresh build would use. Every source build records `toolchain_variant` (`gcc`/`stock_llvm`/`pgo_llvm`) and `toolchain_fingerprint`; Phase 4.25 (`update._phase_toolchain_drift`) compares them with the active toolchain (`get_toolchain_variant`/`get_toolchain_fingerprint`). A different variant drifts, and so does the same variant with a different fingerprint (a toolchain rebuild with unchanged soname, compared only when both sides have one). The recorded variant names the compiler the build actually used (`3.5.0-B125`): a package swapped to the other family by `build --toolchain`, a `[package_compiler_overrides]` row or a profile's own `toolchain` records that family's variant (`stock_llvm` for clang, as a sandboxed build's repo clang does) and no fingerprint (`makepkg_wrapper._own_compiler_variant`, via `pipeline_state.variant_for_compiler`); the toolchain stage's builds keep the run-wide values, and a kernel records its own compiler like any other build (`3.5.0-B126`). Phase 4.25 then skips a record in the other family only while a fresh build would still use that family (`flag_drift.current_compiler`, the same profile resolution flag drift and the sandbox sub-axis use), so a package kept on gcc on purpose is never rebuilt toward clang, and a one-off `--toolchain` build still drifts back. The same rule covers a build the sandbox compiled with the repo LLVM (`3.5.0-B145`): a Rust package pulls Arch's `rust`, which links the repo LLVM, so the sandbox swaps that in and `build_sandbox.recorded_toolchain` records `stock_llvm` with a `sandbox|…` fingerprint, which can never equal a sysforge-built host toolchain. Phase 4.25 skips such a record while `flag_drift.builds_with_repo_compiler` says the next build would do the same (sandboxed for its profile, per `current_sandboxed`, and `build_sandbox.needs_repo_rust`); before B145 every such package drifted on every run and, under `rebuild_on_drift`, was rebuilt each time to record the same stamp again. A change of the repo LLVM itself is not detected for these packages: the sandbox's working copy is shared by every build, so its compiler identity at update time reflects whichever package built last, not this one's next build. No toolchain stage (`system`) means no toolchain drift. Reported by default; `--rebuild-on-toolchain-drift` rebuilds. **Config defaults.** All three drift-rebuild flags (`--rebuild-on-drift`, `--rebuild-on-toolchain-drift`, `--rebuild-on-flag-drift`) fall back to a sysforge.toml `[update]` default (`rebuild_on_drift`, `rebuild_on_toolchain_drift`, `rebuild_on_flag_drift` respectively) via `_resolve_drift_axes`, which resolves each axis through `config.resolve_flag_default`. The CLI flag always wins when explicitly passed; the config value only supplies the default when it isn't. diff --git a/docs/design/11-makepkg-wrapper.md b/docs/design/11-makepkg-wrapper.md index 2464f405..08b93937 100644 --- a/docs/design/11-makepkg-wrapper.md +++ b/docs/design/11-makepkg-wrapper.md @@ -30,7 +30,7 @@ Why the argv *shape* branches rather than composing another `build_throttle.wrap **The final conf is audited before the build (`3.4.0-F2`).** Every correction above was found the slow way: a build error several layers away from a host path or binary that crossed into the container. `audit_chroot_conf` runs on `chroot_conf_text`'s output *after* `provision_toolchain` and `provision_profile_data`, so it judges the real `root/` tree rather than predicting it, and refuses (`SandboxUnavailable`) on anything the correction sets missed. It folds the text into its **effective** assignments (`config.parse_makepkg_conf_text`, last assignment wins, so the copied host `PKGDEST` is superseded by `/pkgdest`) and scans only what a tool reads at build time: keys ending in `FLAGS`, exported keys, `BUILDENV` and the dest keys. Package-internal locations (`DBGSRCDIR`, `MAN_DIRS`, `PURGE_TARGETS`, …) are not filesystem inputs and are not scanned, and a toolchain key counts only when it is **exported**, because makepkg exports the `*FLAGS` keys, `CHOST` and `MAKEFLAGS` and nothing else, so a plain `AR=llvm-ar` in a host conf is inert. An absolute path passes when it is under a container prefix (the dest keys, `/startdir`, `/tmp`), under `/usr` or `/opt` (dependencies install into the working copy after the root is cloned), present in `root/`, or part of a `-f{file,debug,macro}-prefix-map=` rewrite rule; a binary (an exported toolchain key, a `-fuse-ld=` linker, an exported `*_WRAPPER`, an enabled `ccache`/`distcc`) must resolve in `root/usr/bin`. The refusal names the key, the value and the correction set that should have covered it. Every member of the bug class so far (`3.2.0-B2`, `B4`, `B6`, `B7`, `B10`, `B12`) would have been one such refusal; `3.2.0-B5`, a stale chroot package database, is outside it. -**Provenance comes from the build that ran (`3.4.0-B3`).** `toolchain_variant`/`toolchain_fingerprint` are resolved once per run from the host, but a sandboxed build compiles with the container's compiler, so stamping the host values made toolchain drift compare the host against itself. `build_argv` now passes `-l sysforge` (`COPY_NAME`), which fixes the working copy at `/sysforge`. After a successful sandboxed build the seam parks a `SandboxProvenance(cc_name, copy_identity)` on the module session, where `copy_identity` is `build_fingerprint.static_identity` of the copy's compiler: size and mtime of the driver and its `libLLVM.so*`, never executed (it would load host libraries). `_record_build_state` takes it before anything can return; `recorded_toolchain` keeps the host's variant and fingerprint when the host's `/usr/bin/` has the same static identity (pacman restores mtimes from the archive, so the same package file matches), and otherwise records `stock_llvm`/`gcc` with a `sandbox|…` fingerprint that can never equal a host one. `invoke_makepkg` clears any pending provenance on entry, so a failed build never inherits an earlier success's. `build_state` gains `sandboxed`, and flag drift a matching sub-axis (§Drift detection). The same holds for a host build on the other compiler family (`3.5.0-B125`): `_own_compiler_variant` records `gcc`/`stock_llvm` and no fingerprint when the build's `CC` (the raw `--cc`, else the resolved profile's) is not the active variant's family. The decision is `pipeline_state.variant_built_with`. The toolchain stage's builds (which record their own variants) and an unrecognised compiler name keep the run-wide values; a kernel built with the other family records it through its `cc_override` (`3.5.0-B126`). +**Provenance comes from the build that ran (`3.4.0-B3`).** `toolchain_variant`/`toolchain_fingerprint` are resolved once per run from the host, but a sandboxed build compiles with the container's compiler, so stamping the host values made toolchain drift compare the host against itself. `build_argv` now passes `-l sysforge` (`COPY_NAME`), which fixes the working copy at `/sysforge`. After a successful sandboxed build the seam parks a `SandboxProvenance(cc_name, copy_identity)` on the module session, where `copy_identity` is `build_fingerprint.static_identity` of the copy's compiler: size and mtime of the driver and its `libLLVM.so*`, never executed (it would load host libraries). `_record_build_state` takes it before anything can return; `recorded_toolchain` keeps the host's variant and fingerprint when the host's `/usr/bin/` has the same static identity (pacman restores mtimes from the archive, so the same package file matches), and otherwise records `stock_llvm`/`gcc` with a `sandbox|…` fingerprint that can never equal a host one. Toolchain drift does not report that stamp while the next build would record it again (`3.5.0-B145`, `flag_drift.builds_with_repo_compiler`; §Drift detection). `invoke_makepkg` clears any pending provenance on entry, so a failed build never inherits an earlier success's. `build_state` gains `sandboxed`, and flag drift a matching sub-axis (§Drift detection). The same holds for a host build on the other compiler family (`3.5.0-B125`): `_own_compiler_variant` records `gcc`/`stock_llvm` and no fingerprint when the build's `CC` (the raw `--cc`, else the resolved profile's) is not the active variant's family. The decision is `pipeline_state.variant_built_with`. The toolchain stage's builds (which record their own variants) and an unrecognised compiler name keep the run-wide values; a kernel built with the other family records it through its `cc_override` (`3.5.0-B126`). **The heartbeat sets detail; the pty loop paints.** `makepkg_invoke._on_idle` fires every `MAKEPKG_HEARTBEAT_S` of child silence, logs a `[heartbeat]` line (the latest `\r`-redrawn ninja status, or "no output") to the per-package log, and hands the same text to `progress_hooks.hooks().heartbeat()`, which only attaches it to the bar's line. Putting it on screen is the scheduled painter's job (`3.3.0-F2`): at default verbosity `run_with_pty` forwards makepkg's raw bytes, so it enters `forwarding_output()` (the bar's ticker stands down) and calls `refresh()` at most once a second, only when `_StreamBoundary` says the forwarded stream sits between escape sequences and UTF-8 characters. At `-vvv` output arrives as whole log lines and the ticker stays in charge. diff --git a/docs/design/18-reconverge.md b/docs/design/18-reconverge.md index 860f4706..e0ccf7ae 100644 --- a/docs/design/18-reconverge.md +++ b/docs/design/18-reconverge.md @@ -8,7 +8,7 @@ *Flag drift.* Same package version but a different resolved compiler configuration — e.g. profile changed, new flag added, or build mode switched. At build time the resolved flags string is recorded per package in `build_state.toml`; Phase 4.3 re-resolves the current profile for each source-built package and diffs the result against the stored string (via `primitives/flag_drift.resolve_flag_drift`). Both the recorded and the re-resolved string are the **effective** flags — the resolved profile after the `[preserved_system_tokens]` pass, produced on both sides by `makepkg_conf.serialize_effective_flags` (3.1.0-B11). Consequently the axis also covers changes to `[preserved_system_tokens]`, `[append_conflict_groups]`, and `/etc/makepkg.conf`'s own hardening baseline, none of which were visible to it before. Flag drift is **reported by default** and is network-free (`--offline --dry-run` gives a read-only report); `--rebuild-on-flag-drift` rebuilds the drifted packages in this run's walk — its `ui()` notice is built from that queue (`update._partition_flag_drift`, the promotion loop's own predicate) and names separately what drifted but was not queued, with the command that would rebuild it (`sysforge run ` for a stage-owned kernel/toolchain package, else `sysforge build `; 3.2.0-B40) — and `--rebuild-on-drift` is the umbrella over both the flag and toolchain-variant axes. See §`update.py` → *Phase 4.3 — Flag drift*. The axis has one sub-axis that is not a flag: **`sandboxed`** (3.4.0-B3). A record whose `sandboxed` differs from what a fresh build would do (the process-wide sandbox policy through `for_profile`, always `False` for a stage-owned package) gets a ` sandboxed: → ` line in the same diff vocabulary and is promoted by `--rebuild-on-flag-drift` like any other. It is judged before the `!buildflags` and no-recorded-flags returns, because the sandbox changes the built artifact (dependency versions, the compiler, automatically detected optional features) even where makepkg ignores conf flags. A record without the field never drifts. **Build inputs** (3.2.0-F18) are the second non-flag sub-axis: settings that change the artifact through a seam the profile never sees. `primitives/build_inputs.resolve_build_inputs` returns the ones that apply to a package — `python` (the `major.minor` of the interpreter `makepkg_env.resolve_build_python` pins, for a package whose runtime `depends` names `python`/`python-*`; `major.minor` rather than the path, so a patch-level update never drifts, and runtime `depends` rather than `makedepends`, so a package that only runs python while building is not tied to its version), `mesa_gallium`/`mesa_vulkan` (`resolve_or_detect_mesa_drivers`, only with `[mesa] filter_drivers` on), `llvm_targets` (`resolve_or_detect_llvm_targets`, never for `lib32-*`), and `pgo_profile` (`mesa_pgo.reuse_profdata` — the merged profile a rebuild's reuse branch would splice in through `compiler_flags_extra` — only for an LLVM compiler and a non-stage-owned build, mirroring that branch's gates). `_record_build_state` stores the result as `build_inputs`, serialized in `flags_string`'s own `KEY=value` line form, and `flag_drift._inputs_diff` re-derives it through the **same function** with the compiler resolved the same way, then renders it with `diff_flags`, so drift lines read ` python: '3.13' → '3.14'`, ` +pgo_profile: …`. Recording the resolved value rather than what a patcher reported applying is deliberate: a patch whose anchor was missing would otherwise record "no drivers" against a resolver that says "drivers", and the drift would never clear however often the package rebuilt (the B11 rule). The sub-axis is enabled by passing `state_dir` (which locates `hardware_profile.toml`), is judged before the `!buildflags` and no-recorded-flags returns like the sandbox line, and reads an absent field as unknown, so an upgrade flags nothing. It is also the **only** axis judged for a PGO-mode entry (`build_mode` `pgo`): `flag_drift.drift_candidate` admits such an entry once it has recorded `build_inputs`, and its profile flags stay unjudged, as they were before the field existed. Out of scope: `kernel.toml` kconfig inputs (stage-owned, diffed request-vs-result by the kernel stage's Gate 2) and the toolchain stage's own PGO/BOLT settings; an LLVM package's `llvm_targets` drift lands in the build-state-wide fold with the owning-stage hint like any other stage-owned entry. -*Toolchain drift.* Same package version, built by a different compiler than a fresh build would use. Every source build records `toolchain_variant` (`gcc`/`stock_llvm`/`pgo_llvm`) and `toolchain_fingerprint`; Phase 4.25 (`update._phase_toolchain_drift`) compares them with the active toolchain (`get_toolchain_variant`/`get_toolchain_fingerprint`). A different variant drifts, and so does the same variant with a different fingerprint (a toolchain rebuild with unchanged soname, compared only when both sides have one). The recorded variant names the compiler the build actually used (`3.5.0-B125`): a package swapped to the other family by `build --toolchain`, a `[package_compiler_overrides]` row or a profile's own `toolchain` records that family's variant (`stock_llvm` for clang, as a sandboxed build's repo clang does) and no fingerprint (`makepkg_wrapper._own_compiler_variant`, via `pipeline_state.variant_for_compiler`); the toolchain stage's builds keep the run-wide values, and a kernel records its own compiler like any other build (`3.5.0-B126`). Phase 4.25 then skips a record in the other family only while a fresh build would still use that family (`flag_drift.current_compiler`, the same profile resolution flag drift and the sandbox sub-axis use), so a package kept on gcc on purpose is never rebuilt toward clang, and a one-off `--toolchain` build still drifts back. No toolchain stage (`system`) means no toolchain drift. Reported by default; `--rebuild-on-toolchain-drift` rebuilds. +*Toolchain drift.* Same package version, built by a different compiler than a fresh build would use. Every source build records `toolchain_variant` (`gcc`/`stock_llvm`/`pgo_llvm`) and `toolchain_fingerprint`; Phase 4.25 (`update._phase_toolchain_drift`) compares them with the active toolchain (`get_toolchain_variant`/`get_toolchain_fingerprint`). A different variant drifts, and so does the same variant with a different fingerprint (a toolchain rebuild with unchanged soname, compared only when both sides have one). The recorded variant names the compiler the build actually used (`3.5.0-B125`): a package swapped to the other family by `build --toolchain`, a `[package_compiler_overrides]` row or a profile's own `toolchain` records that family's variant (`stock_llvm` for clang, as a sandboxed build's repo clang does) and no fingerprint (`makepkg_wrapper._own_compiler_variant`, via `pipeline_state.variant_for_compiler`); the toolchain stage's builds keep the run-wide values, and a kernel records its own compiler like any other build (`3.5.0-B126`). Phase 4.25 then skips a record in the other family only while a fresh build would still use that family (`flag_drift.current_compiler`, the same profile resolution flag drift and the sandbox sub-axis use), so a package kept on gcc on purpose is never rebuilt toward clang, and a one-off `--toolchain` build still drifts back. The same rule covers a build the sandbox compiled with the repo LLVM (`3.5.0-B145`): a Rust package pulls Arch's `rust`, which links the repo LLVM, so the sandbox swaps that in and `build_sandbox.recorded_toolchain` records `stock_llvm` with a `sandbox|…` fingerprint, which can never equal a sysforge-built host toolchain. Phase 4.25 skips such a record while `flag_drift.builds_with_repo_compiler` says the next build would do the same (sandboxed for its profile, per `current_sandboxed`, and `build_sandbox.needs_repo_rust`); before B145 every such package drifted on every run and, under `rebuild_on_drift`, was rebuilt each time to record the same stamp again. A change of the repo LLVM itself is not detected for these packages: the sandbox's working copy is shared by every build, so its compiler identity at update time reflects whichever package built last, not this one's next build. No toolchain stage (`system`) means no toolchain drift. Reported by default; `--rebuild-on-toolchain-drift` rebuilds. **Config defaults.** All three drift-rebuild flags (`--rebuild-on-drift`, `--rebuild-on-toolchain-drift`, `--rebuild-on-flag-drift`) fall back to a sysforge.toml `[update]` default (`rebuild_on_drift`, `rebuild_on_toolchain_drift`, `rebuild_on_flag_drift` respectively) via `_resolve_drift_axes`, which resolves each axis through `config.resolve_flag_default`. The CLI flag always wins when explicitly passed; the config value only supplies the default when it isn't. diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index 60a5ac07..ed8dac54 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -911,6 +911,10 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-B144` — `update --devel` no longer re-fetches every `-git` package's AUR repo, and a fork is never mistaken for the AUR package of the same name.** `--devel` checks whether *upstream* has new commits, and the version check does that itself (`git ls-remote`, or the PKGBUILD's `pkgver()`). Source sync also fetched every VCS package's AUR packaging repo, even when the AUR reported it unchanged: one `git fetch` per package, which adds up when you track many `-git` packages, and more when the AUR is slow. Those repos are now fetched only when the AUR says they changed, or your copy moved, as already happened without `--devel`. With the source freeze on, such a package is still reported as held by the freeze, so `update --devel --frozen` exits 1 as before. Separately, that check matched packages by name only, so a package you cloned from your own fork, when the AUR has one of the same name, could go unfetched until the AUR's copy changed. It now applies only to a checkout cloned from that package's own AUR repo; a fork, another package's repo or a checkout with no `origin` is always fetched, and switching a checkout from a fork back to the AUR fetches once before the check applies again. +--- + +- **`3.5.0-B145` — Rust packages built in the sandbox no longer show up as toolchain drift on every run.** In the build sandbox, a package that builds with Arch's `rust` is compiled with the repo's LLVM, because that is the LLVM Arch's `rust` is linked against. sysforge recorded that honestly, but then compared it with your own toolchain on every `update`, so with a sysforge-built LLVM every such package (the COSMIC `-git` packages, `mesa`, and other Rust-based packages) was reported as toolchain drift each time. With `rebuild_on_drift` or `--rebuild-on-toolchain-drift` on, they were rebuilt on every run, only to record the same thing again. Toolchain drift now leaves such a package alone while its next build would still use the repo LLVM, as it already did for a package kept on the other compiler family on purpose. It is reported again if you turn the sandbox off for it, or it stops needing Arch's `rust`. An update of the repo LLVM itself is not detected for these packages. + ## Security - **`3.5.0-B37` — the root pacman hook no longer follows links in the state directory.** The hook helper runs as root in every pacman transaction and wrote into `/var/lib/sysforge/sentinels`, whose parent any `sysforge` group member can write to. A member could swap `sentinels` for a symlink to another directory, which the next transaction made group-writable, or symlink one of its files so the hook appended to any file as root. The helper now enters the directory and acts only if it is the real `sentinels`, owned by root. It appends only to a regular file, opened so that a symlink is refused and a planted FIFO cannot stall the transaction. As before, it never fails a transaction. diff --git a/sysforge/primitives/flag_drift.py b/sysforge/primitives/flag_drift.py index d1743a5b..3f2b0668 100644 --- a/sysforge/primitives/flag_drift.py +++ b/sysforge/primitives/flag_drift.py @@ -25,6 +25,7 @@ preserved_system_tokens=None, sandbox_policy=None, state_dir=None) -> FlagDriftResult current_sandboxed(entry, config, conflict_groups, policy) -> bool | None + builds_with_repo_compiler(entry, config, conflict_groups, policy) -> bool | None current_compiler(entry, config, conflict_groups) -> str | None """ from __future__ import annotations @@ -191,6 +192,25 @@ def current_sandboxed(entry: dict, config: dict, conflict_groups, policy) -> boo return _sandboxed_for(entry, resolved, policy, build_mode) +def builds_with_repo_compiler(entry: dict, config: dict, conflict_groups, + policy) -> bool | None: + """Whether a fresh build of a recorded package would compile with the + repo LLVM in the sandbox, not the host toolchain (3.5.0-B145). + + That is the case ``build_sandbox.recorded_toolchain`` stamps as + ``sandbox|…``: the package builds sandboxed and pulls Arch's ``rust``, + which links the repo LLVM, so the sandbox swaps that in. Toolchain drift + asks this before reporting such a stamp, since a rebuild would record it + again. None when the PKGBUILD is missing or unparsable. + """ + from sysforge.primitives.build_sandbox import needs_repo_rust + + sandboxed = current_sandboxed(entry, config, conflict_groups, policy) + if sandboxed is None: + return None + return sandboxed and needs_repo_rust(Path(entry["pkgbuild_dir"]) / "PKGBUILD") + + def current_compiler(entry: dict, config: dict, conflict_groups) -> str | None: """The ``CC`` a fresh build of a recorded package would use, so toolchain drift can tell a package kept on the other toolchain on purpose from one diff --git a/sysforge/update.py b/sysforge/update.py index 02bdb2d5..7ad941a9 100644 --- a/sysforge/update.py +++ b/sysforge/update.py @@ -79,6 +79,7 @@ ) from sysforge.primitives.flag_drift import ( STATUS_PARSE_ERROR, + builds_with_repo_compiler, current_compiler, current_sandboxed, drift_candidate, @@ -1148,6 +1149,17 @@ def _phase_toolchain_drift(run: UpdateRun) -> int | None: rec_variant = rec.get("toolchain_variant") if rec_variant is None: continue + # Built in the sandbox with the repo LLVM (a Rust package: + # Arch's rust links it), recorded as `sandbox|…`. While the next + # build would do the same, the host toolchain is not its + # compiler and a rebuild would record the same stamp, so it is + # not drift (3.5.0-B145; the B125 rule, for the same family). + if str(rec.get("toolchain_fingerprint") or "").startswith("sandbox|"): + if _cgroups is None: + _cgroups = load_conflict_groups() + if builds_with_repo_compiler(rec, run.config, _cgroups, + build_sandbox.get_policy()): + break if rec_variant != run.active_variant: # The other family on purpose (a [package_compiler_overrides] # row, a profile's toolchain): the build used the compiler diff --git a/tests/test_flag_drift.py b/tests/test_flag_drift.py index 74446245..7ed0c2d9 100644 --- a/tests/test_flag_drift.py +++ b/tests/test_flag_drift.py @@ -603,3 +603,21 @@ def fake_resolve(pkgmeta, *, state_dir, cc, owner_stage): assert seen["cc"] == "clang" _inputs_diff({"build_inputs": ""}, {}, {}, tmp_path) assert seen["cc"] == "gcc" + + + +# 3.5.0-B145: the one predicate for "the next build records a sandbox +# repo-compiler stamp" — sandboxed now, and Arch's rust in the build deps. + +@pytest.mark.parametrize("sandboxed, repo_rust, expected", [ + (True, True, True), + (True, False, False), # sandboxed, but builds with the host LLVM + (False, True, False), # a host build uses the host toolchain + (None, True, None), # PKGBUILD gone or unparsable: unknown +]) +def test_builds_with_repo_compiler(tmp_path, monkeypatch, sandboxed, repo_rust, expected): + from sysforge.primitives import build_sandbox, flag_drift + monkeypatch.setattr(flag_drift, "current_sandboxed", lambda *a: sandboxed) + monkeypatch.setattr(build_sandbox, "needs_repo_rust", lambda path: repo_rust) + entry = {"pkgbuild_dir": str(tmp_path)} + assert flag_drift.builds_with_repo_compiler(entry, {}, {}, None) is expected diff --git a/tests/test_update_phases.py b/tests/test_update_phases.py index d2d7026a..06dbeef5 100644 --- a/tests/test_update_phases.py +++ b/tests/test_update_phases.py @@ -231,3 +231,34 @@ def test_toolchain_drift_phase_honours_a_deliberate_other_family( run.results = [_result("foo")] update._phase_toolchain_drift(run) assert bool(run.drifted) is drifts + + +# 3.5.0-B145: the sandbox builds a Rust package with Arch's rust, and so with +# the repo LLVM, and records that as a `sandbox|` fingerprint. While the next +# build would do the same, a host toolchain change cannot reach it: rebuilding +# would record the same thing, so it is not drift. + +_SANDBOX_FP = "sandbox|clang|size=1|mtime=2|libllvm=3:4" + + +@pytest.mark.parametrize("fingerprint, active, repo_compiler, drifts", [ + (_SANDBOX_FP, "pgo_llvm", True, False), # Rust in the sandbox, as before + (_SANDBOX_FP, "stock_llvm", True, False), # same variant, other fingerprint + (_SANDBOX_FP, "pgo_llvm", False, True), # sandbox off now / no repo rust + (_SANDBOX_FP, "pgo_llvm", None, True), # PKGBUILD gone: report as before + ("/usr/bin/clang|size=9", "pgo_llvm", True, True), # a host build: unchanged +]) +def test_toolchain_drift_phase_skips_a_repo_compiler_sandbox_build( + tmp_path, monkeypatch, fingerprint, active, repo_compiler, drifts): + monkeypatch.setattr(update, "load_conflict_groups", lambda: {}) + monkeypatch.setattr(update, "current_compiler", lambda *a: "clang") + monkeypatch.setattr(update, "builds_with_repo_compiler", lambda *a: repo_compiler) + run = _run(tmp_path) + run.bs.record("foo-git", "1", "1", "0", "foo-git", tmp_path, + build_mode="source_built", toolchain_variant="stock_llvm", + toolchain_fingerprint=fingerprint) + run.active_variant = active + run.active_fingerprint = "/usr/bin/clang|size=7" + run.results = [_result("foo-git")] + update._phase_toolchain_drift(run) + assert bool(run.drifted) is drifts