Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -3410,7 +3410,7 @@ Why the argv *shape* branches rather than composing another `build_throttle.wrap

**The final conf is audited before the build (`3.4.0-F2`).** Every correction above was found the slow way: a build error several layers away from a host path or binary that crossed into the container. `audit_chroot_conf` runs on `chroot_conf_text`'s output *after* `provision_toolchain` and `provision_profile_data`, so it judges the real `root/` tree rather than predicting it, and refuses (`SandboxUnavailable`) on anything the correction sets missed. It folds the text into its **effective** assignments (`config.parse_makepkg_conf_text`, last assignment wins, so the copied host `PKGDEST` is superseded by `/pkgdest`) and scans only what a tool reads at build time: keys ending in `FLAGS`, exported keys, `BUILDENV` and the dest keys. Package-internal locations (`DBGSRCDIR`, `MAN_DIRS`, `PURGE_TARGETS`, …) are not filesystem inputs and are not scanned, and a toolchain key counts only when it is **exported**, because makepkg exports the `*FLAGS` keys, `CHOST` and `MAKEFLAGS` and nothing else, so a plain `AR=llvm-ar` in a host conf is inert. An absolute path passes when it is under a container prefix (the dest keys, `/startdir`, `/tmp`), under `/usr` or `/opt` (dependencies install into the working copy after the root is cloned), present in `root/`, or part of a `-f{file,debug,macro}-prefix-map=` rewrite rule; a binary (an exported toolchain key, a `-fuse-ld=` linker, an exported `*_WRAPPER`, an enabled `ccache`/`distcc`) must resolve in `root/usr/bin`. The refusal names the key, the value and the correction set that should have covered it. Every member of the bug class so far (`3.2.0-B2`, `B4`, `B6`, `B7`, `B10`, `B12`) would have been one such refusal; `3.2.0-B5`, a stale chroot package database, is outside it.

**Provenance comes from the build that ran (`3.4.0-B3`).** `toolchain_variant`/`toolchain_fingerprint` are resolved once per run from the host, but a sandboxed build compiles with the container's compiler, so stamping the host values made toolchain drift compare the host against itself. `build_argv` now passes `-l sysforge` (`COPY_NAME`), which fixes the working copy at `<chroot>/sysforge`. After a successful sandboxed build the seam parks a `SandboxProvenance(cc_name, copy_identity)` on the module session, where `copy_identity` is `build_fingerprint.static_identity` of the copy's compiler: size and mtime of the driver and its `libLLVM.so*`, never executed (it would load host libraries). `_record_build_state` takes it before anything can return; `recorded_toolchain` keeps the host's variant and fingerprint when the host's `/usr/bin/<cc>` has the same static identity (pacman restores mtimes from the archive, so the same package file matches), and otherwise records `stock_llvm`/`gcc` with a `sandbox|…` fingerprint that can never equal a host one. `invoke_makepkg` clears any pending provenance on entry, so a failed build never inherits an earlier success's. `build_state` gains `sandboxed`, and flag drift a matching sub-axis (§Drift detection). The same holds for a host build on the other compiler family (`3.5.0-B125`): `_own_compiler_variant` records `gcc`/`stock_llvm` and no fingerprint when the build's `CC` (the raw `--cc`, else the resolved profile's) is not the active variant's family. The decision is `pipeline_state.variant_built_with`. The toolchain stage's builds (which record their own variants) and an unrecognised compiler name keep the run-wide values; a kernel built with the other family records it through its `cc_override` (`3.5.0-B126`).
**Provenance comes from the build that ran (`3.4.0-B3`).** `toolchain_variant`/`toolchain_fingerprint` are resolved once per run from the host, but a sandboxed build compiles with the container's compiler, so stamping the host values made toolchain drift compare the host against itself. `build_argv` now passes `-l sysforge` (`COPY_NAME`), which fixes the working copy at `<chroot>/sysforge`. After a successful sandboxed build the seam parks a `SandboxProvenance(cc_name, copy_identity)` on the module session, where `copy_identity` is `build_fingerprint.static_identity` of the copy's compiler: size and mtime of the driver and its `libLLVM.so*`, never executed (it would load host libraries). `_record_build_state` takes it before anything can return; `recorded_toolchain` keeps the host's variant and fingerprint when the host's `/usr/bin/<cc>` has the same static identity (pacman restores mtimes from the archive, so the same package file matches), and otherwise records `stock_llvm`/`gcc` with a `sandbox|…` fingerprint that can never equal a host one. Toolchain drift does not report that stamp while the next build would record it again (`3.5.0-B145`, `flag_drift.builds_with_repo_compiler`; §Drift detection). `invoke_makepkg` clears any pending provenance on entry, so a failed build never inherits an earlier success's. `build_state` gains `sandboxed`, and flag drift a matching sub-axis (§Drift detection). The same holds for a host build on the other compiler family (`3.5.0-B125`): `_own_compiler_variant` records `gcc`/`stock_llvm` and no fingerprint when the build's `CC` (the raw `--cc`, else the resolved profile's) is not the active variant's family. The decision is `pipeline_state.variant_built_with`. The toolchain stage's builds (which record their own variants) and an unrecognised compiler name keep the run-wide values; a kernel built with the other family records it through its `cc_override` (`3.5.0-B126`).

**The heartbeat sets detail; the pty loop paints.** `makepkg_invoke._on_idle` fires every `MAKEPKG_HEARTBEAT_S` of child silence, logs a `[heartbeat]` line (the latest `\r`-redrawn ninja status, or "no output") to the per-package log, and hands the same text to `progress_hooks.hooks().heartbeat()`, which only attaches it to the bar's line. Putting it on screen is the scheduled painter's job (`3.3.0-F2`): at default verbosity `run_with_pty` forwards makepkg's raw bytes, so it enters `forwarding_output()` (the bar's ticker stands down) and calls `refresh()` at most once a second, only when `_StreamBoundary` says the forwarded stream sits between escape sequences and UTF-8 characters. At `-vvv` output arrives as whole log lines and the ticker stays in charge.

Expand Down Expand Up @@ -4232,7 +4232,7 @@ Escape hatches:

*Flag drift.* Same package version but a different resolved compiler configuration — e.g. profile changed, new flag added, or build mode switched. At build time the resolved flags string is recorded per package in `build_state.toml`; Phase 4.3 re-resolves the current profile for each source-built package and diffs the result against the stored string (via `primitives/flag_drift.resolve_flag_drift`). Both the recorded and the re-resolved string are the **effective** flags — the resolved profile after the `[preserved_system_tokens]` pass, produced on both sides by `makepkg_conf.serialize_effective_flags` (3.1.0-B11). Consequently the axis also covers changes to `[preserved_system_tokens]`, `[append_conflict_groups]`, and `/etc/makepkg.conf`'s own hardening baseline, none of which were visible to it before. Flag drift is **reported by default** and is network-free (`--offline --dry-run` gives a read-only report); `--rebuild-on-flag-drift` rebuilds the drifted packages in this run's walk — its `ui()` notice is built from that queue (`update._partition_flag_drift`, the promotion loop's own predicate) and names separately what drifted but was not queued, with the command that would rebuild it (`sysforge run <owner_stage>` for a stage-owned kernel/toolchain package, else `sysforge build <pkg>`; 3.2.0-B40) — and `--rebuild-on-drift` is the umbrella over both the flag and toolchain-variant axes. See §`update.py` → *Phase 4.3 — Flag drift*. The axis has one sub-axis that is not a flag: **`sandboxed`** (3.4.0-B3). A record whose `sandboxed` differs from what a fresh build would do (the process-wide sandbox policy through `for_profile`, always `False` for a stage-owned package) gets a ` sandboxed: <old> → <new>` line in the same diff vocabulary and is promoted by `--rebuild-on-flag-drift` like any other. It is judged before the `!buildflags` and no-recorded-flags returns, because the sandbox changes the built artifact (dependency versions, the compiler, automatically detected optional features) even where makepkg ignores conf flags. A record without the field never drifts. **Build inputs** (3.2.0-F18) are the second non-flag sub-axis: settings that change the artifact through a seam the profile never sees. `primitives/build_inputs.resolve_build_inputs` returns the ones that apply to a package — `python` (the `major.minor` of the interpreter `makepkg_env.resolve_build_python` pins, for a package whose runtime `depends` names `python`/`python-*`; `major.minor` rather than the path, so a patch-level update never drifts, and runtime `depends` rather than `makedepends`, so a package that only runs python while building is not tied to its version), `mesa_gallium`/`mesa_vulkan` (`resolve_or_detect_mesa_drivers`, only with `[mesa] filter_drivers` on), `llvm_targets` (`resolve_or_detect_llvm_targets`, never for `lib32-*`), and `pgo_profile` (`mesa_pgo.reuse_profdata` — the merged profile a rebuild's reuse branch would splice in through `compiler_flags_extra` — only for an LLVM compiler and a non-stage-owned build, mirroring that branch's gates). `_record_build_state` stores the result as `build_inputs`, serialized in `flags_string`'s own `KEY=value` line form, and `flag_drift._inputs_diff` re-derives it through the **same function** with the compiler resolved the same way, then renders it with `diff_flags`, so drift lines read ` python: '3.13' → '3.14'`, ` +pgo_profile: …`. Recording the resolved value rather than what a patcher reported applying is deliberate: a patch whose anchor was missing would otherwise record "no drivers" against a resolver that says "drivers", and the drift would never clear however often the package rebuilt (the B11 rule). The sub-axis is enabled by passing `state_dir` (which locates `hardware_profile.toml`), is judged before the `!buildflags` and no-recorded-flags returns like the sandbox line, and reads an absent field as unknown, so an upgrade flags nothing. It is also the **only** axis judged for a PGO-mode entry (`build_mode` `pgo`): `flag_drift.drift_candidate` admits such an entry once it has recorded `build_inputs`, and its profile flags stay unjudged, as they were before the field existed. Out of scope: `kernel.toml` kconfig inputs (stage-owned, diffed request-vs-result by the kernel stage's Gate 2) and the toolchain stage's own PGO/BOLT settings; an LLVM package's `llvm_targets` drift lands in the build-state-wide fold with the owning-stage hint like any other stage-owned entry.

*Toolchain drift.* Same package version, built by a different compiler than a fresh build would use. Every source build records `toolchain_variant` (`gcc`/`stock_llvm`/`pgo_llvm`) and `toolchain_fingerprint`; Phase 4.25 (`update._phase_toolchain_drift`) compares them with the active toolchain (`get_toolchain_variant`/`get_toolchain_fingerprint`). A different variant drifts, and so does the same variant with a different fingerprint (a toolchain rebuild with unchanged soname, compared only when both sides have one). The recorded variant names the compiler the build actually used (`3.5.0-B125`): a package swapped to the other family by `build --toolchain`, a `[package_compiler_overrides]` row or a profile's own `toolchain` records that family's variant (`stock_llvm` for clang, as a sandboxed build's repo clang does) and no fingerprint (`makepkg_wrapper._own_compiler_variant`, via `pipeline_state.variant_for_compiler`); the toolchain stage's builds keep the run-wide values, and a kernel records its own compiler like any other build (`3.5.0-B126`). Phase 4.25 then skips a record in the other family only while a fresh build would still use that family (`flag_drift.current_compiler`, the same profile resolution flag drift and the sandbox sub-axis use), so a package kept on gcc on purpose is never rebuilt toward clang, and a one-off `--toolchain` build still drifts back. No toolchain stage (`system`) means no toolchain drift. Reported by default; `--rebuild-on-toolchain-drift` rebuilds.
*Toolchain drift.* Same package version, built by a different compiler than a fresh build would use. Every source build records `toolchain_variant` (`gcc`/`stock_llvm`/`pgo_llvm`) and `toolchain_fingerprint`; Phase 4.25 (`update._phase_toolchain_drift`) compares them with the active toolchain (`get_toolchain_variant`/`get_toolchain_fingerprint`). A different variant drifts, and so does the same variant with a different fingerprint (a toolchain rebuild with unchanged soname, compared only when both sides have one). The recorded variant names the compiler the build actually used (`3.5.0-B125`): a package swapped to the other family by `build --toolchain`, a `[package_compiler_overrides]` row or a profile's own `toolchain` records that family's variant (`stock_llvm` for clang, as a sandboxed build's repo clang does) and no fingerprint (`makepkg_wrapper._own_compiler_variant`, via `pipeline_state.variant_for_compiler`); the toolchain stage's builds keep the run-wide values, and a kernel records its own compiler like any other build (`3.5.0-B126`). Phase 4.25 then skips a record in the other family only while a fresh build would still use that family (`flag_drift.current_compiler`, the same profile resolution flag drift and the sandbox sub-axis use), so a package kept on gcc on purpose is never rebuilt toward clang, and a one-off `--toolchain` build still drifts back. The same rule covers a build the sandbox compiled with the repo LLVM (`3.5.0-B145`): a Rust package pulls Arch's `rust`, which links the repo LLVM, so the sandbox swaps that in and `build_sandbox.recorded_toolchain` records `stock_llvm` with a `sandbox|…` fingerprint, which can never equal a sysforge-built host toolchain. Phase 4.25 skips such a record while `flag_drift.builds_with_repo_compiler` says the next build would do the same (sandboxed for its profile, per `current_sandboxed`, and `build_sandbox.needs_repo_rust`); before B145 every such package drifted on every run and, under `rebuild_on_drift`, was rebuilt each time to record the same stamp again. A change of the repo LLVM itself is not detected for these packages: the sandbox's working copy is shared by every build, so its compiler identity at update time reflects whichever package built last, not this one's next build. No toolchain stage (`system`) means no toolchain drift. Reported by default; `--rebuild-on-toolchain-drift` rebuilds.

**Config defaults.** All three drift-rebuild flags (`--rebuild-on-drift`, `--rebuild-on-toolchain-drift`, `--rebuild-on-flag-drift`) fall back to a sysforge.toml `[update]` default (`rebuild_on_drift`, `rebuild_on_toolchain_drift`, `rebuild_on_flag_drift` respectively) via `_resolve_drift_axes`, which resolves each axis through `config.resolve_flag_default`. The CLI flag always wins when explicitly passed; the config value only supplies the default when it isn't.

Expand Down
Loading
Loading