diff --git a/CLAUDE.md b/CLAUDE.md index e1709318..646d6480 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -79,12 +79,14 @@ Shipped-file edits must pass `make check-shipped`; doc/design edits `make check- vocabulary outside the closing "In sysforge" section, upstream sources cited, no numbers measured on one machine (`tests/test_concept_pages.py` checks the closing sections, the upstream links and the vocabulary); guides link them (and the glossary) rather than re-explaining. -- **`DOC` IDs are for user-facing docs only** (`docs/guides/`, `docs/concepts/`, README, man page) and file under +- **`DOC` IDs are for user-facing docs only** (`docs/guides/`, `docs/concepts/`, README, man page, + comments in shipped `etc/sysforge/*.toml`) and file under `## Changed` (patch; lint-enforced). Internal docs (design, CLAUDE.md, ROADMAP) get no ID; wrong `--help`/man output is a `B`, since it is CLI behaviour. - **`DEV` IDs are for contributor-only work** (only `tests/`, `tools/`, `Makefile`, `.claude/`, dev docs) and file their release-note entry under `## Changed` only, like `DOC` (lint-enforced, - `3.4.0-STD1`). Touching `sysforge/` or a shipped file makes it `B`/`F`. + `3.4.0-STD1`). Touching `sysforge/` or a shipped file makes it `B`/`F` (a comment-only edit to a + shipped config is `DOC`). - **Completions stay in lockstep with the CLI** (`completions/_sysforge` + bash) in the same change. - **A user-facing change regenerates `tests/data/surface.json`** (`make surface`) and names what changed in the release notes; `make check-surface` (a required CI gate) enforces both. diff --git a/DESIGN.md b/DESIGN.md index 3b66f6c8..1f06813e 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -394,7 +394,7 @@ The orthogonality of the roles means: Each entry overrides at most these fields (all optional except `name`): - `source` — `repo` (pacman) vs `aur`/`git`/`local`. A **routing hint**: it tells the bootstrap/build paths how to obtain the package, falling through to pacman / AUR RPC inference if omitted. It does **not** by itself put a package under steady-state tracking — tracking comes from sysforge having built the package (build_state), not from a manifest entry. So an entry with only `name` + `source` has no override effect at steady-state, and `sysforge packages add` rejects it (pair it with a behavior-changing field — `enable_build_from_source`, `cache`, `reason` — for the entry to override anything). When sysforge builds a package, it records the resolved `source` in build_state, so the registry is self-describing without re-inferring origin later. - `enable_build_from_source` *(bool)* — if `true`, this **repo** package is built from source (via `pkgctl repo clone` + makepkg with sysforge flag profiles) instead of installed as the binary via pacman. It also opts the package into `sysforge build` without a confirmation prompt (see §`build`). The boolean replaced the misleadingly-named `pkgbuild_patch` (which never patched anything — real PKGBUILD patching is gated by separate predicates); as of 3.0.0 the legacy `pkgbuild_patch` key is no longer honored on read (`config.normalize_package_entry` is now an identity function) — a manifest that still spells it no longer opts the package into a source build. `packages_cmd` still rewrites the key to the current name on the next `packages.toml` write (`add`/`remove`), so a file touched since the rename self-cleans; a never-rewritten file keeps the stale key inert. -- `cache` *(bool)* — `false` disables ccache/sccache for this package (required for PGO stages). +- `cache` *(bool)* — `false` builds this package with ccache and sccache off: `CCACHE_DISABLE=1` and an empty `RUSTC_WRAPPER` in its makepkg environment (`config.cache_disabled_for`, applied in `makepkg_wrapper.run`). ```toml [build] @@ -780,7 +780,7 @@ The hardware stage (stage 2) needs no config — it auto-detects and writes `har - On `NotImplementedError`: prints `--start-from` guidance and exits - On `BootstrapRebootRequired`: saves state, prints reboot + resume instructions, exits 0 (clean stop, not failure) - On `RuntimeError`: saves state and exits with resume instructions -- `--dry-run`: logs what would run without calling `stage.run()` +- `--dry-run`: `run pipeline` lists the stages it would run without calling `stage.run()` (the bootstrap stages are not dry-run safe). A standalone `run --dry-run` calls `stage.run()` with `options.dry_run` set, after a `[dry-run] ` line through `ui()`: each stage stops short of its own changes and says what it would do, so the preview reads the real config (the kernel's "Kernel build plan", the toolchain's install set). The runner opens `PipelineState(read_only=True)`, so a stage's mid-run checkpoints never reach the file, the kernel and packages stages take no sentinel, and the toolchain stage's `resolve_all_pkgbuilds(dry_run=True)` reads only checked-out PKGBUILDs, raising `CheckoutsMissing` (the stage names what it would clone and ends the preview) instead of cloning, syncing or purging (`3.5.0-B141`; before it the runner returned before `stage.run()` and logged one INFO line, so the preview showed nothing). Guard against accidental state clobber: if a state file exists and neither `--resume` nor `--start-from` is passed, the runner exits with instructions rather than overwriting. Both flags are supported on `sysforge run pipeline`. @@ -1421,7 +1421,8 @@ spelling per meaning, across every verb: | `revert-to-stock` | resolve state dir + `plan_revert` over `BuildState` (pure, no mutation) | prompt/`--yes`/`--dry-run` gate, then per-target reinstall / atomic replace / derename-then-reinstall via `pacman` | (state written inline) | yes | | `bench toolchain` | null | resolve preconditions → workload → unprivileged stock fetch + extract → interleaved timing → report → `bench.toml` | null | no | | `search` | null | print installed → repo → AUR sections (fixed order, empty omitted) for one term, with in-line installed / stand-in (`-sysforge`, `-git` drop-in) markers | null | no | -| `uninstall` | resolve state dir + `plan_uninstall` over `BuildState` (pure) | `pacman -Rnsu`, then demote any tracked target via `state forget` + reconcile | (state written inline) | yes | +| `uninstall` | resolve state dir + `plan_uninstall` over `BuildState` (pure) | `--yes`/terminal gate, `pacman -Rnsu`, then demote any tracked target via `state forget` + reconcile | (state written inline) | yes | +| `self-uninstall` | `plan_self_uninstall` over `BuildState`, the sync DB, pacman.conf and the package dirs (reads only) | `--dry-run`/`--yes` gate; `apply_reverts` under its own sentinel; drop `IgnoreGroup` and the chroot repo block; one `sudo sh -c` removing the package, then the data and group | null | own scope, reverts only | | `run …` namespace | build `RunOptions` | delegate to `pipeline.run_pipeline` / `run_stage_standalone` | pipeline framework | (pipeline owns it) | | `artifact list` | null | `primitives/artifacts.unified_rows()` + PATH check + optional `scan()` for `--unmanaged` | null | no | | `artifact review` | `--all`, `--include-unknown` | interactively offer discovered candidates for adoption via `primitives/artifacts.iter_offerable()`; off-TTY lists candidates + adopt hint; `--all` bulk-adopts the same offer set without prompting | null | no | @@ -1432,7 +1433,7 @@ spelling per meaning, across every verb: ### `revert-to-stock` -Undoes a source-built or optimized package back to its official repo version. `pre_check` resolves the state dir and calls `plan_revert(bs, targets)` (pure — no mutation) to classify each target against `BuildState`, including a reverse lookup via the shared `install_reconcile.resolve_installed_name(bs, name)` helper (iterated in sorted key order so a stock base always resolves to the same pkgname deterministically) so naming the *stock* base of a renamed build (e.g. `mesa` when the tracked entry is `mesa-sysforge`) still resolves, as does a renamed split member's own stock name (`vulkan-radeon` → `vulkan-radeon-sysforge`, whose `origin_pkgbase` is `mesa`, `3.5.0-B62`). That helper is the single home for the reverse lookup — `uninstall` uses it too. Four actions, distinguished by `profile.is_optimized_build_mode(mode)` and — for optimized builds — `profile.rename_mode_for_build_mode(mode)`: +Undoes a source-built or optimized package back to its official repo version. `pre_check` resolves the state dir and calls `plan_revert(bs, targets)` (pure — no mutation) to classify each target against `BuildState`, including a reverse lookup via the shared `install_reconcile.resolve_installed_name(bs, name)` helper (iterated in sorted key order so a stock base always resolves to the same pkgname deterministically) so naming the *stock* base of a renamed build (e.g. `mesa` when the tracked entry is `mesa-sysforge`) still resolves, as does a renamed split member's own stock name (`vulkan-radeon` → `vulkan-radeon-sysforge`, whose `origin_pkgbase` is `mesa`, `3.5.0-B62`). That helper is the single home for the reverse lookup — `uninstall` uses it too. `plan_revert` and `apply_reverts(actionable, state_dir)` live in `verbs/shared.py`, shared with `self-uninstall`. Four actions, distinguished by `profile.is_optimized_build_mode(mode)` and — for optimized builds — `profile.rename_mode_for_build_mode(mode)`: - **`skip`** — untracked or already `build_mode = "pacman"`; nothing to do. - **`reinstall`** — plain `source_built` (or `bolt_llvm`, a BOLTed toolchain member, which also keeps its stock name), installed under the stock name: just reinstall the repo package (`pacman -S `). The name is the member itself, never its pkgbase (`pacman -S mesa` left a source-built `vulkan-radeon` in place); naming the pkgbase of a split set reinstalls every installed member, matching what `forget_packages` drops (`3.5.0-B62`). @@ -1465,7 +1466,13 @@ Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspe ### `uninstall` -Mutating lifecycle verb (`requires_sentinel = True`). `pre_check` resolves each name through the shared `resolve_installed_name` (so naming a stock base reaches its `-sysforge` build) and builds a pure `plan_uninstall` classifying each target's installed name + whether it's tracked. `execute` narrates the plan, removes via `pacman.uninstall_pkgs` (`pacman -Rnsu`, interactive — pacman prints its own confirmation; a `CalledProcessError` returns exit 1 and demotes nothing), clears the removed targets' recorded build failures through `install_reconcile.clear_removed_failures` (tracked or not, before the forget whose deleted records map a split member to its pkgbase; `3.5.0-B120`), then for any tracked target demotes it out of the build-state authority via `cmd_state_forget` (the same `state forget` code) followed by an `install_reconcile.consume_external_installs(bs)` pass. This is the exact demotion composition `revert-to-stock` uses — no parallel path — so `update` stops rebuilding an uninstalled package. +Mutating lifecycle verb (`requires_sentinel = True`). `pre_check` resolves each name through the shared `resolve_installed_name` (so naming a stock base reaches its `-sysforge` build) and builds a pure `plan_uninstall` classifying each target's installed name + whether it's tracked. `execute` narrates the plan, removes via `pacman.uninstall_pkgs` (`pacman -Rnsu`, interactive — pacman prints its own confirmation; a `CalledProcessError` returns exit 1 and demotes nothing), clears the removed targets' recorded build failures through `install_reconcile.clear_removed_failures` (tracked or not, before the forget whose deleted records map a split member to its pkgbase; `3.5.0-B120`), then for any tracked target demotes it out of the build-state authority via `cmd_state_forget` (the same `state forget` code) followed by an `install_reconcile.consume_external_installs(bs)` pass. This is the exact demotion composition `revert-to-stock` uses — no parallel path — so `update` stops rebuilding an uninstalled package. Without a terminal it refuses before pacman runs unless `--yes` is given, which passes `--noconfirm`: pacman's own `[Y/n]` read EOF and aborted after printing the transaction (`3.5.0-B143`). + +### `self-uninstall` + +Removes SysForge in one command (`3.5.0-F19`), replacing the manual list the "What SysForge touches" guide gave. `pre_check` builds a `SelfUninstallPlan` by reading only: every `build_state` entry not built by pacman goes through `plan_revert`, a split set planned once; a plan whose stock names (`pacman.get_sync_pkgnames`, one `pacman -Slq`) a repo carries is reverted, any other (an AUR or local build, a custom kernel, an FDO kernel whose plain kernel is itself local) is kept installed with the reason. It also records whether `/etc/pacman.conf` has `sf-build` in `IgnoreGroup` (`verbs.shared.has_ignore_group`), whether the sandbox chroot lists the local repo (`local_repo.has_chroot_section`), the package that owns `sysforge/__init__.py` (none from a checkout), which of `paths.PACKAGE_DIRS` and the XDG user dirs exist, and whether the `sysforge` group exists. `PACKAGE_DIRS` is fixed (`/etc/sysforge`, `/var/lib/sysforge`, `/var/cache/sysforge`), never `SYSFORGE_CONFIG_DIR`/`SYSFORGE_STATE_DIR`, which can name a developer's own tree. + +`execute` prints the plan, stops on `--dry-run`, and otherwise needs `--yes` or a terminal's `y` and an authorised `sudo`. Order is what makes a failure safe: (1) the reverts through `verbs.shared.apply_reverts`, the one home `revert-to-stock` also calls, inside the verb's own `sentinel_scope` (recovery: reinstall the stock packages) — the verb sets `requires_sentinel = False` and `wants_run_log = False` because the runner's scope and log live in the state dir it is about to delete; a failed revert stops the run with sysforge still installed; (2) `verbs.shared.remove_ignore_group` (`drop_ignore_group`, the inverse of setup's `_patch_conf_text`; the IgnoreGroup facts live in `verbs/shared.py`, since a verb module may not import a sibling) and `local_repo.remove_chroot_section`; (3) `removal_script`, one `sudo sh -c` running `pacman -Rns --noconfirm `, then `rm -rf` of the package dirs and `groupdel sysforge`. Step 3 is last and single because once pacman has deleted the package this process must import nothing more from it; after it only `shutil.rmtree` of the user dirs and a `print` run. The package's pacman hooks are gone before pacman runs post-transaction hooks, so none writes into the state dir being deleted. `--keep-data` uses `pacman -Rs` (edited config kept as `.pacsave`) and skips the data, group and user dirs. `tools/e2e/scenarios/99-self-uninstall.sh`, the tier's last scenario, runs the whole thing on a live system. ### `artifact list` @@ -2366,7 +2373,7 @@ Log tag: `[PROV]`. The `suggest_*` / `files_db_present` queries are pure read-on Cross-cutting failure scenario handler. Imported by `makepkg_wrapper` and `dep_analysis` to avoid circular imports. -`handle_failure(scenario, message, config, fallback=None)` dispatches to `abort`, `error`, `warn_and_fallback`, or `fallback` based on `[failure_handling]` config. `profile_missing` and `tempfile_write_failed` always abort regardless of config. +`handle_failure(scenario, message, config, fallback=None)` dispatches to `abort`, `error`, `warn_and_fallback`, or `fallback` based on `sysforge.toml [failure_handling]`, validated per key by `resolve_failure_handling` against `SCENARIOS` (the one home for every key's values and default). `tempfile_write_failed` always aborts regardless of config, as does a scenario not in `SCENARIOS`. ### `run.py` @@ -2963,11 +2970,11 @@ toolchain = "gcc" # global package-compiler default (see "Toolchain field [profiles.bare] # Fallback profile, no flags +# No profile names a compiler or linker: both come from `toolchain` (see +# "Toolchain field"). Override individual CC/CXX/AR/… keys to win over it. + [profiles.standard] extends = "bare" -# Compiler comes from the `toolchain` field (defaults to "gcc" via [defaults]). -# Override individual CC/CXX/AR/… keys to win over the bundle, or set -# `toolchain = "llvm"` here / in a rule to switch the whole bundle. CFLAGS = "-march=native -O2 -pipe" CXXFLAGS = "$CFLAGS" LDFLAGS = "-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" @@ -3427,23 +3434,18 @@ Preflight refuses rather than downgrades — missing `devtools`, a missing `/src`, and `makepkg_env._logdest_tail(pkgbuild_path)` reads the tail of the newest `$LOGDEST/-*.log` (makepkg's captured stdout when `OPTIONS+=log`). Both `BUILDDIR` and `LOGDEST` are resolved through `pacman.get_builddir()` / `pacman.get_logdest()` — env first, then the layered system `makepkg.conf` — so a user who configures these only in `/etc/makepkg.conf` still gets a diagnosis. Don't read `os.environ["BUILDDIR"]` or assume `~/builds` directly here. -Configuration extends `[failure_handling]` with four scenario keys: +Auto-repair has four keys of its own in the same table: ```toml -vendored_deps_missing = "auto_repair" +vendored_deps_missing = "auto_repair" # auto_repair | auto_repair_with_warning | abort pgp_key_missing = "auto_repair" -srcinfo_drift = "auto_repair_with_warning" -checksum_mismatch = "prompt_user" +srcinfo_drift = "auto_repair_with_warning" # … | warn_and_fallback (preflight only) +checksum_mismatch = "prompt_user" # prompt_user | abort, nothing else ``` -Three new behaviour values join the existing `abort` / `warn_and_fallback` / `fallback` / `error` set: `auto_repair` (repair silently, retry, log at info), `auto_repair_with_warning` (repair, retry, log at `[WARN]`), `prompt_user` (surface the diff and require explicit consent before repairing). In **batch mode** (`batch=true` on the resolved profile) the `prompt_user` behaviour is short-circuited to `aborted` — auto-repair never runs unattended for security-sensitive scenarios. This is the load-bearing rule for `checksum_mismatch`. +`auto_repair` repairs, retries and logs at info; `auto_repair_with_warning` logs the same at `[WARN]`; `prompt_user` surfaces the failing files and requires explicit consent before repairing. `checksum_mismatch` accepts only `prompt_user` or `abort`, so neither an edit nor a typo can select an unattended `updpkgsums`, which would mask a tampered source. In **batch mode** (`batch=true` on the resolved profile) `prompt_user` is short-circuited to `aborted`, so the repair never runs unattended. `auto_repair._normalise_behaviour` maps anything unexpected that still reaches it to `abort`, never to a repair. `.SRCINFO` drift is the one scenario that is **not** in `REGISTRY` — it has a different lifecycle. `_run_build` calls `auto_repair.preflight_srcinfo(pkgbuild_dir, behaviour)` before the build starts, regenerating `.SRCINFO` (with a `[WARN]` by default) when `makepkg --printsrcinfo` differs from the committed file. The other three scenarios fire on retry from a captured failure. @@ -3861,7 +3863,7 @@ This gives hand-crafted prose with OPTIONS that stay automatically in sync with --- ## Hardware Detection -Pipeline stage 3. Probes the running system via `/proc/cpuinfo` and `lspci`, emits `hardware_profile.toml` to `state_dir`. The file feeds kconfig automation (kernel stage) and is shown in the reconfigure config review. +Pipeline stage 2. Probes the running system via `/proc/cpuinfo` and `lspci`, emits `hardware_profile.toml` to `state_dir`. The file feeds kconfig automation (kernel stage) and is shown in the reconfigure config review. **Detections and kconfig output:** @@ -4111,11 +4113,14 @@ codes, `pacman -Qi`, the state files and leftover sentinels, with prompts answer pseudo-terminal. It covers what the unit suite stubs: root versus the user, group membership that applies only at the next login, the state one command leaves for the next. It builds sysforge from the checkout itself, so it needs no `vm-pkg-stable`. `pre-release` runs it; the release preflight -does not (`_RELEASE_GATE_EXEMPT`: minutes long, podman, and one networked scenario, so not +does not (`_RELEASE_GATE_EXEMPT`: minutes long, podman, and four networked scenarios, so not deterministic). CI runs the same scenarios in an `e2e` job of its own, directly in the job's container (`make e2e E2E_HERE=1`), listed in the Makefile's `CI_JOBS` beside `GATES`. A scenario marked `xfail: ` fails until that item is fixed and reports `XFAIL`; once it passes it fails -the run, so the marker comes off with the fix. +the run, so the marker comes off with the fix. Every command in `tests/data/surface.json` is run by a scenario or +listed, with its reason, in `tools/e2e/uncovered.toml` (`3.5.0-DEV23`); `tools/e2e_coverage.py` +does the comparison, `tests/test_e2e_coverage.py` fails the suite on a command in neither set or +a listing a scenario has made redundant, and `make e2e-coverage` prints both. **CI** (`.github/workflows/ci.yml`, `3.4.0-DEV1`) runs every `pre-release` gate on each pull request (a newer push to it cancels the superseded run) and on each push to `main`, in a fresh `archlinux:base-devel` container, as an unprivileged user, with only the @@ -4231,7 +4236,7 @@ Escape hatches: **Config defaults.** All three drift-rebuild flags (`--rebuild-on-drift`, `--rebuild-on-toolchain-drift`, `--rebuild-on-flag-drift`) fall back to a sysforge.toml `[update]` default (`rebuild_on_drift`, `rebuild_on_toolchain_drift`, `rebuild_on_flag_drift` respectively) via `_resolve_drift_axes`, which resolves each axis through `config.resolve_flag_default`. The CLI flag always wins when explicitly passed; the config value only supplies the default when it isn't. -**PGO toolchain packages** (`build_mode = "pgo_llvm_toolchain"`) are handled specially during update. `makepkg_wrapper.run()` reads `toolchain.toml → pgo_store`, checks for a saved `clang.profdata` and its `clang.profdata.version` sidecar, and compares the sidecar's LLVM major version against the PKGBUILD's `pkgver` major. If they match, `-fprofile-use=` is injected and the build proceeds as a PGO-optimised build. If profdata is absent or version-mismatched (e.g. after a major LLVM bump), the user is prompted: **[p]lain build or [s]kip (default: skip)**. In non-interactive mode the build is skipped automatically. Skipped packages are counted separately in the update summary and do not count as failures. To rebuild profdata after a major version bump, run `sysforge run toolchain`. The toolchain stage itself also reuses compatible profdata — see the **Profdata reuse** section under stage 6. +**PGO toolchain packages** (`build_mode = "pgo_llvm_toolchain"`) are handled specially during update. `makepkg_wrapper.run()` reads `toolchain.toml → pgo_store`, checks for a saved `clang.profdata` and its `clang.profdata.version` sidecar, and compares the sidecar's LLVM major version against the PKGBUILD's `pkgver` major. If they match, `-fprofile-use=` is injected and the build proceeds as a PGO-optimised build. If profdata is absent or version-mismatched (e.g. after a major LLVM bump), the user is prompted: **[p]lain build or [s]kip (default: skip)**. In non-interactive mode the build is skipped automatically. Skipped packages are counted separately in the update summary and do not count as failures. To rebuild profdata after a major version bump, run `sysforge run toolchain`. The toolchain stage itself also reuses compatible profdata — see the **Profdata reuse** section under stage 5. **Stale-profraw post-build check.** After every non-PGO-managed build, `makepkg_wrapper.run()` globs `pgo_store` for `*.profraw` files. Any file with `mtime >= build_start - 1s` is treated as **fresh** — it was written by the build just completed, which means an instrumented LLVM is still installed on the system and the build was leaking profile data. The wrapper fatals, telling the user to reinstall `llvm`/`llvm-libs` or run `sysforge run toolchain`. Files strictly older than `build_start` are **orphans** left behind by a prior failed or partial toolchain run whose instrumented binaries the user has since cleaned up; these are unlinked in place and an info line is logged. The split makes the safety net self-healing: once the system is clean, the next build purges the residue automatically instead of requiring manual cleanup of `pgo_store`. **Neither verdict applies while a toolchain PGO run is in progress** (`3.3.0-B8`): if `build_lock.is_held(makepkg_pgo.resolve_pgo_lock_path(tcfg))` reports the `sysforge-pgo.lock` held, the store is that run's live Pass-3 training data, so a fresh file is not a leak and an "orphan" is data its merge monitor has not read yet. The check logs that it is deferring and touches nothing. Before this, a `sysforge build` started alongside `run toolchain` either fatally aborted on the training data or deleted whatever the monitor had not yet merged. `is_held` is a read-only probe: it never creates the lock file or overwrites the holder's PID. diff --git a/Makefile b/Makefile index 088841a8..5b026aed 100644 --- a/Makefile +++ b/Makefile @@ -7,7 +7,7 @@ preflight release-major release-minor release-patch release-resume \ vm-deps vm-image vm-boot vm-boot-gui vm-snapshot vm-loadvm vm-iso vm-monitor vm-console vm-savevm vm-snapshots vm-ssh vm-ssh-builder vm-stop vm-clean \ vm-pkg-stable vm-pkg-git vm-pkg-all vm-install-stable vm-install-git vm-smoke vm-test \ - container-build container-smoke container-smoke-cachyos container-shell container-clean e2e \ + container-build container-smoke container-smoke-cachyos container-shell container-clean e2e e2e-coverage \ ci-local # --------------------------------------------------------------------------- @@ -658,6 +658,12 @@ container-clean: ## Remove the test container and image e2e: ## Run the end-to-end scenarios in fresh Arch containers (SCENARIO=, needs podman) E2E_COMMIT=$(E2E_COMMIT) bash tools/e2e/run.sh $(if $(E2E_HERE),--here) $(SCENARIO) +# Which commands a scenario runs and which tools/e2e/uncovered.toml excuses, +# with its reason (3.5.0-DEV23). tests/test_e2e_coverage.py fails the suite on +# a command in neither set, so this is the readable view of that check. +e2e-coverage: ## List the commands the end-to-end scenarios run, and the excused rest + uv run --no-sync python tools/e2e_coverage.py + # Replay the CI job (.github/workflows/ci.yml, 3.4.0-DEV1) against the working # tree, uncommitted edits included, in the same base image. The tree is mounted # read-only: tools/ci.sh copies it to an unprivileged user before running the diff --git a/ROADMAP.md b/ROADMAP.md index 394d480d..fdacb539 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -44,7 +44,8 @@ neighbour by a `---` rule, so a single item is readable on its own rather than running into the next one. `docs/release-notes/` entries use the same two rules. **`DOC` covers user-facing documentation only** — guides under `docs/guides/`, concept -pages under `docs/concepts/`, `README.md`, the man page — so doc work stays visible in release notes without +pages under `docs/concepts/`, `README.md`, the man page, and the comments in shipped config files +(`etc/sysforge/*.toml`) — so doc work stays visible in release notes without being forced into `F` or `B`. A `DOC` entry always lands under `## Changed`: the section drives the derived SemVer bump, and documentation changes no API (a `DOC` under `## Added` would demand a minor release). `make check-standards` diff --git a/completions/_sysforge b/completions/_sysforge index ff9193ef..d433018b 100644 --- a/completions/_sysforge +++ b/completions/_sysforge @@ -40,6 +40,7 @@ _sysforge() { search) _sysforge_search ;; setup) _sysforge_setup ;; uninstall) _sysforge_uninstall ;; + self-uninstall) _sysforge_self_uninstall ;; env) _sysforge_env ;; help) _sysforge_help ;; log) _sysforge_log ;; @@ -95,6 +96,7 @@ _sysforge_commands() { 'run:build toolchain, kernel or packages; or bootstrap' 'revert-to-stock:restore a source-built package to the repo version' 'uninstall:remove package(s); demote tracked ones from build state' + 'self-uninstall:revert sysforge'"'"'s builds to stock, then remove sysforge' ) _describe 'command' commands } @@ -435,7 +437,7 @@ _sysforge_packages_add() { '--packages[path to packages.toml]:file:_files' \ '--source[pin routing (metadata only)]:source:(repo aur local)' \ '--enable-build-from-source[build from source]' \ - '--no-cache[disable ccache/sccache (required for PGO)]' \ + '--no-cache[build this package with ccache and sccache off]' \ '--reason[free-form note attached to the entry]:text' \ '1: :_sysforge_pkg_names' } @@ -582,10 +584,19 @@ _sysforge_search() { _sysforge_uninstall() { _arguments \ + '--yes[remove without pacman'"'"'s confirmation prompt]' \ '--state-dir[override state directory]:dir:_directories' \ '*:installed package:_sysforge_installed_pkg_names' } +_sysforge_self_uninstall() { + _arguments \ + '--keep-data[remove only the package; keep config, state and profiles]' \ + '--dry-run[show the plan without changing anything]' \ + '--yes[do it without asking]' \ + '--state-dir[override state directory]:dir:_directories' +} + _sysforge_run() { local context state state_descr line typeset -A opt_args diff --git a/completions/sysforge.bash b/completions/sysforge.bash index 5fbc562f..ef997e2b 100644 --- a/completions/sysforge.bash +++ b/completions/sysforge.bash @@ -26,7 +26,7 @@ _sysforge() { # Ordered by usage tier to mirror `sysforge --help` (2.5.0-F1): # Everyday, then Inspect, then Maintain. - local commands="build update fetch search help doctor resolve env log state bench artifact setup config packages run revert-to-stock uninstall" + local commands="build update fetch search help doctor resolve env log state bench artifact setup config packages run revert-to-stock uninstall self-uninstall" # Locate the top-level verb (first non-flag arg after `sysforge`) and an # optional subverb (first non-flag arg after the verb). @@ -80,6 +80,7 @@ _sysforge() { search) _sysforge_search ;; setup) _sysforge_setup ;; uninstall) _sysforge_uninstall ;; + self-uninstall) _sysforge_self_uninstall ;; env) _sysforge_env ;; log) _sysforge_log ;; help) _sysforge_help ;; @@ -417,12 +418,17 @@ _sysforge_search() { _sysforge_uninstall() { if [[ $cur == -* ]]; then - COMPREPLY=( $(compgen -W "--state-dir" -- "$cur") ) + COMPREPLY=( $(compgen -W "--yes --state-dir" -- "$cur") ) else _sysforge_installed_pkg_names fi } +_sysforge_self_uninstall() { + _sysforge_flag_arg && return + [[ $cur == -* ]] && COMPREPLY=( $(compgen -W "--keep-data --dry-run --yes --state-dir" -- "$cur") ) +} + _sysforge_run() { _sysforge_flag_arg && return if [[ -z $subverb ]]; then diff --git a/docs/design/05-package-manifest.md b/docs/design/05-package-manifest.md index 73adb45e..bf87fc7c 100644 --- a/docs/design/05-package-manifest.md +++ b/docs/design/05-package-manifest.md @@ -49,7 +49,7 @@ The orthogonality of the roles means: Each entry overrides at most these fields (all optional except `name`): - `source` — `repo` (pacman) vs `aur`/`git`/`local`. A **routing hint**: it tells the bootstrap/build paths how to obtain the package, falling through to pacman / AUR RPC inference if omitted. It does **not** by itself put a package under steady-state tracking — tracking comes from sysforge having built the package (build_state), not from a manifest entry. So an entry with only `name` + `source` has no override effect at steady-state, and `sysforge packages add` rejects it (pair it with a behavior-changing field — `enable_build_from_source`, `cache`, `reason` — for the entry to override anything). When sysforge builds a package, it records the resolved `source` in build_state, so the registry is self-describing without re-inferring origin later. - `enable_build_from_source` *(bool)* — if `true`, this **repo** package is built from source (via `pkgctl repo clone` + makepkg with sysforge flag profiles) instead of installed as the binary via pacman. It also opts the package into `sysforge build` without a confirmation prompt (see §`build`). The boolean replaced the misleadingly-named `pkgbuild_patch` (which never patched anything — real PKGBUILD patching is gated by separate predicates); as of 3.0.0 the legacy `pkgbuild_patch` key is no longer honored on read (`config.normalize_package_entry` is now an identity function) — a manifest that still spells it no longer opts the package into a source build. `packages_cmd` still rewrites the key to the current name on the next `packages.toml` write (`add`/`remove`), so a file touched since the rename self-cleans; a never-rewritten file keeps the stale key inert. -- `cache` *(bool)* — `false` disables ccache/sccache for this package (required for PGO stages). +- `cache` *(bool)* — `false` builds this package with ccache and sccache off: `CCACHE_DISABLE=1` and an empty `RUSTC_WRAPPER` in its makepkg environment (`config.cache_disabled_for`, applied in `makepkg_wrapper.run`). ```toml [build] diff --git a/docs/design/07-pipeline-layer.md b/docs/design/07-pipeline-layer.md index ce9cdc55..9ed5f16a 100644 --- a/docs/design/07-pipeline-layer.md +++ b/docs/design/07-pipeline-layer.md @@ -156,7 +156,7 @@ The hardware stage (stage 2) needs no config — it auto-detects and writes `har - On `NotImplementedError`: prints `--start-from` guidance and exits - On `BootstrapRebootRequired`: saves state, prints reboot + resume instructions, exits 0 (clean stop, not failure) - On `RuntimeError`: saves state and exits with resume instructions -- `--dry-run`: logs what would run without calling `stage.run()` +- `--dry-run`: `run pipeline` lists the stages it would run without calling `stage.run()` (the bootstrap stages are not dry-run safe). A standalone `run --dry-run` calls `stage.run()` with `options.dry_run` set, after a `[dry-run] ` line through `ui()`: each stage stops short of its own changes and says what it would do, so the preview reads the real config (the kernel's "Kernel build plan", the toolchain's install set). The runner opens `PipelineState(read_only=True)`, so a stage's mid-run checkpoints never reach the file, the kernel and packages stages take no sentinel, and the toolchain stage's `resolve_all_pkgbuilds(dry_run=True)` reads only checked-out PKGBUILDs, raising `CheckoutsMissing` (the stage names what it would clone and ends the preview) instead of cloning, syncing or purging (`3.5.0-B141`; before it the runner returned before `stage.run()` and logged one INFO line, so the preview showed nothing). Guard against accidental state clobber: if a state file exists and neither `--resume` nor `--start-from` is passed, the runner exits with instructions rather than overwriting. Both flags are supported on `sysforge run pipeline`. diff --git a/docs/design/08-cli-verb-framework.md b/docs/design/08-cli-verb-framework.md index 5b0957ec..2518a3f6 100644 --- a/docs/design/08-cli-verb-framework.md +++ b/docs/design/08-cli-verb-framework.md @@ -108,7 +108,8 @@ spelling per meaning, across every verb: | `revert-to-stock` | resolve state dir + `plan_revert` over `BuildState` (pure, no mutation) | prompt/`--yes`/`--dry-run` gate, then per-target reinstall / atomic replace / derename-then-reinstall via `pacman` | (state written inline) | yes | | `bench toolchain` | null | resolve preconditions → workload → unprivileged stock fetch + extract → interleaved timing → report → `bench.toml` | null | no | | `search` | null | print installed → repo → AUR sections (fixed order, empty omitted) for one term, with in-line installed / stand-in (`-sysforge`, `-git` drop-in) markers | null | no | -| `uninstall` | resolve state dir + `plan_uninstall` over `BuildState` (pure) | `pacman -Rnsu`, then demote any tracked target via `state forget` + reconcile | (state written inline) | yes | +| `uninstall` | resolve state dir + `plan_uninstall` over `BuildState` (pure) | `--yes`/terminal gate, `pacman -Rnsu`, then demote any tracked target via `state forget` + reconcile | (state written inline) | yes | +| `self-uninstall` | `plan_self_uninstall` over `BuildState`, the sync DB, pacman.conf and the package dirs (reads only) | `--dry-run`/`--yes` gate; `apply_reverts` under its own sentinel; drop `IgnoreGroup` and the chroot repo block; one `sudo sh -c` removing the package, then the data and group | null | own scope, reverts only | | `run …` namespace | build `RunOptions` | delegate to `pipeline.run_pipeline` / `run_stage_standalone` | pipeline framework | (pipeline owns it) | | `artifact list` | null | `primitives/artifacts.unified_rows()` + PATH check + optional `scan()` for `--unmanaged` | null | no | | `artifact review` | `--all`, `--include-unknown` | interactively offer discovered candidates for adoption via `primitives/artifacts.iter_offerable()`; off-TTY lists candidates + adopt hint; `--all` bulk-adopts the same offer set without prompting | null | no | @@ -119,7 +120,7 @@ spelling per meaning, across every verb: ### `revert-to-stock` -Undoes a source-built or optimized package back to its official repo version. `pre_check` resolves the state dir and calls `plan_revert(bs, targets)` (pure — no mutation) to classify each target against `BuildState`, including a reverse lookup via the shared `install_reconcile.resolve_installed_name(bs, name)` helper (iterated in sorted key order so a stock base always resolves to the same pkgname deterministically) so naming the *stock* base of a renamed build (e.g. `mesa` when the tracked entry is `mesa-sysforge`) still resolves, as does a renamed split member's own stock name (`vulkan-radeon` → `vulkan-radeon-sysforge`, whose `origin_pkgbase` is `mesa`, `3.5.0-B62`). That helper is the single home for the reverse lookup — `uninstall` uses it too. Four actions, distinguished by `profile.is_optimized_build_mode(mode)` and — for optimized builds — `profile.rename_mode_for_build_mode(mode)`: +Undoes a source-built or optimized package back to its official repo version. `pre_check` resolves the state dir and calls `plan_revert(bs, targets)` (pure — no mutation) to classify each target against `BuildState`, including a reverse lookup via the shared `install_reconcile.resolve_installed_name(bs, name)` helper (iterated in sorted key order so a stock base always resolves to the same pkgname deterministically) so naming the *stock* base of a renamed build (e.g. `mesa` when the tracked entry is `mesa-sysforge`) still resolves, as does a renamed split member's own stock name (`vulkan-radeon` → `vulkan-radeon-sysforge`, whose `origin_pkgbase` is `mesa`, `3.5.0-B62`). That helper is the single home for the reverse lookup — `uninstall` uses it too. `plan_revert` and `apply_reverts(actionable, state_dir)` live in `verbs/shared.py`, shared with `self-uninstall`. Four actions, distinguished by `profile.is_optimized_build_mode(mode)` and — for optimized builds — `profile.rename_mode_for_build_mode(mode)`: - **`skip`** — untracked or already `build_mode = "pacman"`; nothing to do. - **`reinstall`** — plain `source_built` (or `bolt_llvm`, a BOLTed toolchain member, which also keeps its stock name), installed under the stock name: just reinstall the repo package (`pacman -S `). The name is the member itself, never its pkgbase (`pacman -S mesa` left a source-built `vulkan-radeon` in place); naming the pkgbase of a split set reinstalls every installed member, matching what `forget_packages` drops (`3.5.0-B62`). @@ -152,7 +153,13 @@ Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspe ### `uninstall` -Mutating lifecycle verb (`requires_sentinel = True`). `pre_check` resolves each name through the shared `resolve_installed_name` (so naming a stock base reaches its `-sysforge` build) and builds a pure `plan_uninstall` classifying each target's installed name + whether it's tracked. `execute` narrates the plan, removes via `pacman.uninstall_pkgs` (`pacman -Rnsu`, interactive — pacman prints its own confirmation; a `CalledProcessError` returns exit 1 and demotes nothing), clears the removed targets' recorded build failures through `install_reconcile.clear_removed_failures` (tracked or not, before the forget whose deleted records map a split member to its pkgbase; `3.5.0-B120`), then for any tracked target demotes it out of the build-state authority via `cmd_state_forget` (the same `state forget` code) followed by an `install_reconcile.consume_external_installs(bs)` pass. This is the exact demotion composition `revert-to-stock` uses — no parallel path — so `update` stops rebuilding an uninstalled package. +Mutating lifecycle verb (`requires_sentinel = True`). `pre_check` resolves each name through the shared `resolve_installed_name` (so naming a stock base reaches its `-sysforge` build) and builds a pure `plan_uninstall` classifying each target's installed name + whether it's tracked. `execute` narrates the plan, removes via `pacman.uninstall_pkgs` (`pacman -Rnsu`, interactive — pacman prints its own confirmation; a `CalledProcessError` returns exit 1 and demotes nothing), clears the removed targets' recorded build failures through `install_reconcile.clear_removed_failures` (tracked or not, before the forget whose deleted records map a split member to its pkgbase; `3.5.0-B120`), then for any tracked target demotes it out of the build-state authority via `cmd_state_forget` (the same `state forget` code) followed by an `install_reconcile.consume_external_installs(bs)` pass. This is the exact demotion composition `revert-to-stock` uses — no parallel path — so `update` stops rebuilding an uninstalled package. Without a terminal it refuses before pacman runs unless `--yes` is given, which passes `--noconfirm`: pacman's own `[Y/n]` read EOF and aborted after printing the transaction (`3.5.0-B143`). + +### `self-uninstall` + +Removes SysForge in one command (`3.5.0-F19`), replacing the manual list the "What SysForge touches" guide gave. `pre_check` builds a `SelfUninstallPlan` by reading only: every `build_state` entry not built by pacman goes through `plan_revert`, a split set planned once; a plan whose stock names (`pacman.get_sync_pkgnames`, one `pacman -Slq`) a repo carries is reverted, any other (an AUR or local build, a custom kernel, an FDO kernel whose plain kernel is itself local) is kept installed with the reason. It also records whether `/etc/pacman.conf` has `sf-build` in `IgnoreGroup` (`verbs.shared.has_ignore_group`), whether the sandbox chroot lists the local repo (`local_repo.has_chroot_section`), the package that owns `sysforge/__init__.py` (none from a checkout), which of `paths.PACKAGE_DIRS` and the XDG user dirs exist, and whether the `sysforge` group exists. `PACKAGE_DIRS` is fixed (`/etc/sysforge`, `/var/lib/sysforge`, `/var/cache/sysforge`), never `SYSFORGE_CONFIG_DIR`/`SYSFORGE_STATE_DIR`, which can name a developer's own tree. + +`execute` prints the plan, stops on `--dry-run`, and otherwise needs `--yes` or a terminal's `y` and an authorised `sudo`. Order is what makes a failure safe: (1) the reverts through `verbs.shared.apply_reverts`, the one home `revert-to-stock` also calls, inside the verb's own `sentinel_scope` (recovery: reinstall the stock packages) — the verb sets `requires_sentinel = False` and `wants_run_log = False` because the runner's scope and log live in the state dir it is about to delete; a failed revert stops the run with sysforge still installed; (2) `verbs.shared.remove_ignore_group` (`drop_ignore_group`, the inverse of setup's `_patch_conf_text`; the IgnoreGroup facts live in `verbs/shared.py`, since a verb module may not import a sibling) and `local_repo.remove_chroot_section`; (3) `removal_script`, one `sudo sh -c` running `pacman -Rns --noconfirm `, then `rm -rf` of the package dirs and `groupdel sysforge`. Step 3 is last and single because once pacman has deleted the package this process must import nothing more from it; after it only `shutil.rmtree` of the user dirs and a `print` run. The package's pacman hooks are gone before pacman runs post-transaction hooks, so none writes into the state dir being deleted. `--keep-data` uses `pacman -Rs` (edited config kept as `.pacsave`) and skips the data, group and user dirs. `tools/e2e/scenarios/99-self-uninstall.sh`, the tier's last scenario, runs the whole thing on a live system. ### `artifact list` diff --git a/docs/design/09-primitives-layer.md b/docs/design/09-primitives-layer.md index 26d4b623..3e5c00c2 100644 --- a/docs/design/09-primitives-layer.md +++ b/docs/design/09-primitives-layer.md @@ -478,7 +478,7 @@ Log tag: `[PROV]`. The `suggest_*` / `files_db_present` queries are pure read-on Cross-cutting failure scenario handler. Imported by `makepkg_wrapper` and `dep_analysis` to avoid circular imports. -`handle_failure(scenario, message, config, fallback=None)` dispatches to `abort`, `error`, `warn_and_fallback`, or `fallback` based on `[failure_handling]` config. `profile_missing` and `tempfile_write_failed` always abort regardless of config. +`handle_failure(scenario, message, config, fallback=None)` dispatches to `abort`, `error`, `warn_and_fallback`, or `fallback` based on `sysforge.toml [failure_handling]`, validated per key by `resolve_failure_handling` against `SCENARIOS` (the one home for every key's values and default). `tempfile_write_failed` always aborts regardless of config, as does a scenario not in `SCENARIOS`. ### `run.py` diff --git a/docs/design/10-flag-profile-system.md b/docs/design/10-flag-profile-system.md index 82a73775..aa34a921 100644 --- a/docs/design/10-flag-profile-system.md +++ b/docs/design/10-flag-profile-system.md @@ -10,11 +10,11 @@ toolchain = "gcc" # global package-compiler default (see "Toolchain field [profiles.bare] # Fallback profile, no flags +# No profile names a compiler or linker: both come from `toolchain` (see +# "Toolchain field"). Override individual CC/CXX/AR/… keys to win over it. + [profiles.standard] extends = "bare" -# Compiler comes from the `toolchain` field (defaults to "gcc" via [defaults]). -# Override individual CC/CXX/AR/… keys to win over the bundle, or set -# `toolchain = "llvm"` here / in a rule to switch the whole bundle. CFLAGS = "-march=native -O2 -pipe" CXXFLAGS = "$CFLAGS" LDFLAGS = "-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" diff --git a/docs/design/11-makepkg-wrapper.md b/docs/design/11-makepkg-wrapper.md index a009615a..4879570a 100644 --- a/docs/design/11-makepkg-wrapper.md +++ b/docs/design/11-makepkg-wrapper.md @@ -53,23 +53,18 @@ Preflight refuses rather than downgrades — missing `devtools`, a missing `/src`, and `makepkg_env._logdest_tail(pkgbuild_path)` reads the tail of the newest `$LOGDEST/-*.log` (makepkg's captured stdout when `OPTIONS+=log`). Both `BUILDDIR` and `LOGDEST` are resolved through `pacman.get_builddir()` / `pacman.get_logdest()` — env first, then the layered system `makepkg.conf` — so a user who configures these only in `/etc/makepkg.conf` still gets a diagnosis. Don't read `os.environ["BUILDDIR"]` or assume `~/builds` directly here. -Configuration extends `[failure_handling]` with four scenario keys: +Auto-repair has four keys of its own in the same table: ```toml -vendored_deps_missing = "auto_repair" +vendored_deps_missing = "auto_repair" # auto_repair | auto_repair_with_warning | abort pgp_key_missing = "auto_repair" -srcinfo_drift = "auto_repair_with_warning" -checksum_mismatch = "prompt_user" +srcinfo_drift = "auto_repair_with_warning" # … | warn_and_fallback (preflight only) +checksum_mismatch = "prompt_user" # prompt_user | abort, nothing else ``` -Three new behaviour values join the existing `abort` / `warn_and_fallback` / `fallback` / `error` set: `auto_repair` (repair silently, retry, log at info), `auto_repair_with_warning` (repair, retry, log at `[WARN]`), `prompt_user` (surface the diff and require explicit consent before repairing). In **batch mode** (`batch=true` on the resolved profile) the `prompt_user` behaviour is short-circuited to `aborted` — auto-repair never runs unattended for security-sensitive scenarios. This is the load-bearing rule for `checksum_mismatch`. +`auto_repair` repairs, retries and logs at info; `auto_repair_with_warning` logs the same at `[WARN]`; `prompt_user` surfaces the failing files and requires explicit consent before repairing. `checksum_mismatch` accepts only `prompt_user` or `abort`, so neither an edit nor a typo can select an unattended `updpkgsums`, which would mask a tampered source. In **batch mode** (`batch=true` on the resolved profile) `prompt_user` is short-circuited to `aborted`, so the repair never runs unattended. `auto_repair._normalise_behaviour` maps anything unexpected that still reaches it to `abort`, never to a repair. `.SRCINFO` drift is the one scenario that is **not** in `REGISTRY` — it has a different lifecycle. `_run_build` calls `auto_repair.preflight_srcinfo(pkgbuild_dir, behaviour)` before the build starts, regenerating `.SRCINFO` (with a `[WARN]` by default) when `makepkg --printsrcinfo` differs from the committed file. The other three scenarios fire on retry from a captured failure. diff --git a/docs/design/14-hardware-detection.md b/docs/design/14-hardware-detection.md index c912909b..861015f8 100644 --- a/docs/design/14-hardware-detection.md +++ b/docs/design/14-hardware-detection.md @@ -1,6 +1,6 @@ ## Hardware Detection -Pipeline stage 3. Probes the running system via `/proc/cpuinfo` and `lspci`, emits `hardware_profile.toml` to `state_dir`. The file feeds kconfig automation (kernel stage) and is shown in the reconfigure config review. +Pipeline stage 2. Probes the running system via `/proc/cpuinfo` and `lspci`, emits `hardware_profile.toml` to `state_dir`. The file feeds kconfig automation (kernel stage) and is shown in the reconfigure config review. **Detections and kconfig output:** diff --git a/docs/design/17-release-plan.md b/docs/design/17-release-plan.md index 0bf74fd7..bffd4e0f 100644 --- a/docs/design/17-release-plan.md +++ b/docs/design/17-release-plan.md @@ -43,11 +43,14 @@ codes, `pacman -Qi`, the state files and leftover sentinels, with prompts answer pseudo-terminal. It covers what the unit suite stubs: root versus the user, group membership that applies only at the next login, the state one command leaves for the next. It builds sysforge from the checkout itself, so it needs no `vm-pkg-stable`. `pre-release` runs it; the release preflight -does not (`_RELEASE_GATE_EXEMPT`: minutes long, podman, and one networked scenario, so not +does not (`_RELEASE_GATE_EXEMPT`: minutes long, podman, and four networked scenarios, so not deterministic). CI runs the same scenarios in an `e2e` job of its own, directly in the job's container (`make e2e E2E_HERE=1`), listed in the Makefile's `CI_JOBS` beside `GATES`. A scenario marked `xfail: ` fails until that item is fixed and reports `XFAIL`; once it passes it fails -the run, so the marker comes off with the fix. +the run, so the marker comes off with the fix. Every command in `tests/data/surface.json` is run by a scenario or +listed, with its reason, in `tools/e2e/uncovered.toml` (`3.5.0-DEV23`); `tools/e2e_coverage.py` +does the comparison, `tests/test_e2e_coverage.py` fails the suite on a command in neither set or +a listing a scenario has made redundant, and `make e2e-coverage` prints both. **CI** (`.github/workflows/ci.yml`, `3.4.0-DEV1`) runs every `pre-release` gate on each pull request (a newer push to it cancels the superseded run) and on each push to `main`, in a fresh `archlinux:base-devel` container, as an unprivileged user, with only the diff --git a/docs/design/18-reconverge.md b/docs/design/18-reconverge.md index 89916957..860f4706 100644 --- a/docs/design/18-reconverge.md +++ b/docs/design/18-reconverge.md @@ -12,7 +12,7 @@ **Config defaults.** All three drift-rebuild flags (`--rebuild-on-drift`, `--rebuild-on-toolchain-drift`, `--rebuild-on-flag-drift`) fall back to a sysforge.toml `[update]` default (`rebuild_on_drift`, `rebuild_on_toolchain_drift`, `rebuild_on_flag_drift` respectively) via `_resolve_drift_axes`, which resolves each axis through `config.resolve_flag_default`. The CLI flag always wins when explicitly passed; the config value only supplies the default when it isn't. -**PGO toolchain packages** (`build_mode = "pgo_llvm_toolchain"`) are handled specially during update. `makepkg_wrapper.run()` reads `toolchain.toml → pgo_store`, checks for a saved `clang.profdata` and its `clang.profdata.version` sidecar, and compares the sidecar's LLVM major version against the PKGBUILD's `pkgver` major. If they match, `-fprofile-use=` is injected and the build proceeds as a PGO-optimised build. If profdata is absent or version-mismatched (e.g. after a major LLVM bump), the user is prompted: **[p]lain build or [s]kip (default: skip)**. In non-interactive mode the build is skipped automatically. Skipped packages are counted separately in the update summary and do not count as failures. To rebuild profdata after a major version bump, run `sysforge run toolchain`. The toolchain stage itself also reuses compatible profdata — see the **Profdata reuse** section under stage 6. +**PGO toolchain packages** (`build_mode = "pgo_llvm_toolchain"`) are handled specially during update. `makepkg_wrapper.run()` reads `toolchain.toml → pgo_store`, checks for a saved `clang.profdata` and its `clang.profdata.version` sidecar, and compares the sidecar's LLVM major version against the PKGBUILD's `pkgver` major. If they match, `-fprofile-use=` is injected and the build proceeds as a PGO-optimised build. If profdata is absent or version-mismatched (e.g. after a major LLVM bump), the user is prompted: **[p]lain build or [s]kip (default: skip)**. In non-interactive mode the build is skipped automatically. Skipped packages are counted separately in the update summary and do not count as failures. To rebuild profdata after a major version bump, run `sysforge run toolchain`. The toolchain stage itself also reuses compatible profdata — see the **Profdata reuse** section under stage 5. **Stale-profraw post-build check.** After every non-PGO-managed build, `makepkg_wrapper.run()` globs `pgo_store` for `*.profraw` files. Any file with `mtime >= build_start - 1s` is treated as **fresh** — it was written by the build just completed, which means an instrumented LLVM is still installed on the system and the build was leaking profile data. The wrapper fatals, telling the user to reinstall `llvm`/`llvm-libs` or run `sysforge run toolchain`. Files strictly older than `build_start` are **orphans** left behind by a prior failed or partial toolchain run whose instrumented binaries the user has since cleaned up; these are unlinked in place and an info line is logged. The split makes the safety net self-healing: once the system is clean, the next build purges the residue automatically instead of requiring manual cleanup of `pgo_store`. **Neither verdict applies while a toolchain PGO run is in progress** (`3.3.0-B8`): if `build_lock.is_held(makepkg_pgo.resolve_pgo_lock_path(tcfg))` reports the `sysforge-pgo.lock` held, the store is that run's live Pass-3 training data, so a fresh file is not a leak and an "orphan" is data its merge monitor has not read yet. The check logs that it is deferring and touches nothing. Before this, a `sysforge build` started alongside `run toolchain` either fatally aborted on the training data or deleted whatever the monitor had not yet merged. `is_held` is a read-only probe: it never creates the lock file or overwrites the holder's PID. diff --git a/docs/guides/commands.md b/docs/guides/commands.md index d8165811..ceace11c 100644 --- a/docs/guides/commands.md +++ b/docs/guides/commands.md @@ -34,6 +34,7 @@ sysforge resolve PKG # show which profile applies to PKG, and why sysforge search cosmic # search installed, repo, and AUR for a term sysforge uninstall mesa # remove a package (demotes it out of build state) sysforge revert-to-stock mesa # undo a source-built package back to the repo version +sysforge self-uninstall # revert everything to stock, then remove SysForge itself ``` ## Manifest and build state diff --git a/docs/guides/what-sysforge-touches.md b/docs/guides/what-sysforge-touches.md index f78282bf..23f196cd 100644 --- a/docs/guides/what-sysforge-touches.md +++ b/docs/guides/what-sysforge-touches.md @@ -57,9 +57,23 @@ partitions disks and installs a whole system. For one package, `sysforge revert-to-stock ` reinstalls the repo build, and `sysforge state forget ` stops maintaining it but keeps your build installed. -To remove SysForge entirely, revert the packages you want back to stock, then -`sudo pacman -Rns sysforge`. Pacman removes the hooks and package files and keeps any config you -edited as `/etc/sysforge/*.pacsave`. Delete `/var/lib/sysforge`, `/var/cache/sysforge` and the -`sysforge` directories under `~/.cache`, `~/.local/share` and `~/.config` yourself, remove -`sf-build` from `IgnoreGroup` in `/etc/pacman.conf` if `setup` added it, and -`sudo groupdel sysforge`. Packages it built stay installed and keep working. +To remove SysForge entirely, run: + +```bash +sysforge self-uninstall --dry-run # see what it will do +sysforge self-uninstall +``` + +It puts back the repo build of every package SysForge built that a repo carries. A build no repo +carries, such as an AUR package or your own kernel, stays installed and keeps working, since +there is nothing to go back to. Then it takes `sf-build` out of `IgnoreGroup` in +`/etc/pacman.conf` and the build sandbox's local repo out of the chroot's `pacman.conf`, removes +the package, and deletes `/etc/sysforge`, `/var/lib/sysforge`, `/var/cache/sysforge` (with any +optimization profiles), the `sysforge` directories under `~/.cache`, `~/.local/share`, +`~/.config` and `~/.local/state`, and the `sysforge` group. If a revert fails, it stops there, +with SysForge still installed, so you can fix it and run the command again. + +`--keep-data` removes only the package: config you edited stays as `/etc/sysforge/*.pacsave`, +and the state, profiles, group and your own directories stay too. Scripts you adopted with +`sysforge artifact` stay where you deployed them either way; only SysForge's copies go. A +`sysforge run reconfigure` setting in `/etc/environment` is yours to remove. diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index be873f9e..97871729 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -52,6 +52,10 @@ https://keepachangelog.com/en/1.1.0/ --- +- **`3.5.0-B143` — `sysforge uninstall --yes`, for scripts and timers.** `uninstall` passed its confirmation to pacman and had no `--yes`, so without a terminal pacman printed the whole transaction, read nothing at its `[Y/n]` and gave up, and sysforge reported the raw command that failed. `--yes` now removes without asking, and without it and without a terminal `uninstall` refuses before doing anything and says to pass `--yes`, as `revert-to-stock` does. At a terminal nothing changes: pacman still asks. + +--- + - **`3.5.0-F1` — `bench toolchain` gets past an out-of-date sync database.** The bench downloads the stock `clang` and `llvm-libs` that your sync database names. When the repo rebuilt them after your last `pacman -Sy`, every mirror had already dropped those files, so each download failed with HTTP 404 and the bench stopped. It now notices that every mirror failed, downloads a fresh copy of the sync database into its own temporary directory, and compares against the build the repo carries now. It does this as your user with `fakeroot` (from `base-devel`), never with sudo, and your system's sync database is left alone, because syncing it without upgrading would leave the system partially upgraded. The report shows the stock version it used, so a newer stock build is flagged as a version difference, as before. If the refresh fails too, the error says so and suggests `pacman -Syu`. `--offline` never refreshes. The PGO guide describes the fallback. --- @@ -90,7 +94,6 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-F12` — a per-package compiler swap switches the whole toolchain.** When a build failed and you picked gcc (or clang) from the build-failure menu, sysforge retried and remembered the choice in `profiles.toml`'s `[package_compiler_overrides]`, but only the C and C++ compiler and the linker changed: a gcc swap on an LLVM profile kept `llvm-ar`, `llvm-nm`, `llvm-ranlib`, `llvm-strip` and lld. A row can now say `toolchain = "gcc"` or `"llvm"`, which switches the whole set the way a `[[rules]]` profile's `toolchain` does, and the menu writes that form and retries with the whole set. At its linker question, leaving the answer blank now means the chosen toolchain's own linker. Rows that name `cc`, `cxx` and `ld` directly keep working and remain the way to name a compiler outside the two sets. When `sysforge build --cc`, `--cxx` or `--ld` overrides a row for a package, `-vv` says so. The comment above the table in `profiles.toml` lists which setting wins; `sysforge config merge` picks it up. - --- - **`3.5.0-F13` — `sysforge log --verb` pages the log a verb left.** `log` could page only the unified log and per-package logs, although `build`, `update`, `doctor`, `fetch`, `setup`, `uninstall`, `revert-to-stock`, `config merge`, `state repair`, `state forget` and each `run ` leave their own `sysforge-.log`, which the end of the run named once. `sysforge log --verb update` (or `build`, `state-repair`, `run-toolchain` and so on; Tab lists them) pages it, and `--log-dir` finds a log written with the same `--log-dir`, for verb, unified and per-package logs alike. @@ -103,6 +106,10 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-F18` — find and fix retired or misspelt settings in your config files.** An upgrade never rewrites your config files, so a setting sysforge stopped reading stayed there, silently ignored, and a misspelt key was never reported. `sysforge config check` lists both, with the line (`--json` prints them as one JSON document for scripts): settings sysforge renamed or retired (such as `[git] pull_timeout`, the per-package `pkgbuild_patch`, `repo_mode = "profiled"` and `build_mode = "patched_pkgbuild"`) and keys it does not know. `sysforge config clean` gives retired settings their current name or value, keeping your comments, and with `--unknown` removes the unknown ones. `sysforge config reset FILE [KEY ...]` puts keys back to the shipped default, or the whole file, keeping the old one as `FILE.bak-