Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,14 @@ Shipped-file edits must pass `make check-shipped`; doc/design edits `make check-
vocabulary outside the closing "In sysforge" section, upstream sources cited, no numbers
measured on one machine (`tests/test_concept_pages.py` checks the closing sections, the upstream
links and the vocabulary); guides link them (and the glossary) rather than re-explaining.
- **`DOC` IDs are for user-facing docs only** (`docs/guides/`, `docs/concepts/`, README, man page) and file under
- **`DOC` IDs are for user-facing docs only** (`docs/guides/`, `docs/concepts/`, README, man page,
comments in shipped `etc/sysforge/*.toml`) and file under
`## Changed` (patch; lint-enforced). Internal docs (design, CLAUDE.md, ROADMAP) get no ID;
wrong `--help`/man output is a `B`, since it is CLI behaviour.
- **`DEV` IDs are for contributor-only work** (only `tests/`, `tools/`, `Makefile`, `.claude/`,
dev docs) and file their release-note entry under `## Changed` only, like `DOC` (lint-enforced,
`3.4.0-STD1`). Touching `sysforge/` or a shipped file makes it `B`/`F`.
`3.4.0-STD1`). Touching `sysforge/` or a shipped file makes it `B`/`F` (a comment-only edit to a
shipped config is `DOC`).
- **Completions stay in lockstep with the CLI** (`completions/_sysforge` + bash) in the same change.
- **A user-facing change regenerates `tests/data/surface.json`** (`make surface`) and names what
changed in the release notes; `make check-surface` (a required CI gate) enforces both.
Expand Down
63 changes: 34 additions & 29 deletions DESIGN.md

Large diffs are not rendered by default.

8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
preflight release-major release-minor release-patch release-resume \
vm-deps vm-image vm-boot vm-boot-gui vm-snapshot vm-loadvm vm-iso vm-monitor vm-console vm-savevm vm-snapshots vm-ssh vm-ssh-builder vm-stop vm-clean \
vm-pkg-stable vm-pkg-git vm-pkg-all vm-install-stable vm-install-git vm-smoke vm-test \
container-build container-smoke container-smoke-cachyos container-shell container-clean e2e \
container-build container-smoke container-smoke-cachyos container-shell container-clean e2e e2e-coverage \
ci-local

# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -658,6 +658,12 @@ container-clean: ## Remove the test container and image
e2e: ## Run the end-to-end scenarios in fresh Arch containers (SCENARIO=, needs podman)
E2E_COMMIT=$(E2E_COMMIT) bash tools/e2e/run.sh $(if $(E2E_HERE),--here) $(SCENARIO)

# Which commands a scenario runs and which tools/e2e/uncovered.toml excuses,
# with its reason (3.5.0-DEV23). tests/test_e2e_coverage.py fails the suite on
# a command in neither set, so this is the readable view of that check.
e2e-coverage: ## List the commands the end-to-end scenarios run, and the excused rest
uv run --no-sync python tools/e2e_coverage.py

# Replay the CI job (.github/workflows/ci.yml, 3.4.0-DEV1) against the working
# tree, uncommitted edits included, in the same base image. The tree is mounted
# read-only: tools/ci.sh copies it to an unprivileged user before running the
Expand Down
3 changes: 2 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ neighbour by a `---` rule, so a single item is readable on its own rather than
running into the next one. `docs/release-notes/` entries use the same two rules.

**`DOC` covers user-facing documentation only** — guides under `docs/guides/`, concept
pages under `docs/concepts/`, `README.md`, the man page — so doc work stays visible in release notes without
pages under `docs/concepts/`, `README.md`, the man page, and the comments in shipped config files
(`etc/sysforge/*.toml`) — so doc work stays visible in release notes without
being forced into `F` or `B`. A `DOC` entry always lands under `## Changed`: the
section drives the derived SemVer bump, and documentation changes no API (a
`DOC` under `## Added` would demand a minor release). `make check-standards`
Expand Down
13 changes: 12 additions & 1 deletion completions/_sysforge
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ _sysforge() {
search) _sysforge_search ;;
setup) _sysforge_setup ;;
uninstall) _sysforge_uninstall ;;
self-uninstall) _sysforge_self_uninstall ;;
env) _sysforge_env ;;
help) _sysforge_help ;;
log) _sysforge_log ;;
Expand Down Expand Up @@ -95,6 +96,7 @@ _sysforge_commands() {
'run:build toolchain, kernel or packages; or bootstrap'
'revert-to-stock:restore a source-built package to the repo version'
'uninstall:remove package(s); demote tracked ones from build state'
'self-uninstall:revert sysforge'"'"'s builds to stock, then remove sysforge'
)
_describe 'command' commands
}
Expand Down Expand Up @@ -435,7 +437,7 @@ _sysforge_packages_add() {
'--packages[path to packages.toml]:file:_files' \
'--source[pin routing (metadata only)]:source:(repo aur local)' \
'--enable-build-from-source[build from source]' \
'--no-cache[disable ccache/sccache (required for PGO)]' \
'--no-cache[build this package with ccache and sccache off]' \
'--reason[free-form note attached to the entry]:text' \
'1: :_sysforge_pkg_names'
}
Expand Down Expand Up @@ -582,10 +584,19 @@ _sysforge_search() {

_sysforge_uninstall() {
_arguments \
'--yes[remove without pacman'"'"'s confirmation prompt]' \
'--state-dir[override state directory]:dir:_directories' \
'*:installed package:_sysforge_installed_pkg_names'
}

_sysforge_self_uninstall() {
_arguments \
'--keep-data[remove only the package; keep config, state and profiles]' \
'--dry-run[show the plan without changing anything]' \
'--yes[do it without asking]' \
'--state-dir[override state directory]:dir:_directories'
}

_sysforge_run() {
local context state state_descr line
typeset -A opt_args
Expand Down
10 changes: 8 additions & 2 deletions completions/sysforge.bash
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ _sysforge() {

# Ordered by usage tier to mirror `sysforge --help` (2.5.0-F1):
# Everyday, then Inspect, then Maintain.
local commands="build update fetch search help doctor resolve env log state bench artifact setup config packages run revert-to-stock uninstall"
local commands="build update fetch search help doctor resolve env log state bench artifact setup config packages run revert-to-stock uninstall self-uninstall"

# Locate the top-level verb (first non-flag arg after `sysforge`) and an
# optional subverb (first non-flag arg after the verb).
Expand Down Expand Up @@ -80,6 +80,7 @@ _sysforge() {
search) _sysforge_search ;;
setup) _sysforge_setup ;;
uninstall) _sysforge_uninstall ;;
self-uninstall) _sysforge_self_uninstall ;;
env) _sysforge_env ;;
log) _sysforge_log ;;
help) _sysforge_help ;;
Expand Down Expand Up @@ -417,12 +418,17 @@ _sysforge_search() {

_sysforge_uninstall() {
if [[ $cur == -* ]]; then
COMPREPLY=( $(compgen -W "--state-dir" -- "$cur") )
COMPREPLY=( $(compgen -W "--yes --state-dir" -- "$cur") )
else
_sysforge_installed_pkg_names
fi
}

_sysforge_self_uninstall() {
_sysforge_flag_arg && return
[[ $cur == -* ]] && COMPREPLY=( $(compgen -W "--keep-data --dry-run --yes --state-dir" -- "$cur") )
}

_sysforge_run() {
_sysforge_flag_arg && return
if [[ -z $subverb ]]; then
Expand Down
2 changes: 1 addition & 1 deletion docs/design/05-package-manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The orthogonality of the roles means:
Each entry overrides at most these fields (all optional except `name`):
- `source` — `repo` (pacman) vs `aur`/`git`/`local`. A **routing hint**: it tells the bootstrap/build paths how to obtain the package, falling through to pacman / AUR RPC inference if omitted. It does **not** by itself put a package under steady-state tracking — tracking comes from sysforge having built the package (build_state), not from a manifest entry. So an entry with only `name` + `source` has no override effect at steady-state, and `sysforge packages add` rejects it (pair it with a behavior-changing field — `enable_build_from_source`, `cache`, `reason` — for the entry to override anything). When sysforge builds a package, it records the resolved `source` in build_state, so the registry is self-describing without re-inferring origin later.
- `enable_build_from_source` *(bool)* — if `true`, this **repo** package is built from source (via `pkgctl repo clone` + makepkg with sysforge flag profiles) instead of installed as the binary via pacman. It also opts the package into `sysforge build` without a confirmation prompt (see §`build`). The boolean replaced the misleadingly-named `pkgbuild_patch` (which never patched anything — real PKGBUILD patching is gated by separate predicates); as of 3.0.0 the legacy `pkgbuild_patch` key is no longer honored on read (`config.normalize_package_entry` is now an identity function) — a manifest that still spells it no longer opts the package into a source build. `packages_cmd` still rewrites the key to the current name on the next `packages.toml` write (`add`/`remove`), so a file touched since the rename self-cleans; a never-rewritten file keeps the stale key inert.
- `cache` *(bool)* — `false` disables ccache/sccache for this package (required for PGO stages).
- `cache` *(bool)* — `false` builds this package with ccache and sccache off: `CCACHE_DISABLE=1` and an empty `RUSTC_WRAPPER` in its makepkg environment (`config.cache_disabled_for`, applied in `makepkg_wrapper.run`).

```toml
[build]
Expand Down
2 changes: 1 addition & 1 deletion docs/design/07-pipeline-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ The hardware stage (stage 2) needs no config — it auto-detects and writes `har
- On `NotImplementedError`: prints `--start-from` guidance and exits
- On `BootstrapRebootRequired`: saves state, prints reboot + resume instructions, exits 0 (clean stop, not failure)
- On `RuntimeError`: saves state and exits with resume instructions
- `--dry-run`: logs what would run without calling `stage.run()`
- `--dry-run`: `run pipeline` lists the stages it would run without calling `stage.run()` (the bootstrap stages are not dry-run safe). A standalone `run <stage> --dry-run` calls `stage.run()` with `options.dry_run` set, after a `[dry-run] <stage>` line through `ui()`: each stage stops short of its own changes and says what it would do, so the preview reads the real config (the kernel's "Kernel build plan", the toolchain's install set). The runner opens `PipelineState(read_only=True)`, so a stage's mid-run checkpoints never reach the file, the kernel and packages stages take no sentinel, and the toolchain stage's `resolve_all_pkgbuilds(dry_run=True)` reads only checked-out PKGBUILDs, raising `CheckoutsMissing` (the stage names what it would clone and ends the preview) instead of cloning, syncing or purging (`3.5.0-B141`; before it the runner returned before `stage.run()` and logged one INFO line, so the preview showed nothing).

Guard against accidental state clobber: if a state file exists and neither `--resume` nor `--start-from` is passed, the runner exits with instructions rather than overwriting. Both flags are supported on `sysforge run pipeline`.

Expand Down
13 changes: 10 additions & 3 deletions docs/design/08-cli-verb-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,8 @@ spelling per meaning, across every verb:
| `revert-to-stock` | resolve state dir + `plan_revert` over `BuildState` (pure, no mutation) | prompt/`--yes`/`--dry-run` gate, then per-target reinstall / atomic replace / derename-then-reinstall via `pacman` | (state written inline) | yes |
| `bench toolchain` | null | resolve preconditions → workload → unprivileged stock fetch + extract → interleaved timing → report → `bench.toml` | null | no |
| `search` | null | print installed → repo → AUR sections (fixed order, empty omitted) for one term, with in-line installed / stand-in (`-sysforge`, `-git` drop-in) markers | null | no |
| `uninstall` | resolve state dir + `plan_uninstall` over `BuildState` (pure) | `pacman -Rnsu`, then demote any tracked target via `state forget` + reconcile | (state written inline) | yes |
| `uninstall` | resolve state dir + `plan_uninstall` over `BuildState` (pure) | `--yes`/terminal gate, `pacman -Rnsu`, then demote any tracked target via `state forget` + reconcile | (state written inline) | yes |
| `self-uninstall` | `plan_self_uninstall` over `BuildState`, the sync DB, pacman.conf and the package dirs (reads only) | `--dry-run`/`--yes` gate; `apply_reverts` under its own sentinel; drop `IgnoreGroup` and the chroot repo block; one `sudo sh -c` removing the package, then the data and group | null | own scope, reverts only |
| `run …` namespace | build `RunOptions` | delegate to `pipeline.run_pipeline` / `run_stage_standalone` | pipeline framework | (pipeline owns it) |
| `artifact list` | null | `primitives/artifacts.unified_rows()` + PATH check + optional `scan()` for `--unmanaged` | null | no |
| `artifact review` | `--all`, `--include-unknown` | interactively offer discovered candidates for adoption via `primitives/artifacts.iter_offerable()`; off-TTY lists candidates + adopt hint; `--all` bulk-adopts the same offer set without prompting | null | no |
Expand All @@ -119,7 +120,7 @@ spelling per meaning, across every verb:

### `revert-to-stock`

Undoes a source-built or optimized package back to its official repo version. `pre_check` resolves the state dir and calls `plan_revert(bs, targets)` (pure — no mutation) to classify each target against `BuildState`, including a reverse lookup via the shared `install_reconcile.resolve_installed_name(bs, name)` helper (iterated in sorted key order so a stock base always resolves to the same pkgname deterministically) so naming the *stock* base of a renamed build (e.g. `mesa` when the tracked entry is `mesa-sysforge`) still resolves, as does a renamed split member's own stock name (`vulkan-radeon` → `vulkan-radeon-sysforge`, whose `origin_pkgbase` is `mesa`, `3.5.0-B62`). That helper is the single home for the reverse lookup — `uninstall` uses it too. Four actions, distinguished by `profile.is_optimized_build_mode(mode)` and — for optimized builds — `profile.rename_mode_for_build_mode(mode)`:
Undoes a source-built or optimized package back to its official repo version. `pre_check` resolves the state dir and calls `plan_revert(bs, targets)` (pure — no mutation) to classify each target against `BuildState`, including a reverse lookup via the shared `install_reconcile.resolve_installed_name(bs, name)` helper (iterated in sorted key order so a stock base always resolves to the same pkgname deterministically) so naming the *stock* base of a renamed build (e.g. `mesa` when the tracked entry is `mesa-sysforge`) still resolves, as does a renamed split member's own stock name (`vulkan-radeon` → `vulkan-radeon-sysforge`, whose `origin_pkgbase` is `mesa`, `3.5.0-B62`). That helper is the single home for the reverse lookup — `uninstall` uses it too. `plan_revert` and `apply_reverts(actionable, state_dir)` live in `verbs/shared.py`, shared with `self-uninstall`. Four actions, distinguished by `profile.is_optimized_build_mode(mode)` and — for optimized builds — `profile.rename_mode_for_build_mode(mode)`:

- **`skip`** — untracked or already `build_mode = "pacman"`; nothing to do.
- **`reinstall`** — plain `source_built` (or `bolt_llvm`, a BOLTed toolchain member, which also keeps its stock name), installed under the stock name: just reinstall the repo package (`pacman -S <name>`). The name is the member itself, never its pkgbase (`pacman -S mesa` left a source-built `vulkan-radeon` in place); naming the pkgbase of a split set reinstalls every installed member, matching what `forget_packages` drops (`3.5.0-B62`).
Expand Down Expand Up @@ -152,7 +153,13 @@ Read-only (`3.4.0-F6`, `BenchToolchainVerb` under the `bench` `VerbGroup`, Inspe

### `uninstall`

Mutating lifecycle verb (`requires_sentinel = True`). `pre_check` resolves each name through the shared `resolve_installed_name` (so naming a stock base reaches its `-sysforge` build) and builds a pure `plan_uninstall` classifying each target's installed name + whether it's tracked. `execute` narrates the plan, removes via `pacman.uninstall_pkgs` (`pacman -Rnsu`, interactive — pacman prints its own confirmation; a `CalledProcessError` returns exit 1 and demotes nothing), clears the removed targets' recorded build failures through `install_reconcile.clear_removed_failures` (tracked or not, before the forget whose deleted records map a split member to its pkgbase; `3.5.0-B120`), then for any tracked target demotes it out of the build-state authority via `cmd_state_forget` (the same `state forget` code) followed by an `install_reconcile.consume_external_installs(bs)` pass. This is the exact demotion composition `revert-to-stock` uses — no parallel path — so `update` stops rebuilding an uninstalled package.
Mutating lifecycle verb (`requires_sentinel = True`). `pre_check` resolves each name through the shared `resolve_installed_name` (so naming a stock base reaches its `-sysforge` build) and builds a pure `plan_uninstall` classifying each target's installed name + whether it's tracked. `execute` narrates the plan, removes via `pacman.uninstall_pkgs` (`pacman -Rnsu`, interactive — pacman prints its own confirmation; a `CalledProcessError` returns exit 1 and demotes nothing), clears the removed targets' recorded build failures through `install_reconcile.clear_removed_failures` (tracked or not, before the forget whose deleted records map a split member to its pkgbase; `3.5.0-B120`), then for any tracked target demotes it out of the build-state authority via `cmd_state_forget` (the same `state forget` code) followed by an `install_reconcile.consume_external_installs(bs)` pass. This is the exact demotion composition `revert-to-stock` uses — no parallel path — so `update` stops rebuilding an uninstalled package. Without a terminal it refuses before pacman runs unless `--yes` is given, which passes `--noconfirm`: pacman's own `[Y/n]` read EOF and aborted after printing the transaction (`3.5.0-B143`).

### `self-uninstall`

Removes SysForge in one command (`3.5.0-F19`), replacing the manual list the "What SysForge touches" guide gave. `pre_check` builds a `SelfUninstallPlan` by reading only: every `build_state` entry not built by pacman goes through `plan_revert`, a split set planned once; a plan whose stock names (`pacman.get_sync_pkgnames`, one `pacman -Slq`) a repo carries is reverted, any other (an AUR or local build, a custom kernel, an FDO kernel whose plain kernel is itself local) is kept installed with the reason. It also records whether `/etc/pacman.conf` has `sf-build` in `IgnoreGroup` (`verbs.shared.has_ignore_group`), whether the sandbox chroot lists the local repo (`local_repo.has_chroot_section`), the package that owns `sysforge/__init__.py` (none from a checkout), which of `paths.PACKAGE_DIRS` and the XDG user dirs exist, and whether the `sysforge` group exists. `PACKAGE_DIRS` is fixed (`/etc/sysforge`, `/var/lib/sysforge`, `/var/cache/sysforge`), never `SYSFORGE_CONFIG_DIR`/`SYSFORGE_STATE_DIR`, which can name a developer's own tree.

`execute` prints the plan, stops on `--dry-run`, and otherwise needs `--yes` or a terminal's `y` and an authorised `sudo`. Order is what makes a failure safe: (1) the reverts through `verbs.shared.apply_reverts`, the one home `revert-to-stock` also calls, inside the verb's own `sentinel_scope` (recovery: reinstall the stock packages) — the verb sets `requires_sentinel = False` and `wants_run_log = False` because the runner's scope and log live in the state dir it is about to delete; a failed revert stops the run with sysforge still installed; (2) `verbs.shared.remove_ignore_group` (`drop_ignore_group`, the inverse of setup's `_patch_conf_text`; the IgnoreGroup facts live in `verbs/shared.py`, since a verb module may not import a sibling) and `local_repo.remove_chroot_section`; (3) `removal_script`, one `sudo sh -c` running `pacman -Rns --noconfirm <pkg>`, then `rm -rf` of the package dirs and `groupdel sysforge`. Step 3 is last and single because once pacman has deleted the package this process must import nothing more from it; after it only `shutil.rmtree` of the user dirs and a `print` run. The package's pacman hooks are gone before pacman runs post-transaction hooks, so none writes into the state dir being deleted. `--keep-data` uses `pacman -Rs` (edited config kept as `.pacsave`) and skips the data, group and user dirs. `tools/e2e/scenarios/99-self-uninstall.sh`, the tier's last scenario, runs the whole thing on a live system.

### `artifact list`

Expand Down
Loading
Loading