Retry failed packages, faster --devel source sync (3.5.0-F23, B144)

completions/_sysforge

@@ -161,6 +161,7 @@ _sysforge_build() {
     $_sysforge_help_spec \
     '(-m --makepkg)'{-m,--makepkg}'[extra makepkg flags, appended after profile flags]:flags' \
     '--interactive[run makepkg without --noconfirm, attached to the terminal]' \
+    '--failed[also build every package `state failed` lists]' \
     '--profile-conf[alternate profiles.toml]:file:_files' \
     '--toolchain[build with this toolchain as a whole]:toolchain:(gcc llvm)' \
     '(--reset-toolchain)--save-toolchain[also save --toolchain for these packages]' \

completions/sysforge.bash

@@ -206,7 +206,7 @@ _sysforge_installed_pkg_names() {
 
 _sysforge_build() {
     _sysforge_flag_arg && return
-    local flags="-m --makepkg --interactive --profile-conf \
+    local flags="-m --makepkg --interactive --failed --profile-conf \
         --toolchain --save-toolchain --reset-toolchain --cc --cxx --ld \
         --no-pkg-log --log-dir --persist-log --cache-report --abi-check \
         --no-update --cleansrc --cleansrc-force --no-llvm-preflight \

docs/design/09-primitives-layer.md

@@ -631,7 +631,7 @@ Flow per request:
 7. **Repo-track pin (`source = "repo"` only).** Gated by `config.resolve_repo_track` (`sysforge.toml [build] repo_track`, default `"stable"`) — a `"main"` track never pins and this step is a no-op. `_pin_repo_checkout(pkgbase, pkgbuild_dir)` looks up `pacman.get_repo_candidate_version(sync_db_name or pkgbase)` — the sync-DB read cross-checked against a refreshed side copy, so a sync DB that has not been refreshed since the last repo push cannot pin an already-superseded release (3.1.0-B3) — and, when found, runs `pkgctl_switch_version(pkgbuild_dir, version)` to check out the matching release tag (detached HEAD); no candidate in the sync DB is **not** an error — it warns and leaves the checkout on `main` (the package may live only in a custom repo not tracked by pacman). A `pkgctl_switch_version` failure returns `STATUS_FAILED` with the error attached. Two call sites: **clone-pin** — immediately after a fresh `pkgctl_checkout` in `_clone`, before the head is recorded in `SourceMetaCache`. **Fetch-repin** — in `_fetch`, on every subsequent sync of an already-pinned checkout: because a pinned checkout is a detached HEAD with no tracking branch, `git_fetch_and_compare` reports `no_tracking` for it; the repo-stable branch treats that as the steady state, refreshes tags with a plain `git fetch --tags origin` (`_fetch_repo_tags` — `git_fetch_and_compare` would otherwise bail before ever seeing new tags), then falls through to re-run `_pin_repo_checkout` so the checkout follows the sync DB's release forward. If the work-tree is dirty (`git_is_dirty(is_vcs=…)`) the re-pin is skipped and the result is `STATUS_DIVERGED` with `error="local edits present — not re-pinning"` — real operator work is never silently discarded by the pin logic. When the pin moves HEAD, the delta is reported as `STATUS_FETCHED` (`head_before`/`head_after` bracket the switch) rather than `STATUS_UP_TO_DATE`.
 
 Singletons:
-- `get_scheduler(*, state_dir=None, offline=False, cleansrc=False, cleansrc_force=False, force_devel=False, min_fetch_interval_ms=None, rate_limit_abort_s=None, fetch_timeout=None, clone_timeout=None)` — returns the process-wide scheduler, constructing it on first call. Subsequent calls with the same args are memoised; dedup keys: `(pkgbase)` — any given pkgbase is synced at most once per process. `cleansrc_force=True` implies `cleansrc=True` and propagates to `purge_src(force=True)` so `STATUS_PURGE_REFUSED` cannot occur — the operator has explicitly opted in to overwriting local work. `force_devel` only gates the forced-fetch behaviour for VCS pkgbases that *reach* the scheduler; the higher-level filter in `update.py:_sync_sources` is what keeps VCS pkgbases out of the request batch entirely when `--devel` is off (so `--cleansrc` never purges a `-git` tree the user hasn't opted in to rebuild).
+- `get_scheduler(*, state_dir=None, offline=False, cleansrc=False, cleansrc_force=False, min_fetch_interval_ms=None, rate_limit_abort_s=None, fetch_timeout=None, clone_timeout=None)` — returns the process-wide scheduler, constructing it on first call. Subsequent calls with the same args are memoised; dedup keys: `(pkgbase)` — any given pkgbase is synced at most once per process. `cleansrc_force=True` implies `cleansrc=True` and propagates to `purge_src(force=True)` so `STATUS_PURGE_REFUSED` cannot occur — the operator has explicitly opted in to overwriting local work. The scheduler knows nothing of `--devel` (`3.5.0-B144` dropped its `force_devel`, which only forced VCS packages past the RPC short-circuit): the filter in `update_sync._sync_sources` keeps VCS pkgbases out of the request batch entirely when `--devel` is off (so `--cleansrc` never purges a `-git` tree the user hasn't opted in to rebuild), and once in, a VCS package short-circuits like any other. The short-circuit also requires `_rpc_vouches`: `aur.aur_origin_pkgbase(pkgbuild_dir)` (strict, unlike the fail-open `is_aur_checkout` used for the AUR-disabled gate) must equal the entry's `PackageBase`, since the RPC is looked up by name and cannot vouch for a fork, another package's repo or a checkout with no origin. RPC fields are cached only when the entry vouches (`_record_rpc`; otherwise `SourceMetaCache.forget_rpc` clears them, as `update` preserves unset fields). Under the source freeze a VCS package that would short-circuit returns `STATUS_FROZEN`, since its version check needs upstream.
 - `reset_scheduler()` — test-only hook. Tests that need fresh state call this between runs.
 
 ### `build_state.py`
@@ -656,7 +656,7 @@ Read by `sysforge update` for version drift detection (every installed AUR packa
 
 On the write path, after a successful build `makepkg_wrapper.py` derives `pkgver`/`pkgrel`/`epoch` from the produced `.pkg.tar.*` filenames rather than the static PKGBUILD parse. The static parser intentionally leaves shell parameter-expansion forms (e.g. `${_ver/[a-z]/.${_ver//[0-9.]/}}`) untouched so it never produces a misleading partial substitution, but a built package's filename always carries the fully resolved version. Falling back to filenames prevents source-built entries from storing literal `$...` strings that would mismatch every subsequent vercmp and cause the package to be flagged for rebuild on every `sysforge update` run.
 
-**Build failures** live in a reserved top-level `[failures]` table (keyed by pkgbase), held apart from the per-package install mirror so `all_packages()` / `sync_with_installed()` stay a clean superset of `pacman -Q` (the `failures` key is popped into a private dict on load and re-serialized separately; a package literally named `failures` would collide but none exists in practice). Each entry carries `failed_at` (ISO 8601 UTC), `error` (the failure message tail — last ~6 lines / 600 chars), and optionally `pkgver`, `signature`, and `fix_cmd` (the latter two from `build_diag` postflight diagnosis when a known pattern matched). API: `record_failure(pkgbase, *, error, pkgver=None, signature=None, fix_cmd=None, failed_at=None)`, `clear_failure(pkgbase) -> bool`, `all_failures() -> {pkgbase: record}`. A successful `record()` calls `clear_failure(pkgbase)` so the failure list self-heals on the next good build; a removal clears it too (`3.5.0-B120`, `install_reconcile.clear_removed_failures`), as does `forget_packages` for a pkgbase it leaves with no tracked entry. `sysforge update`'s build fan-out records failures via `_record_build_failure` (opening a fresh `BuildState` so loop-time success writes aren't clobbered, and pulling `signature`/`fix_cmd` from the exception's `.diagnosis`, attached by `makepkg_wrapper`). Surfaced by `sysforge state failed`.
+**Build failures** live in a reserved top-level `[failures]` table (keyed by pkgbase), held apart from the per-package install mirror so `all_packages()` / `sync_with_installed()` stay a clean superset of `pacman -Q` (the `failures` key is popped into a private dict on load and re-serialized separately; a package literally named `failures` would collide but none exists in practice). Each entry carries `failed_at` (ISO 8601 UTC), `error` (the failure message tail — last ~6 lines / 600 chars), and optionally `pkgver`, `signature`, and `fix_cmd` (the latter two from `build_diag` postflight diagnosis when a known pattern matched), and `kind = "review"` for a package refused at the review gate rather than one that failed to build (`3.5.0-F23`: `build_core._record_review_refusal`, called where `build_and_install` adds to `outcome.review_refused`, both for a target's own change and for a refused dependency, so an unattended run's refusals outlive it). `sysforge build --failed` (`build_cmd._add_recorded_failures`) builds every entry of either kind; the doctor `state` axis words a review entry as a refusal with `build --failed` as its remediation. API: `record_failure(pkgbase, *, error, pkgver=None, signature=None, fix_cmd=None, failed_at=None, kind=None)`, `clear_failure(pkgbase) -> bool`, `all_failures() -> {pkgbase: record}`. A successful `record()` calls `clear_failure(pkgbase)` so the failure list self-heals on the next good build; a removal clears it too (`3.5.0-B120`, `install_reconcile.clear_removed_failures`), as does `forget_packages` for a pkgbase it leaves with no tracked entry. `sysforge update`'s build fan-out records failures via `_record_build_failure` (opening a fresh `BuildState` so loop-time success writes aren't clobbered, and pulling `signature`/`fix_cmd` from the exception's `.diagnosis`, attached by `makepkg_wrapper`). Surfaced by `sysforge state failed`.
 
 **Reverted builds** live in a second reserved table, `[reverted]`, keyed by artifact filename and held apart the same way (popped on load, serialized after `[failures]`, excluded from the lost-update diff). When a refused install reverts an entry, `revert_record(pkgname, prior, *, artifact=)` keeps the entry it undid there (with its `pkgname`), so the build's facts survive while its artifact waits in `PKGDEST`. `restore_reverted(artifact) -> pkgname | None` reinstates it once pacman installs that exact file. It is the only route by which a reused ("already built") artifact gets a record, because `build_and_install`'s `AlreadyBuilt` branch installs without calling `makepkg_wrapper` (3.3.0-B2). Only an entry recorded since `prior` is kept: a second refusal finds the entry already reverted (equal to `prior` once the carried-forward `_BUILD_FACTS` are ignored) and leaves the first stash alone. A later `record()` for the pkgname drops its stash (a newer build supersedes it; installing the stale artifact must not roll back to it), as does `sync_with_installed` pruning the package. Per-package fields serialize in `_ENTRY_KEYS` order, shared by both tables.
 

docs/design/verbs/state.md

@@ -8,7 +8,7 @@ Implements `sysforge state` — a small read/repair namespace for `build_state.t
 - **`state orphans`** — scans `PKGDEST` for `.pkg.tar*` artifacts whose pkgname is installed AND whose version is strictly older than the installed version (the **superseded** category). Read-only by default; `--delete` deletes after a y/N prompt (`--yes` skips). The detection primitive `pacman.detect_orphan_artifacts(pkgdest, installed)` returns `{"superseded": [...]}`.
   - **Files whose pkgname is *not* installed are intentionally NOT surfaced.** They could be a build kept on purpose (a kernel artifact whose source has local commits the user wants to keep available for later install, a test build, etc.) and we can't safely tell the difference. Per the load-bearing rule: if `--delete` wouldn't safely delete a file, don't list it. Users who want a broader view can use `paccache`/manual inspection.
   - Files whose `.PKGINFO` can't be read or whose filename doesn't parse are silently skipped — never deleted.
-- **`state failed`** — lists the `[failures]` table of `build_state.toml`: pkgbase, failure timestamp, diagnosis signature, the diagnosed fix command (when `build_diag` matched a known signature), and the truncated error tail. Read-only by default. `--names` (`3.1.0-F2`) prints only the pkgbases, one per line, with no header, no pager and no output at all when nothing failed, so `sysforge build $(sysforge state failed --names)` is the supported retry spelling; it is verb-local on purpose (this is the one listing whose output is already a set of package names), not a CLI-wide machine-output mode, and a distinct name because the global `--quiet` already means verbosity 0. `--clear PKGBASE` removes one entry and `--clear-all` removes all (both rewrite `build_state.toml`, so they take the sentinel like `state repair`). Failures are recorded by `sysforge update`'s build fan-out (`_record_build_failure`) and auto-clear on the next successful build of the same pkgbase (`BuildState.record` pops the matching failure) or once the package is removed (`3.5.0-B120`: `install_reconcile.clear_removed_failures`, see §`install_reconcile.py`; `state forget` also clears the failure of a pkgbase it leaves untracked), so the list stays a live view of *currently-broken* packages. The listing applies the removal pass in memory over the open hook window, so a `pacman -R` shows at once, before the next `update` clears the window for good.
+- **`state failed`** — lists the `[failures]` table of `build_state.toml` (build failures, and since `3.5.0-F23` review-gate refusals, marked `kind = "review"` and shown with `review` in the signature column): pkgbase, failure timestamp, diagnosis signature, the diagnosed fix command (when `build_diag` matched a known signature), and the truncated error tail. Read-only by default. `--names` (`3.1.0-F2`) prints only the pkgbases, one per line, with no header, no pager and no output at all when nothing failed, for scripts; the retry spelling is `sysforge build --failed` (`3.5.0-F23`), which reads the same table and is a clean no-op when it is empty; it is verb-local on purpose (this is the one listing whose output is already a set of package names), not a CLI-wide machine-output mode, and a distinct name because the global `--quiet` already means verbosity 0. `--clear PKGBASE` removes one entry and `--clear-all` removes all (both rewrite `build_state.toml`, so they take the sentinel like `state repair`). Failures are recorded by `sysforge update`'s build fan-out (`_record_build_failure`) and auto-clear on the next successful build of the same pkgbase (`BuildState.record` pops the matching failure) or once the package is removed (`3.5.0-B120`: `install_reconcile.clear_removed_failures`, see §`install_reconcile.py`; `state forget` also clears the failure of a pkgbase it leaves untracked), so the list stays a live view of *currently-broken* packages. The listing applies the removal pass in memory over the open hook window, so a `pacman -R` shows at once, before the next `update` clears the window for good.
 - **`state forget PKG…`** — deletes the named packages' `build_state.toml` records so `sysforge update` stops maintaining them (the "hand it back to pacman" escape hatch for the durable-by-default tracking model — see §Package Manifest). A name matching a pkgbase forgets every split-package member sharing it (`BuildState.delete` per pkgname). The *installed* artifact is untouched and still carries the `sf-build` pacman group, so `pacman -Syu` won't replace it; reverting fully to the stock repo binary is a separate `pacman -S <pkg>`. Rewrites `build_state.toml`, so it takes the sentinel like `state repair`. It also records the decision in the opt-out record (`forget_packages(opt_out=VIA_FORGET)`, `3.4.0-F4`): deleting the record alone did not stop `update`, whose adoption sources (every foreign package, a non-inert override, `repo_mode = "build_from_source"`) re-adopted the package on the next run. The record is keyed by the stock pkgbase and lists every member in both spellings (renamed and stock), so a split set opts out together and `update` matches a candidate by name without a `%BASE%` lookup. Only an explicit adoption clears it: `build <pkg>`, `update <pkg>` on a mutating run, and `packages add <pkg>`; passive sources never do. (Uninstalling a package already auto-stops tracking via `sync_with_installed`'s prune and writes no opt-out: a removed package is outside the installed set, and a later reinstall is a fresh decision. `forget` covers the keep-installed-but-unmanage case.)
 - **`state profiles`** (`3.2.0-F11`) — the one inspection/reclamation surface for the shared profile store (`/var/cache/sysforge`): one row per non-empty store — method, or method/target for the per-package methods (`pgo`, `autofdo`, `propeller`, `bolt`) — with size, age and the `3.2.0-B16` collected-version sidecar where present (`makepkg_pgo.list_profile_stores`). Read-only by default. `--delete <method>[/<target>]` (`makepkg_pgo.resolve_purge_target`: an unknown method or a target that is not one plain path component is refused, never guessed) deletes one store after a confirmation (default No) and, without a TTY, refuses unless `--yes` is given, because a collected profile needs a workload to regenerate; it takes the sentinel like `state failed --clear`. **Nothing purges implicitly**: a merged profile's existence is the durable PGO opt-in (`mesa_pgo.reuse_profdata`), so removing a package must not delete it — `revert-to-stock` and `uninstall` instead print one `ui()` line per non-empty store they leave behind (`verbs.shared.profile_store_names` reads the pre-rename `origin_pkgbase` before the record is forgotten; `state_cmd.orphaned_profile_lines` → `makepkg_pgo.stores_for_package`) with the `--purge` spelling that reclaims it.
 - **Pagination.** `state list`, `state orphans`, `state failed` and `state profiles` pipe their output through `$PAGER` (or `less -RF` / `more` as fallbacks) when stdout is a TTY. `--no-pager` disables. The pager wrapper `_maybe_pager(use_pager)` lives in `state_cmd.py` and degrades gracefully when no pager binary is available.