From 48d5e7974f94667f9717becab9b40bf7347a334a Mon Sep 17 00:00:00 2001 From: Keith Raghubar Date: Wed, 21 Oct 2026 19:12:35 -0400 Subject: [PATCH 01/16] test(shipped): check completions command by command (3.5.0-DEV14, B127) New check_shipped group completion_commands. The old completions group only walked top-level commands, so after F15 left group parents with no flags, no subcommand flag was checked, and presence meant "anywhere in the file". Now each command is checked where it is completed (its zsh function, minus description text; its bash case arm): every long flag offered, nothing it lacks offered, every argparse choice value offered, and code-derived lists (desktop catalog, PROFILE_METHODS) equal to their constants. B127: its first run found bash offering no values for `run toolchain --rebuild-soname-consumers`; fixed. Co-Authored-By: Claude Opus 5.5 --- DESIGN.md | 1 + completions/sysforge.bash | 4 + docs/design/17-release-plan.md | 1 + docs/release-notes/unreleased.md | 8 ++ tests/test_check_shipped.py | 51 +++++++++ tools/check_shipped.py | 172 +++++++++++++++++++++++++++++++ 6 files changed, 237 insertions(+) diff --git a/DESIGN.md b/DESIGN.md index 54c12fb1..e314ad84 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -4129,6 +4129,7 @@ The version markers in `README.md` and `DESIGN.md` wrap the single live version - **`pkgbuild_parity`** — `PKGBUILD` and `PKGBUILD-git` parse to the same dict (via `pkgbuild_meta.parse_pkgbuild`) except for a tightly-scoped allowlist of keys that are *supposed* to differ (`pkgname`, `pkgver`, `pkgrel`, `pkgdesc`, `source`, `sha256sums`, `conflicts`, `provides`, `validpgpkeys` — stable-only, since the VCS package has no release `.asc`). `depends` / `makedepends` / `optdepends` / `backup` arrays must be byte-identical, and both `package()` bodies must install the same set of targets (`_parse_pkgbuild_install_targets`), which is how a `pkgbuild`-group rule checked on the stable file, such as the license install, reaches `PKGBUILD-git`. - **`hooks`** — every `etc/pacman.d/hooks/sysforge-*.hook` `Exec` line must invoke `tools/pacman-hook-helper.sh` and pass a subcommand the helper documents (`kernel`, `toolchain`, `buildstate`). - **`completions`** — every verb and every long-flag in the argparse parser tree (reached via `sysforge.verbs.registry.build_parser`) must appear in both `completions/_sysforge` and `completions/sysforge.bash`; stale top-level verb entries in the zsh case statement (function-suffix matches case-word but parser doesn't know the verb) are an error. Mirrors the `completions-cli-parity` subagent's audit; this is the mechanical layer that runs every release. +- **`completion_commands`** (`3.5.0-DEV14`) — `completions` only walked top-level commands, so after `3.5.0-F15` left group parents with no flags, no subcommand flag was checked at all, and a flag counted as present if it appeared anywhere in the file. This group checks each command where it is completed: its zsh function (`_sysforge_state_orphans`, description text excluded) and its bash `case` arm. Every long flag must be offered there, nothing the command lacks may be, every argparse `choices` value must be offered (bash keeps values in `_sysforge_flag_arg` keyed by flag, or in the command's own body), and lists copied from code constants (`_MIRRORED_LISTS`: the desktop catalog, `PROFILE_METHODS`) must equal them. Its first run found bash offering no values for `run toolchain --rebuild-soname-consumers` (`3.5.0-B127`). - **`completion_widths`** — every description in `completions/_sysforge` must fit an 80-column listing (`COMPLETION_TARGET_COLUMNS`). zsh renders a match as ` -- `, padding the match column to the longest match in that generator call; once a row exceeds the terminal width zsh abandons the inline two-column table and emits every description as its own list entry, so the whole block renders with names and descriptions detached (the 3.0.0-B2 symptom). The budget is therefore per-call and shared: `COLUMNS - (longest match in the block) - 4`, computed separately for each `_arguments` spec list and each `_describe` array (`_completion_blocks`). One over-wide row degrades every row beside it, and adding a long *option name* to a verb tightens the budget for every *description* in it — `update` is the tightest block in the file at 48 characters, bounded by `--rebuild-on-toolchain-drift`. `completions/sysforge.bash` is not checked: it emits bare `compgen -W` word lists with no descriptions and cannot exhibit this failure. - **`versions`** — `pyproject.toml` `[project] version` must equal `PKGBUILD` `pkgver=`, the leading `X.Y.Z` of `PKGBUILD-git` `pkgver=`, and every `vX.Y.Z` marker in `README.md` and `DESIGN.md` (literal `vX.Y.Z` placeholder strings in prose are filtered out by the `\d+\.\d+\.\d+` constraint). - **`manpage`** — regenerates `man/sysforge.1` via the scdoc-hybrid pipeline `make man` uses (`tools/gen_options.py` splices the argparse-derived COMMANDS sections into `man/sysforge.1.scd.in`, then `scdoc` renders) into temp files and diffs against the committed page; any difference is an error, with the fix `make man && git add man/sysforge.1`. Both sides are passed through `_normalize_roff` before diffing, so scdoc-version-specific artifacts aren't findings: the `.TH … "DATE"` header (daily date change), the `.\" Generated by scdoc ` banner, and hyphen escaping (`\-` vs `-`, functionally inert in roff). This keeps the gate coupled to the CLI surface rather than to the local scdoc build. Skipped with a `warn` if `scdoc` isn't on PATH. See the Man Pages section. diff --git a/completions/sysforge.bash b/completions/sysforge.bash index 47120133..1c512c82 100644 --- a/completions/sysforge.bash +++ b/completions/sysforge.bash @@ -138,6 +138,10 @@ _sysforge_flag_arg() { COMPREPLY=( $(compgen -W "systemd-boot grub none" -- "$cur") ) return 0 ;; + --rebuild-soname-consumers) + COMPREPLY=( $(compgen -W "prompt auto off" -- "$cur") ) + return 0 + ;; --base-config) # A keyword or a path to a .config file. _filedir diff --git a/docs/design/17-release-plan.md b/docs/design/17-release-plan.md index 3c40d5d5..a486c59f 100644 --- a/docs/design/17-release-plan.md +++ b/docs/design/17-release-plan.md @@ -86,6 +86,7 @@ The version markers in `README.md` and `DESIGN.md` wrap the single live version - **`pkgbuild_parity`** — `PKGBUILD` and `PKGBUILD-git` parse to the same dict (via `pkgbuild_meta.parse_pkgbuild`) except for a tightly-scoped allowlist of keys that are *supposed* to differ (`pkgname`, `pkgver`, `pkgrel`, `pkgdesc`, `source`, `sha256sums`, `conflicts`, `provides`, `validpgpkeys` — stable-only, since the VCS package has no release `.asc`). `depends` / `makedepends` / `optdepends` / `backup` arrays must be byte-identical, and both `package()` bodies must install the same set of targets (`_parse_pkgbuild_install_targets`), which is how a `pkgbuild`-group rule checked on the stable file, such as the license install, reaches `PKGBUILD-git`. - **`hooks`** — every `etc/pacman.d/hooks/sysforge-*.hook` `Exec` line must invoke `tools/pacman-hook-helper.sh` and pass a subcommand the helper documents (`kernel`, `toolchain`, `buildstate`). - **`completions`** — every verb and every long-flag in the argparse parser tree (reached via `sysforge.verbs.registry.build_parser`) must appear in both `completions/_sysforge` and `completions/sysforge.bash`; stale top-level verb entries in the zsh case statement (function-suffix matches case-word but parser doesn't know the verb) are an error. Mirrors the `completions-cli-parity` subagent's audit; this is the mechanical layer that runs every release. +- **`completion_commands`** (`3.5.0-DEV14`) — `completions` only walked top-level commands, so after `3.5.0-F15` left group parents with no flags, no subcommand flag was checked at all, and a flag counted as present if it appeared anywhere in the file. This group checks each command where it is completed: its zsh function (`_sysforge_state_orphans`, description text excluded) and its bash `case` arm. Every long flag must be offered there, nothing the command lacks may be, every argparse `choices` value must be offered (bash keeps values in `_sysforge_flag_arg` keyed by flag, or in the command's own body), and lists copied from code constants (`_MIRRORED_LISTS`: the desktop catalog, `PROFILE_METHODS`) must equal them. Its first run found bash offering no values for `run toolchain --rebuild-soname-consumers` (`3.5.0-B127`). - **`completion_widths`** — every description in `completions/_sysforge` must fit an 80-column listing (`COMPLETION_TARGET_COLUMNS`). zsh renders a match as ` -- `, padding the match column to the longest match in that generator call; once a row exceeds the terminal width zsh abandons the inline two-column table and emits every description as its own list entry, so the whole block renders with names and descriptions detached (the 3.0.0-B2 symptom). The budget is therefore per-call and shared: `COLUMNS - (longest match in the block) - 4`, computed separately for each `_arguments` spec list and each `_describe` array (`_completion_blocks`). One over-wide row degrades every row beside it, and adding a long *option name* to a verb tightens the budget for every *description* in it — `update` is the tightest block in the file at 48 characters, bounded by `--rebuild-on-toolchain-drift`. `completions/sysforge.bash` is not checked: it emits bare `compgen -W` word lists with no descriptions and cannot exhibit this failure. - **`versions`** — `pyproject.toml` `[project] version` must equal `PKGBUILD` `pkgver=`, the leading `X.Y.Z` of `PKGBUILD-git` `pkgver=`, and every `vX.Y.Z` marker in `README.md` and `DESIGN.md` (literal `vX.Y.Z` placeholder strings in prose are filtered out by the `\d+\.\d+\.\d+` constraint). - **`manpage`** — regenerates `man/sysforge.1` via the scdoc-hybrid pipeline `make man` uses (`tools/gen_options.py` splices the argparse-derived COMMANDS sections into `man/sysforge.1.scd.in`, then `scdoc` renders) into temp files and diffs against the committed page; any difference is an error, with the fix `make man && git add man/sysforge.1`. Both sides are passed through `_normalize_roff` before diffing, so scdoc-version-specific artifacts aren't findings: the `.TH … "DATE"` header (daily date change), the `.\" Generated by scdoc ` banner, and hyphen escaping (`\-` vs `-`, functionally inert in roff). This keeps the gate coupled to the CLI surface rather than to the local scdoc build. Skipped with a `warn` if `scdoc` isn't on PATH. See the Man Pages section. diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index 2ad9b887..fdf8350c 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -176,6 +176,10 @@ https://keepachangelog.com/en/1.1.0/ --- +- **`3.5.0-DEV14` — completions are checked command by command.** `make check-shipped` gains a `completion_commands` group. The old check looked only at top-level commands, so once group commands such as `state` and `run` stopped taking flags of their own, none of their subcommands' flags were checked, and a flag counted as present if it appeared anywhere in the file. Now each command's flags must be offered where that command is completed, in both zsh and bash; no flag the command lacks may be offered; every value a flag accepts must be offered; and lists copied from the code (desktops, profile-store methods) must match it. + +--- + - **`3.5.0-DOC1` — the package description says what sysforge optimizes.** The `pkgdesc` shown by `pacman -Si` and on the AUR now names what people search for in one short line: "AUR helper that builds and tracks packages tuned to your CPU with PGO, BOLT and LTO", instead of a generic "build and maintenance suite" line. The README's opening paragraph says the same. --- @@ -759,6 +763,10 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-B126` — A kernel built with the other compiler is recorded as such, and stops warning about it.** `run kernel` recorded the system toolchain's variant whatever compiler built the kernel, so a kernel kept on gcc (`compiler = "gcc"` in `kernel.toml`, or `--toolchain gcc`) on an LLVM system was recorded as `pgo_llvm`. The record now names the compiler that built it, as `3.5.0-B125` does for other packages. The kernel stage's "Rebuilding will switch toolchains" warning now compares the record with the compiler this run will use rather than the system one, so it appears only when the kernel's compiler actually changes, and it names that compiler. +--- + +- **`3.5.0-B127` — bash completes the values of `run toolchain --rebuild-soname-consumers`.** zsh offered `prompt`, `auto` and `off` after the flag; bash offered nothing. It now offers the same three. + ## Security - **`3.5.0-B37` — the root pacman hook no longer follows links in the state directory.** The hook helper runs as root in every pacman transaction and wrote into `/var/lib/sysforge/sentinels`, whose parent any `sysforge` group member can write to. A member could swap `sentinels` for a symlink to another directory, which the next transaction made group-writable, or symlink one of its files so the hook appended to any file as root. The helper now enters the directory and acts only if it is the real `sentinels`, owned by root. It appends only to a regular file, opened so that a symlink is refused and a planted FIFO cannot stall the transaction. As before, it never fails a transaction. diff --git a/tests/test_check_shipped.py b/tests/test_check_shipped.py index c2c171bd..1374499d 100644 --- a/tests/test_check_shipped.py +++ b/tests/test_check_shipped.py @@ -1149,3 +1149,54 @@ def test_no_shipped_profile_lists_a_flag_every_build_strips(cfg_dir): offenders = {name: sorted(set(prof.get("makepkg_flags", [])) & BATCH_STRIP_FLAGS) for name, prof in data.get("profiles", {}).items()} assert {n: f for n, f in offenders.items() if f} == {} + + +# --------------------------------------------------------------------------- +# completion_commands (3.5.0-DEV14) +# --------------------------------------------------------------------------- + +_REAL = Path(__file__).resolve().parent.parent + + +def _completion_repo(tmp_path, edit=None): + """A repo whose completion scripts are the real ones, optionally edited.""" + shutil.copytree(_REAL / "completions", tmp_path / "completions") + shutil.copytree(_REAL / "etc/sysforge", tmp_path / "etc/sysforge") + if edit: + script, old, new = edit + path = tmp_path / "completions" / script + text = path.read_text() + assert old in text, old + path.write_text(text.replace(old, new, 1)) + return tmp_path + + +def _completion_findings(repo): + return [f.message for f in _load_check_shipped().check_completion_commands(repo)] + + +def test_completion_commands_clean_against_real_repo(tmp_path): + assert _completion_findings(_completion_repo(tmp_path)) == [] + + +@pytest.mark.parametrize("edit, expect", [ + # a subcommand's flag dropped from its own zsh function + (("_sysforge", "'--delete[delete one profile store", "'--nope[delete one profile store"), + "state profiles: --delete not offered in _sysforge_state_profiles"), + # the same in bash's case arm + (("sysforge.bash", '"--delete --yes --no-pager --state-dir"', '"--yes --no-pager --state-dir"'), + "state profiles: --delete not offered"), + # a flag the command no longer has, still offered + (("sysforge.bash", '"--state-dir --dry-run" -- "$cur"', + '"--state-dir --dry-run --prune" -- "$cur"'), + "state repair: offers --prune"), + # a choice value missing from bash's value case (3.5.0-B127's shape) + (("sysforge.bash", 'compgen -W "prompt auto off"', 'compgen -W "auto"'), + "run toolchain --rebuild-soname-consumers: value 'prompt' not offered"), + # a hand-copied list drifting from its code constant + (("_sysforge", ":store:(instr-pgo pgo autofdo propeller bolt)", ":store:(pgo autofdo)"), + "list differs from makepkg_pgo.PROFILE_METHODS"), +]) +def test_completion_commands_flags(tmp_path, edit, expect): + found = _completion_findings(_completion_repo(tmp_path, edit)) + assert any(expect in m for m in found), found diff --git a/tools/check_shipped.py b/tools/check_shipped.py index 52c5d4fd..a37bad66 100644 --- a/tools/check_shipped.py +++ b/tools/check_shipped.py @@ -36,6 +36,13 @@ completions every verb and long-flag in the argparse parser appears in both completions/_sysforge and completions/sysforge.bash; no stale verb entries in the zsh case statement. + completion_commands per command, in both scripts (3.5.0-DEV14): each + long flag is offered where that command is completed + (its zsh function, its bash case arm), no flag it lacks + is offered there, every argparse choice value is offered + (bash: in _sysforge_flag_arg or the command's own body), + and lists copied from code constants (desktop catalog, + profile-store methods) equal them. versions pyproject.toml == PKGBUILD pkgver == PKGBUILD-git leading pkgver == README.md / DESIGN.md markers. manpage `make man` regen output (tools/gen_options.py splice + @@ -1056,6 +1063,170 @@ def check_completions(repo: Path) -> list[Finding]: return findings +# =========================================================================== +# Group: completion_commands (3.5.0-DEV14) +# =========================================================================== + +# Both scripts are laid out per command, which is what makes this check +# possible: zsh has one function per command path (`_sysforge_state_orphans`), +# bash one function per top-level command with a `case` arm per subcommand, +# and every bash option value is offered from one `_sysforge_flag_arg` case +# keyed by flag. A flag is therefore checked where it is offered, not anywhere +# in the file. +_FN_ALIASES = {"revert-to-stock": "revert"} +_LONG_FLAG_RE = re.compile(r"(? dict[str, str]: + """Shell function name -> body (up to the closing `}` in column 0).""" + bodies = {} + for m in re.finditer(r"^(\w+)\s*\(\)\s*\{\n(.*?)^\}", text, re.MULTILINE | re.DOTALL): + bodies[m.group(1)] = m.group(2) + return bodies + + +def _with_helpers(body: str, bodies: dict[str, str]) -> str: + """A body plus the bodies of the `_sysforge_*` helpers it calls (one level).""" + called = set(re.findall(r"\b(_sysforge_\w+)\b", body)) + return body + "".join(bodies.get(fn, "") for fn in sorted(called)) + + +def _bash_arm(body: str, word: str) -> str | None: + """The text of a `case` arm labelled `word)` inside a bash function body.""" + m = re.search(rf"^\s+(?:[\w|-]+\|)?{re.escape(word)}(?:\|[\w|-]+)?\)\s*\n(.*?);;", + body, re.MULTILINE | re.DOTALL) + return m.group(1) if m else None + + +def _command_parsers(parser, path=()): + for a in parser._actions: + if isinstance(a, argparse._SubParsersAction): + for name, sub in a.choices.items(): + if (path + (name,))[0] == "completions": + continue + yield path + (name,), sub + yield from _command_parsers(sub, path + (name,)) + + +# Value lists the scripts copy from code rather than from argparse choices: +# (what, loader, [(script, function)]). Each list in those functions that +# shares a word with the constant must equal it exactly. +def _desktops(): + from sysforge.primitives.pkg_catalog import DESKTOP_CATALOG + return set(DESKTOP_CATALOG) + + +def _profile_methods(): + from sysforge.primitives.makepkg_pgo import PROFILE_METHODS + return set(PROFILE_METHODS) + + +_MIRRORED_LISTS = ( + ("pkg_catalog.DESKTOP_CATALOG", _desktops, + (("completions/_sysforge", "_sysforge_packages_add_group"), + ("completions/sysforge.bash", "_sysforge_packages"))), + ("makepkg_pgo.PROFILE_METHODS", _profile_methods, + (("completions/_sysforge", "_sysforge_state_profiles"),)), +) +# A zsh `(a b c)` value list or a bash `compgen -W "a b c"` word list. +_WORD_LIST_RE = re.compile(r':\(([\w ._-]+)\)\'|compgen -W "([^"$]+)"') + + +def _check_mirrored_lists(repo: Path) -> list[Finding]: + findings = [] + for what, load, places in _MIRRORED_LISTS: + want = load() + for script, fn in places: + body = _fn_bodies((repo / script).read_text()).get(fn) + lists = [set((m.group(1) or m.group(2)).split()) + for m in _WORD_LIST_RE.finditer(body or "")] + lists = [w for w in lists if w & want] + if not lists: + findings.append(Finding("completion_commands", "error", script, + f"{fn}: no list mirroring {what}")) + for words in lists: + if words != want: + findings.append(Finding( + "completion_commands", "error", script, + f"{fn}: list differs from {what} " + f"(missing {sorted(want - words)}, extra {sorted(words - want)})")) + return findings + + +def check_completion_commands(repo: Path) -> list[Finding]: + zsh_path, bash_path = repo / "completions/_sysforge", repo / "completions/sysforge.bash" + if not (zsh_path.exists() and bash_path.exists()): + return [] + zsh = _fn_bodies(zsh_path.read_text()) + bash = _fn_bodies(bash_path.read_text()) + flag_arg = bash.get("_sysforge_flag_arg", "") + root = _import_build_parser(repo) + global_flags = {o for a in root._actions for o in a.option_strings} + findings: list[Finding] = [] + + def err(where, msg): + findings.append(Finding("completion_commands", "error", where, msg)) + + for path, sub in _command_parsers(root): + actions = [a for a in sub._actions if a.option_strings and a.dest != "help"] + has_subs = any(isinstance(a, argparse._SubParsersAction) for a in sub._actions) + if has_subs and not actions: + continue # a group parent: its subcommands carry the flags + cmd = " ".join(path) + words = [_FN_ALIASES.get(path[0], path[0]), *path[1:]] + zfn = "_sysforge_" + "_".join(w.replace("-", "_") for w in words) + zbody = zsh.get(zfn) + if zbody is None: + err("completions/_sysforge", f"{cmd}: no {zfn} function") + else: + zfull = _with_helpers(zbody, zsh) + # Flags named inside a `[description]` are prose, not offers. + zspecs = re.sub(r"\[[^]\n]*\]", "", zbody) + flags = {o for a in actions for o in a.option_strings} + for flag in sorted(o for o in flags if o.startswith("--")): + if not re.search(rf"(? 1: # a top-level body also holds its subcommands' arms + for flag in sorted(set(_LONG_FLAG_RE.findall(scope)) - flags - global_flags): + err("completions/sysforge.bash", f"{cmd}: offers {flag}, which it does not have") + for a in actions: + if not a.choices: + continue + arm = next((_bash_arm(flag_arg, o) for o in a.option_strings + if _bash_arm(flag_arg, o) is not None), None) + if arm is None and any(f"$prev == {o}" in scope for o in a.option_strings): + arm = scope # values offered by the command itself (setup --preset) + if arm is None: + err("completions/sysforge.bash", + f"{cmd} {a.option_strings[-1]}: no value case in _sysforge_flag_arg") + continue + for value in map(str, a.choices): + if not re.search(rf"(? list[Finding]: "hooks": check_hooks, "completions": check_completions, "completion_widths": check_completion_widths, + "completion_commands": check_completion_commands, "versions": check_versions, "manpage": check_manpage, "dev_install": check_dev_install_parity, From b9f9980b97875713c2464b1b627b4212bd49cdd3 Mon Sep 17 00:00:00 2001 From: Keith Raghubar Date: Wed, 21 Oct 2026 19:16:15 -0400 Subject: [PATCH 02/16] fix(man): generate ENVIRONMENT and JOURNAL from registries (3.5.0-B128, DOC13) primitives/env_vars.py lists every environment variable sysforge reads, as user settings or diagnostic switches; the man page's ENVIRONMENT section is generated from it. It listed 3 of 11 before, with a stale editor order and pager flags, and SYSFORGE_PROFILE_STORE was documented nowhere. journal.JOURNAL_FIELDS names the fields of the journal record (record_verb now builds from its constants), and a new JOURNAL man section is generated from it, with which commands write one and two journalctl queries. tests/test_env_vars.py fails on an unregistered SYSFORGE_* name in the code and on a registered variable nothing reads. gen_options keeps `[`/`|`/`-` words off the start of a line, which scdoc would read as a table or list. Co-Authored-By: Claude Opus 5.5 --- DESIGN.md | 6 ++- docs/design/09-primitives-layer.md | 4 ++ docs/design/13-man-pages.md | 2 +- docs/release-notes/unreleased.md | 8 +++ man/sysforge.1 | 81 ++++++++++++++++++++++++++++-- man/sysforge.1.scd.in | 30 +++++------ sysforge/primitives/env_vars.py | 71 ++++++++++++++++++++++++++ sysforge/primitives/journal.py | 23 +++++++-- tests/test_env_vars.py | 69 +++++++++++++++++++++++++ tools/gen_options.py | 43 +++++++++++++++- 10 files changed, 311 insertions(+), 26 deletions(-) create mode 100644 sysforge/primitives/env_vars.py create mode 100644 tests/test_env_vars.py diff --git a/DESIGN.md b/DESIGN.md index e314ad84..48263d65 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1960,6 +1960,10 @@ Host detection and the derivations built on it: `detect_host_arch()` (uname), `p Pure constants module — the canonical directory of every config file sysforge reads. `CONFIG_BASE` is derived from `$SYSFORGE_CONFIG_DIR` (default `/`, so system config resolves under `/etc/sysforge/`, including `BOOTSTRAP_PATH`). User-side roots follow the XDG Base Directory Specification via the `_xdg_base(env, default)` helper: `USER_CONFIG_DIR` (`$XDG_CONFIG_HOME/sysforge`), `USER_CACHE_DIR` (`$XDG_CACHE_HOME/sysforge`), `USER_STATE_DIR` (`$XDG_STATE_HOME/sysforge`). The resolved path lists (`CONFIG_PATHS`, `CONFLICT_GROUP_PATHS`, `CONSUMES_INFERENCE_PATHS`) layer the user file (`$XDG_CONFIG_HOME/sysforge/…`) over the system file in `extends_system` order. Helpers: `resolve_packages_path(config)` returns the `packages.toml` path the rest of the codebase should use (honouring `--packages` overrides), and **`resolve_state_dir(cli_override)`** is the single home for state-dir resolution — `--state-dir` > `SYSFORGE_STATE_DIR` > `/var/lib/sysforge` > the XDG state dir when the default is neither writable nor provisionable — returning `(Path, source)` where `source` is one of `--state-dir`, `SYSFORGE_STATE_DIR`, `default`, `xdg-fallback`. Read-only callers pass `provision=False` (`state profiles`, `3.4.0-B20`): an unwritable default is then never provisioned, since that path can prompt for sudo, and the XDG dir is read instead when it exists, because that is where this user's earlier runs wrote when provisioning was refused. It lived in `pipeline/state.py` until `3.2.0-F1`, which meant five primitives (`makepkg_wrapper`, `state_probe`, `source_sync`, `llvm_state`, `init_notice`) reached *upward* into the pipeline layer at function level for a path computation; `pipeline.state` re-exports the name so existing imports and test patch points are unaffected. Apart from that resolver's writability probe and its `fs_provision` fallback, the module does no I/O — just path strings. +### `env_vars.py` + +The registry of every environment variable sysforge reads (`3.5.0-B128`): `ENV_VARS`, one `EnvVar(name, audience, text)` each, `audience` `user` (a setting) or `diagnostic` (a switch for bug reports). Pure data; the readers keep their own `os.environ.get` calls. `tools/gen_options.py` renders the man page's ENVIRONMENT section from it, and `tests/test_env_vars.py` fails on a `SYSFORGE_*` name in the code that it does not list (journal fields live in `journal.JOURNAL_FIELDS`), and on an entry nothing reads. + ### `config.py` TOML config loading and path resolution. Public API: @@ -3825,7 +3829,7 @@ tools/gen_options.py --template man/sysforge.1.scd.in --out man/sysforge.1.scd scdoc < man/sysforge.1.scd > man/sysforge.1 ``` -- **`man/sysforge.1.scd.in`** (committed) — hand-written NAME / SYNOPSIS / DESCRIPTION / GLOBAL OPTIONS / FILES / ENVIRONMENT / EXIT STATUS / EXAMPLES / SEE ALSO / AUTHORS prose, with an `@OPTIONS@` marker where the generated COMMANDS sections splice in and an `@SHARED_OPTIONS@` marker under a hand-written SHARED OPTIONS heading. The template stays short: a config key's reference text lives in the shipped config file's own comments (FILES says so), and DESCRIPTION gives an overview without repeating any command's options. scdoc syntax gotcha: indented continuation lines must not start with a table-control character (`[`, `|`, `]`) — scdoc errors with "Tables cannot be indented". +- **`man/sysforge.1.scd.in`** (committed) — hand-written NAME / SYNOPSIS / DESCRIPTION / GLOBAL OPTIONS / FILES / EXIT STATUS / EXAMPLES / SEE ALSO / AUTHORS prose, with an `@OPTIONS@` marker where the generated COMMANDS sections splice in and an `@SHARED_OPTIONS@` marker under a hand-written SHARED OPTIONS heading. The ENVIRONMENT and JOURNAL bodies are generated too, from `@ENVIRONMENT@` (`primitives/env_vars.ENV_VARS`, user settings then diagnostic switches, `3.5.0-B128`) and `@JOURNAL@` (`primitives/journal.JOURNAL_FIELDS`, `3.5.0-DOC13`); `tests/test_env_vars.py` fails on a `SYSFORGE_*` name the code uses that neither table lists, and on a listed variable nothing reads. `gen_options._wrap` keeps a word starting with `[`, `|` or `-` off the start of a line, since scdoc reads one there as a table or list. The template stays short: a config key's reference text lives in the shipped config file's own comments (FILES says so), and DESCRIPTION gives an overview without repeating any command's options. scdoc syntax gotcha: indented continuation lines must not start with a table-control character (`[`, `|`, `]`) — scdoc errors with "Tables cannot be indented". - **`tools/gen_options.py`** — walks `verbs.registry.build_parser()`'s subparser tree depth-first (so `packages add`, `run kernel`, etc. each get a `## ` section) — top-level commands ordered by `verbs.registry.tiered_command_order()` so the COMMANDS sections stay in lockstep with the `sysforge --help` usage tiers (§CLI Verb Framework), sub-commands in registration order — emits a synopsis line plus one definition block per positional/option (a value placeholder is the metavar, else the choices as `_a_|_b_`, matching `--help`), escapes scdoc formatting characters in help text, and performs the splice itself (no sed). Subparsers registered without `help=` (the internal `completions` data sink) are excluded. **Shared options** (`_shared_options`): an option two or more commands carry with the same head *and* the same help text is written once under SHARED OPTIONS, with a `Used by:` line, and each of those commands lists it in a `*Shared options:*` line instead of repeating the block. A flag whose help differs between commands shares only its most common text (first-seen on a tie) and the others stay inline, so SHARED OPTIONS never lists a flag twice. Sharing is keyed on identical text, so wording drift between verbs shows up as an inline duplicate: give a flag that means the same everywhere one help constant (`verbs.helpers.PACKAGES_FILE_HELP`, `CLEANSRC_FORCE_HELP`). Each command section also gets a `*Configuration:*` / `*Environment:*` trailer from the hand-maintained `_VERB_CONFIG` dict at the top of the script (qualified command name → config files / env vars consumed; commands without an entry get no trailer). The FILES and ENVIRONMENT sections of the template carry the inverse index ("Read by: …") — when a verb gains or loses a config source, update both `_VERB_CONFIG` and the template. The trailer is man-page-only by design: it is not wired into argparse epilogs, so it never appears in `--help` output (which has its own tiered `COMMAND` grouping — §CLI Verb Framework). - **`man/sysforge.1.scd`** — intermediate, gitignored. **`man/sysforge.1`** — committed, so AUR-built tarballs ship the page without build-time tooling; the PKGBUILD `package()` installs the committed file directly and needs no man-page makedepend. - Makefile target: `make man` (pins `COLUMNS=80` so any argparse-derived wrapping is deterministic). `scdoc` is a dev-machine dependency only (the `DEV_DEPS_CORE` set, installed by `make dev-deps` / `make dev`); `python-argparse-manpage` is no longer used anywhere. diff --git a/docs/design/09-primitives-layer.md b/docs/design/09-primitives-layer.md index 7d7d72f8..e2ba55a5 100644 --- a/docs/design/09-primitives-layer.md +++ b/docs/design/09-primitives-layer.md @@ -88,6 +88,10 @@ Host detection and the derivations built on it: `detect_host_arch()` (uname), `p Pure constants module — the canonical directory of every config file sysforge reads. `CONFIG_BASE` is derived from `$SYSFORGE_CONFIG_DIR` (default `/`, so system config resolves under `/etc/sysforge/`, including `BOOTSTRAP_PATH`). User-side roots follow the XDG Base Directory Specification via the `_xdg_base(env, default)` helper: `USER_CONFIG_DIR` (`$XDG_CONFIG_HOME/sysforge`), `USER_CACHE_DIR` (`$XDG_CACHE_HOME/sysforge`), `USER_STATE_DIR` (`$XDG_STATE_HOME/sysforge`). The resolved path lists (`CONFIG_PATHS`, `CONFLICT_GROUP_PATHS`, `CONSUMES_INFERENCE_PATHS`) layer the user file (`$XDG_CONFIG_HOME/sysforge/…`) over the system file in `extends_system` order. Helpers: `resolve_packages_path(config)` returns the `packages.toml` path the rest of the codebase should use (honouring `--packages` overrides), and **`resolve_state_dir(cli_override)`** is the single home for state-dir resolution — `--state-dir` > `SYSFORGE_STATE_DIR` > `/var/lib/sysforge` > the XDG state dir when the default is neither writable nor provisionable — returning `(Path, source)` where `source` is one of `--state-dir`, `SYSFORGE_STATE_DIR`, `default`, `xdg-fallback`. Read-only callers pass `provision=False` (`state profiles`, `3.4.0-B20`): an unwritable default is then never provisioned, since that path can prompt for sudo, and the XDG dir is read instead when it exists, because that is where this user's earlier runs wrote when provisioning was refused. It lived in `pipeline/state.py` until `3.2.0-F1`, which meant five primitives (`makepkg_wrapper`, `state_probe`, `source_sync`, `llvm_state`, `init_notice`) reached *upward* into the pipeline layer at function level for a path computation; `pipeline.state` re-exports the name so existing imports and test patch points are unaffected. Apart from that resolver's writability probe and its `fs_provision` fallback, the module does no I/O — just path strings. +### `env_vars.py` + +The registry of every environment variable sysforge reads (`3.5.0-B128`): `ENV_VARS`, one `EnvVar(name, audience, text)` each, `audience` `user` (a setting) or `diagnostic` (a switch for bug reports). Pure data; the readers keep their own `os.environ.get` calls. `tools/gen_options.py` renders the man page's ENVIRONMENT section from it, and `tests/test_env_vars.py` fails on a `SYSFORGE_*` name in the code that it does not list (journal fields live in `journal.JOURNAL_FIELDS`), and on an entry nothing reads. + ### `config.py` TOML config loading and path resolution. Public API: diff --git a/docs/design/13-man-pages.md b/docs/design/13-man-pages.md index 64164f31..ec121d55 100644 --- a/docs/design/13-man-pages.md +++ b/docs/design/13-man-pages.md @@ -7,7 +7,7 @@ tools/gen_options.py --template man/sysforge.1.scd.in --out man/sysforge.1.scd scdoc < man/sysforge.1.scd > man/sysforge.1 ``` -- **`man/sysforge.1.scd.in`** (committed) — hand-written NAME / SYNOPSIS / DESCRIPTION / GLOBAL OPTIONS / FILES / ENVIRONMENT / EXIT STATUS / EXAMPLES / SEE ALSO / AUTHORS prose, with an `@OPTIONS@` marker where the generated COMMANDS sections splice in and an `@SHARED_OPTIONS@` marker under a hand-written SHARED OPTIONS heading. The template stays short: a config key's reference text lives in the shipped config file's own comments (FILES says so), and DESCRIPTION gives an overview without repeating any command's options. scdoc syntax gotcha: indented continuation lines must not start with a table-control character (`[`, `|`, `]`) — scdoc errors with "Tables cannot be indented". +- **`man/sysforge.1.scd.in`** (committed) — hand-written NAME / SYNOPSIS / DESCRIPTION / GLOBAL OPTIONS / FILES / EXIT STATUS / EXAMPLES / SEE ALSO / AUTHORS prose, with an `@OPTIONS@` marker where the generated COMMANDS sections splice in and an `@SHARED_OPTIONS@` marker under a hand-written SHARED OPTIONS heading. The ENVIRONMENT and JOURNAL bodies are generated too, from `@ENVIRONMENT@` (`primitives/env_vars.ENV_VARS`, user settings then diagnostic switches, `3.5.0-B128`) and `@JOURNAL@` (`primitives/journal.JOURNAL_FIELDS`, `3.5.0-DOC13`); `tests/test_env_vars.py` fails on a `SYSFORGE_*` name the code uses that neither table lists, and on a listed variable nothing reads. `gen_options._wrap` keeps a word starting with `[`, `|` or `-` off the start of a line, since scdoc reads one there as a table or list. The template stays short: a config key's reference text lives in the shipped config file's own comments (FILES says so), and DESCRIPTION gives an overview without repeating any command's options. scdoc syntax gotcha: indented continuation lines must not start with a table-control character (`[`, `|`, `]`) — scdoc errors with "Tables cannot be indented". - **`tools/gen_options.py`** — walks `verbs.registry.build_parser()`'s subparser tree depth-first (so `packages add`, `run kernel`, etc. each get a `## ` section) — top-level commands ordered by `verbs.registry.tiered_command_order()` so the COMMANDS sections stay in lockstep with the `sysforge --help` usage tiers (§CLI Verb Framework), sub-commands in registration order — emits a synopsis line plus one definition block per positional/option (a value placeholder is the metavar, else the choices as `_a_|_b_`, matching `--help`), escapes scdoc formatting characters in help text, and performs the splice itself (no sed). Subparsers registered without `help=` (the internal `completions` data sink) are excluded. **Shared options** (`_shared_options`): an option two or more commands carry with the same head *and* the same help text is written once under SHARED OPTIONS, with a `Used by:` line, and each of those commands lists it in a `*Shared options:*` line instead of repeating the block. A flag whose help differs between commands shares only its most common text (first-seen on a tie) and the others stay inline, so SHARED OPTIONS never lists a flag twice. Sharing is keyed on identical text, so wording drift between verbs shows up as an inline duplicate: give a flag that means the same everywhere one help constant (`verbs.helpers.PACKAGES_FILE_HELP`, `CLEANSRC_FORCE_HELP`). Each command section also gets a `*Configuration:*` / `*Environment:*` trailer from the hand-maintained `_VERB_CONFIG` dict at the top of the script (qualified command name → config files / env vars consumed; commands without an entry get no trailer). The FILES and ENVIRONMENT sections of the template carry the inverse index ("Read by: …") — when a verb gains or loses a config source, update both `_VERB_CONFIG` and the template. The trailer is man-page-only by design: it is not wired into argparse epilogs, so it never appears in `--help` output (which has its own tiered `COMMAND` grouping — §CLI Verb Framework). - **`man/sysforge.1.scd`** — intermediate, gitignored. **`man/sysforge.1`** — committed, so AUR-built tarballs ship the page without build-time tooling; the PKGBUILD `package()` installs the committed file directly and needs no man-page makedepend. - Makefile target: `make man` (pins `COLUMNS=80` so any argparse-derived wrapping is deterministic). `scdoc` is a dev-machine dependency only (the `DEV_DEPS_CORE` set, installed by `make dev-deps` / `make dev`); `python-argparse-manpage` is no longer used anywhere. diff --git a/docs/release-notes/unreleased.md b/docs/release-notes/unreleased.md index fdf8350c..54cb4bba 100644 --- a/docs/release-notes/unreleased.md +++ b/docs/release-notes/unreleased.md @@ -228,6 +228,10 @@ https://keepachangelog.com/en/1.1.0/ --- +- **`3.5.0-DOC13` — the man page documents the journal records.** Commands that change installed packages or build state write a record to the systemd journal, queryable by field, but no user document said so. A new JOURNAL section names the fields (`SYSFORGE_VERB`, `SYSFORGE_TARGET`, `SYSFORGE_EXIT`), which commands write them, and two `journalctl` queries; it is generated from the field list the records are built from. + +--- + - **`3.5.0-F2` — `run toolchain` builds the LLVM toolchain from upstream's single `llvm` package.** The repos now build clang, lld, polly, compiler-rt and openmp, along with lldb, libclc and others, as part of the `llvm` package. With PGO on, the whole set now goes through the profiling build together, which also means clang's own code (`libclang-cpp`) is optimized with the profile for the first time; before, only libLLVM was. Building them separately would have meant building the entire llvm source tree two extra times. When it installs, the toolchain stage now also replaces the other packages that come from the same build and that you already have installed (for example lldb and libclc), so they match the new libLLVM, and the rollback covers them too. Packages from that build you never installed, such as the WebAssembly runtimes, are built but not installed. Your `[packages]` lists in `toolchain.toml` keep working unchanged. --- @@ -767,6 +771,10 @@ https://keepachangelog.com/en/1.1.0/ - **`3.5.0-B127` — bash completes the values of `run toolchain --rebuild-soname-consumers`.** zsh offered `prompt`, `auto` and `off` after the flag; bash offered nothing. It now offers the same three. +--- + +- **`3.5.0-B128` — the man page lists every environment variable sysforge reads.** Its ENVIRONMENT section named three of eleven. Now it also covers `SYSFORGE_MERGE` (the merge tool for `config merge`), `SYSFORGE_ASCII` (ASCII in place of decorative glyphs), `SYSFORGE_PGO_STORE` and `SYSFORGE_PROFILE_STORE` (where profiles are kept, the second documented nowhere before), and three diagnostic switches for bug reports. The `SYSFORGE_EDITOR` entry now gives the whole lookup order, which includes `sysforge.toml [ui] editor`, and the pager entry the right fallback flags. The section is generated from one list in the code, and a test fails on a variable missing from it. + ## Security - **`3.5.0-B37` — the root pacman hook no longer follows links in the state directory.** The hook helper runs as root in every pacman transaction and wrote into `/var/lib/sysforge/sentinels`, whose parent any `sysforge` group member can write to. A member could swap `sentinels` for a symlink to another directory, which the next transaction made group-writable, or symlink one of its files so the hook appended to any file as root. The helper now enters the directory and acts only if it is the real `sentinels`, owned by root. It appends only to a regular file, opened so that a symlink is refused and a planted FIFO cannot stall the transaction. As before, it never fails a transaction. diff --git a/man/sysforge.1 b/man/sysforge.1 index 131b3807..b476f109 100644 --- a/man/sysforge.1 +++ b/man/sysforge.1 @@ -1515,16 +1515,91 @@ FILES).\& .RE SYSFORGE_EDITOR .RS 4 -Editor for the interactive config review in \fBrun reconfigure\fR.\& -Falls back to $EDITOR, then $VISUAL.\& +Editor sysforge opens: \fBrun reconfigure\fR'\&s config review, +\fBartifact edit\fR and \fBconfig merge\fR.\& Checked first, then +sysforge.\&toml [ui] editor, $EDITOR and $VISUAL.\& +.PP +.RE +SYSFORGE_MERGE +.RS 4 +Merge tool for \fBconfig merge\fR.\& Checked first, then sysforge.\&toml [ui] merge, $DIFFPROG, then vimdiff.\& +.PP +.RE +SYSFORGE_ASCII +.RS 4 +Any non\-empty value prints ASCII in place of decorative glyphs +(arrows, bullets, progress fills), for terminals and fonts without +them.\& +.PP +.RE +SYSFORGE_PGO_STORE +.RS 4 +Directory for the toolchain'\&s PGO profiles, used when +toolchain.\&toml sets no pgo_store.\& +.PP +.RE +SYSFORGE_PROFILE_STORE +.RS 4 +Root directory of every other profile store (pgo/, +autofdo, propeller, bolt), used when toolchain.\&toml sets no +profile_store.\& .PP .RE PAGER .RS 4 Pager for \fBsysforge log\fR, \fBstate\fR listings, and PKGBUILD review -diffs in \fBbuild\fR/\fBupdate\fR (falls back to less \-RFX, then more).\& +diffs in \fBbuild\fR/\fBupdate\fR (falls back to less \-RF, then more).\& +.PP +.RE +SYSFORGE_PTY_RAW_DUMP (diagnostic) +.RS 4 +A file path: every byte a build writes to its terminal is appended +there, after a line giving the terminal size sysforge handed it.\& +For reports about garbled build output.\& .PP .RE +SYSFORGE_PROGRESS_TRACE (diagnostic) +.RS 4 +A file path: the progress bar logs each repaint, resize and +handover there.\& For reports about the progress bar.\& +.PP +.RE +SYSFORGE_PACMAN_NO_PYALPM (diagnostic) +.RS 4 +Any non\-empty value makes sysforge query pacman through the pacman +command rather than pyalpm, to tell a pyalpm problem from a +sysforge one.\& +.PP +.RE +.SH JOURNAL +.PP +Commands that change installed packages or build state (\fBbuild\fR, \fBupdate\fR, +\fBrevert\-to\-stock\fR, \fBuninstall\fR, \fBartifact deploy\fR/\fBremove\fR, \fBstate +repair\fR/\fBforget\fR) write one record to the systemd journal when they finish, with SYSLOG_IDENTIFIER=sysforge and these +fields, so runs can be queried rather than grepped: +.PP +SYSFORGE_VERB +.RS 4 +The command, with spaces as dashes for subcommands (build, update, +state\-forget, artifact\-deploy).\& +.PP +.RE +SYSFORGE_TARGET +.RS 4 +What it acted on, when it has a subject: pkg: for packages +(space\-separated), mode: for a state operation.\& Absent +otherwise.\& +.PP +.RE +SYSFORGE_EXIT +.RS 4 +The exit status.\& Records of a failed run are also logged at +PRIORITY=3 (err), the others at 6 (info).\& +.PP +.RE +For example, \fBjournalctl SYSFORGE_VERB=update\fR lists every update, and +\fBjournalctl SYSLOG_IDENTIFIER=sysforge PRIORITY=3\fR the failed runs.\& +.PP .SH DISTRIBUTIONS .PP SysForge assumes \fBpacman\fR(8) and \fBmakepkg\fR(8), and nothing narrower.\& It has no diff --git a/man/sysforge.1.scd.in b/man/sysforge.1.scd.in index 2fbd5b91..31321b3c 100644 --- a/man/sysforge.1.scd.in +++ b/man/sysforge.1.scd.in @@ -163,23 +163,19 @@ bench.toml (under the state directory) # ENVIRONMENT -SYSFORGE_STATE_DIR - Override the state directory (stage progress, build_state.toml, - logs, sentinels). Read by every command that touches build state: - *build*, *update*, *state*, *log*, *doctor*, and the *run* stages. - -SYSFORGE_CONFIG_DIR - Override the configuration directory searched before - /etc/sysforge. Read by every command that loads a config file (see - FILES). - -SYSFORGE_EDITOR - Editor for the interactive config review in *run reconfigure*. - Falls back to $EDITOR, then $VISUAL. - -PAGER - Pager for *sysforge log*, *state* listings, and PKGBUILD review - diffs in *build*/*update* (falls back to less -RFX, then more). +@ENVIRONMENT@ + +# JOURNAL + +Commands that change installed packages or build state (*build*, *update*, +*revert-to-stock*, *uninstall*, *artifact deploy*/*remove*, *state +repair*/*forget*) write one record to the systemd journal when they finish, with SYSLOG_IDENTIFIER=sysforge and these +fields, so runs can be queried rather than grepped: + +@JOURNAL@ + +For example, *journalctl SYSFORGE_VERB=update* lists every update, and +*journalctl SYSLOG_IDENTIFIER=sysforge PRIORITY=3* the failed runs. # DISTRIBUTIONS diff --git a/sysforge/primitives/env_vars.py b/sysforge/primitives/env_vars.py new file mode 100644 index 00000000..c49fe769 --- /dev/null +++ b/sysforge/primitives/env_vars.py @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: 2026 Keith Raghubar +# +# SPDX-License-Identifier: MIT + +""" +env_vars.py — every environment variable sysforge reads, in one table (3.5.0-B128). + +The man page's ENVIRONMENT section is generated from ``ENV_VARS`` +(``tools/gen_options.py``, the ``@ENVIRONMENT@`` marker), and +``tests/test_env_vars.py`` fails on a ``SYSFORGE_*`` name the code reads that +is not here. Before this table the man page listed three of the eleven, and +``SYSFORGE_PROFILE_STORE`` was documented nowhere. + +``audience`` is ``"user"`` for a setting a user is expected to change and +``"diagnostic"`` for a switch meant for bug reports; the man page shows both, +diagnostics last. Descriptions are scdoc text, one paragraph each. + +Public API: + EnvVar + ENV_VARS +""" +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class EnvVar: + name: str + audience: str # "user" | "diagnostic" + text: str + + +ENV_VARS: tuple[EnvVar, ...] = ( + EnvVar("SYSFORGE_STATE_DIR", "user", + "Override the state directory (stage progress, build_state.toml, logs, " + "sentinels). Read by every command that touches build state: *build*, " + "*update*, *state*, *log*, *doctor*, and the *run* stages."), + EnvVar("SYSFORGE_CONFIG_DIR", "user", + "Override the configuration directory searched before /etc/sysforge. " + "Read by every command that loads a config file (see FILES)."), + EnvVar("SYSFORGE_EDITOR", "user", + "Editor sysforge opens: *run reconfigure*'s config review, *artifact " + "edit* and *config merge*. Checked first, then sysforge.toml [ui] editor, " + "$EDITOR and $VISUAL."), + EnvVar("SYSFORGE_MERGE", "user", + "Merge tool for *config merge*. Checked first, then sysforge.toml [ui] " + "merge, $DIFFPROG, then vimdiff."), + EnvVar("SYSFORGE_ASCII", "user", + "Any non-empty value prints ASCII in place of decorative glyphs " + "(arrows, bullets, progress fills), for terminals and fonts without them."), + EnvVar("SYSFORGE_PGO_STORE", "user", + "Directory for the toolchain's PGO profiles, used when toolchain.toml " + "sets no pgo_store."), + EnvVar("SYSFORGE_PROFILE_STORE", "user", + "Root directory of every other profile store (pgo/, autofdo, " + "propeller, bolt), used when toolchain.toml sets no profile_store."), + EnvVar("PAGER", "user", + "Pager for *sysforge log*, *state* listings, and PKGBUILD review diffs " + "in *build*/*update* (falls back to less -RF, then more)."), + EnvVar("SYSFORGE_PTY_RAW_DUMP", "diagnostic", + "A file path: every byte a build writes to its terminal is appended there, " + "after a line giving the terminal size sysforge handed it. For reports " + "about garbled build output."), + EnvVar("SYSFORGE_PROGRESS_TRACE", "diagnostic", + "A file path: the progress bar logs each repaint, resize and handover " + "there. For reports about the progress bar."), + EnvVar("SYSFORGE_PACMAN_NO_PYALPM", "diagnostic", + "Any non-empty value makes sysforge query pacman through the pacman " + "command rather than pyalpm, to tell a pyalpm problem from a sysforge one."), +) diff --git a/sysforge/primitives/journal.py b/sysforge/primitives/journal.py index 34e8e3cf..a2cb82c7 100644 --- a/sysforge/primitives/journal.py +++ b/sysforge/primitives/journal.py @@ -57,6 +57,23 @@ def journal_send(**fields: str | int) -> None: TARGET_PKG_PREFIX = "pkg:" TARGET_MODE_PREFIX = "mode:" +FIELD_VERB = "SYSFORGE_VERB" +FIELD_TARGET = "SYSFORGE_TARGET" +FIELD_EXIT = "SYSFORGE_EXIT" + +#: The fields a record carries besides journald's own, as (name, scdoc text). +#: Only commands that change the system write a record. +#: The man page's JOURNAL section is generated from this (3.5.0-DOC13). +JOURNAL_FIELDS: tuple[tuple[str, str], ...] = ( + (FIELD_VERB, "The command, with spaces as dashes for subcommands " + "(build, update, state-forget, artifact-deploy)."), + (FIELD_TARGET, "What it acted on, when it has a subject: pkg: for " + "packages (space-separated), mode: for a state " + "operation. Absent otherwise."), + (FIELD_EXIT, "The exit status. Records of a failed run are also logged at " + "PRIORITY=3 (err), the others at 6 (info)."), +) + def pkg_target(names) -> str | None: """`pkg:`-namespaced ``SYSFORGE_TARGET`` for a package-subject verb. @@ -79,9 +96,9 @@ def record_verb(verb: str, target: str | None, exit_code: int) -> None: "MESSAGE": message, "PRIORITY": 6 if exit_code == 0 else 3, "SYSLOG_IDENTIFIER": "sysforge", - "SYSFORGE_VERB": verb, - "SYSFORGE_EXIT": exit_code, + FIELD_VERB: verb, + FIELD_EXIT: exit_code, } if target: - fields["SYSFORGE_TARGET"] = target + fields[FIELD_TARGET] = target journal_send(**fields) diff --git a/tests/test_env_vars.py b/tests/test_env_vars.py new file mode 100644 index 00000000..51140c14 --- /dev/null +++ b/tests/test_env_vars.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: 2026 Keith Raghubar +# +# SPDX-License-Identifier: MIT + +""" +Every environment variable and journal field sysforge uses is registered +(3.5.0-B128, 3.5.0-DOC13), and the man page is generated from the registry. + +Before primitives/env_vars.py the man page listed three of the eleven +variables, with a stale editor order and pager flags; SYSFORGE_PROFILE_STORE +was documented nowhere, and the journal fields not at all. +""" +import ast +from pathlib import Path + +from sysforge.primitives.env_vars import ENV_VARS +from sysforge.primitives.journal import JOURNAL_FIELDS + +REPO = Path(__file__).resolve().parent.parent + +# SYSFORGE_* strings that are not names sysforge reads: heredoc delimiters in +# the PKGBUILD text llvm_std_exports generates. +_NOT_NAMES = {"SYSFORGE_CMAKE", "SYSFORGE_TU"} + + +def _sysforge_strings(): + """(where, value) for each SYSFORGE_* string literal under sysforge/.""" + for py in sorted((REPO / "sysforge").rglob("*.py")): + tree = ast.parse(py.read_text(encoding="utf-8")) + for node in ast.walk(tree): + if (isinstance(node, ast.Constant) and isinstance(node.value, str) + and node.value.startswith("SYSFORGE_") + and node.value.replace("_", "").isalnum() + and node.value.isupper()): + yield f"{py.relative_to(REPO)}:{node.lineno}", node.value + + +def test_every_sysforge_name_in_the_code_is_registered(): + known = {v.name for v in ENV_VARS} | {f for f, _ in JOURNAL_FIELDS} | _NOT_NAMES + bad = [f"{where}: {name}" for where, name in _sysforge_strings() if name not in known] + assert not bad, ("register these in primitives/env_vars.ENV_VARS " + "(or journal.JOURNAL_FIELDS):\n " + "\n ".join(bad)) + + +def test_every_registered_variable_is_read_somewhere(): + """A stale entry documents a variable that no longer does anything.""" + used = {name for _, name in _sysforge_strings()} + source = "".join(p.read_text(encoding="utf-8") + for p in (REPO / "sysforge").rglob("*.py") + if p.name != "env_vars.py") + stale = [v.name for v in ENV_VARS + if v.name not in used and f'"{v.name}"' not in source] + assert not stale, f"registered but never read: {stale}" + + +def test_registry_shape(): + names = [v.name for v in ENV_VARS] + assert len(names) == len(set(names)) + assert {v.audience for v in ENV_VARS} <= {"user", "diagnostic"} + assert all(v.text.endswith(".") for v in ENV_VARS) + + +def test_man_page_lists_every_variable_and_journal_field(): + man = (REPO / "man/sysforge.1.scd.in").read_text(encoding="utf-8") + assert "@ENVIRONMENT@" in man and "@JOURNAL@" in man + rendered = (REPO / "man/sysforge.1").read_text(encoding="utf-8") + missing = [n for n in [v.name for v in ENV_VARS] + [f for f, _ in JOURNAL_FIELDS] + if n.replace("_", "\\_") not in rendered and n not in rendered] + assert not missing, f"not in man/sysforge.1 (run `make man`): {missing}" diff --git a/tools/gen_options.py b/tools/gen_options.py index e9bf0f33..72e25182 100644 --- a/tools/gen_options.py +++ b/tools/gen_options.py @@ -34,6 +34,8 @@ MARKER = "@OPTIONS@" SHARED_MARKER = "@SHARED_OPTIONS@" +ENV_MARKER = "@ENVIRONMENT@" +JOURNAL_MARKER = "@JOURNAL@" # Per-command configuration/environment summary, rendered as a trailer after # each command's option blocks. Values are raw scdoc (not escaped). Keyed by @@ -277,6 +279,43 @@ def _fill(template: str, marker: str, lines: list[str]) -> str | None: return rendered +def _wrap(text: str) -> list[str]: + """Tab-indented scdoc paragraph lines. scdoc reads a line starting with + `[`, `|` or `-` as a table or list, so such a word is kept on the line + before it.""" + import textwrap + lines: list[str] = [] + for line in textwrap.wrap(text, width=66): + if lines and line[:1] in "[|-": + lines[-1] += " " + line + else: + lines.append(line) + return ["\t" + line for line in lines] + + +def render_environment() -> list[str]: + """The ENVIRONMENT section body, from primitives/env_vars.ENV_VARS + (3.5.0-B128): user settings first, then the diagnostic switches.""" + from sysforge.primitives.env_vars import ENV_VARS + lines: list[str] = [] + ordered = [v for v in ENV_VARS if v.audience == "user"] + \ + [v for v in ENV_VARS if v.audience == "diagnostic"] + for var in ordered: + label = var.name + (" (diagnostic)" if var.audience == "diagnostic" else "") + lines += [label, *_wrap(var.text), ""] + return lines[:-1] + + +def render_journal() -> list[str]: + """The JOURNAL section's field list, from primitives/journal.JOURNAL_FIELDS + (3.5.0-DOC13).""" + from sysforge.primitives.journal import JOURNAL_FIELDS + lines: list[str] = [] + for name, text in JOURNAL_FIELDS: + lines += [name, *_wrap(text), ""] + return lines[:-1] + + def main() -> int: ap_ = argparse.ArgumentParser(description=__doc__) ap_.add_argument("--template", required=True) @@ -286,7 +325,9 @@ def main() -> int: shared_lines, command_lines = render(build_parser(), tiered_command_order()) rendered = Path(ns.template).read_text(encoding="utf-8") - for marker, lines in ((SHARED_MARKER, shared_lines), (MARKER, command_lines)): + for marker, lines in ((SHARED_MARKER, shared_lines), (MARKER, command_lines), + (ENV_MARKER, render_environment()), + (JOURNAL_MARKER, render_journal())): filled = _fill(rendered, marker, lines) if filled is None: print(f"error: marker {marker} not found in {ns.template}", From 03ae418a7b08a18a96674d2c52cb251a3797c788 Mon Sep 17 00:00:00 2001 From: Keith Raghubar Date: Wed, 21 Oct 2026 19:28:08 -0400 Subject: [PATCH 03/16] feat(config): check, clean and reset the live config (3.5.0-F18) An upgrade never rewrites live config files, so retired settings stayed, silently ignored, and misspelt keys were never reported. primitives/config_schema.py is the one home for what a live file may hold: the section/top-key schema (moved from tools/check_shipped.py, which now imports it), the user-named open sections, keys valid because the shipped default documents them (active or commented), and RETIRED_KEYS / RETIRED_VALUES. The pristine defaults ship in the wheel (sysforge/_data/config-defaults) since /etc/sysforge holds the user's copies. Edits are line-based (comments survive), re-parsed with tomllib before writing, and written atomically. - config check: retired and unknown settings with their line; exits 1. - config clean: renames retired keys (or drops one whose replacement is set, saying so), rewrites retired values; --unknown removes unknown ones. - config reset FILE [KEY...]: keys back to the shipped line, or the whole file with a FILE.bak-