Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions DESIGN.md

Large diffs are not rendered by default.

5 changes: 3 additions & 2 deletions completions/_sysforge
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ _sysforge() {
'--py-profile-out[write raw cProfile stats to FILE]:file:_files' \
'--timings[print phase wall-clock timing report after build/update]' \
'--color[colorize output]:when:(auto always never)' \
'--unattended[never prompt, even at a terminal]' \
'--no-throttle[ignore the configured build throttle for this run]' \
'--turbo[raise build priority (stronger than --no-throttle)]' \
'--frozen[refuse all new source downloads for this run]' \
Expand Down Expand Up @@ -159,7 +160,7 @@ _sysforge_build() {
_arguments \
$_sysforge_help_spec \
'(-m --makepkg)'{-m,--makepkg}'[extra makepkg flags, appended after profile flags]:flags' \
'--interactive[strip --noconfirm from makepkg_flags]' \
'--interactive[run makepkg without --noconfirm, attached to the terminal]' \
'--profile-conf[alternate profiles.toml]:file:_files' \
'--toolchain[build with this toolchain as a whole]:toolchain:(gcc llvm)' \
'(--reset-toolchain)--save-toolchain[also save --toolchain for these packages]' \
Expand Down Expand Up @@ -710,7 +711,7 @@ _sysforge_run_kernel() {
'--no-update[skip the source sync, build as checked out]' \
'--cleansrc[purge the kernel src dir and re-clone]' \
'--cleansrc-force[--cleansrc, bypassing the dirty guard]' \
'--no-interactive[use olddefconfig, no interactive kconfig]' \
'--no-review-config[skip the kernel config review (olddefconfig)]' \
'--toolchain[build the kernel with this toolchain]:toolchain:(gcc llvm)' \
'--bootloader[override kernel.toml bootloader]:bootloader:(systemd-boot grub none)' \
'--base-config[override kernel.toml base_config]:source:_sysforge_base_config_source' \
Expand Down
4 changes: 2 additions & 2 deletions completions/sysforge.bash
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ _sysforge() {

if [[ -z $verb ]]; then
if [[ $cur == -* ]]; then
COMPREPLY=( $(compgen -W "-h --help -v --verbose --quiet --py-profile --py-profile-out --timings --color --no-throttle --turbo --frozen --no-frozen --thaw" -- "$cur") )
COMPREPLY=( $(compgen -W "-h --help -v --verbose --quiet --py-profile --py-profile-out --timings --color --unattended --no-throttle --turbo --frozen --no-frozen --thaw" -- "$cur") )
else
COMPREPLY=( $(compgen -W "$commands" -- "$cur") )
fi
Expand Down Expand Up @@ -465,7 +465,7 @@ _sysforge_run() {
kernel)
[[ $cur == -* ]] && COMPREPLY=( $(compgen -W "\
--dry-run --no-update --cleansrc --cleansrc-force \
--no-interactive --toolchain --bootloader --base-config \
--no-review-config --toolchain --bootloader --base-config \
--base-config-merge \
--allow-no-fallback --allow-boot-audit-failure \
--headers --no-headers --docs --no-docs \
Expand Down
2 changes: 1 addition & 1 deletion docs/design/05-package-manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ An entry with only `name` and no override fields has no effect on the build. `sy

```toml
[group.cosmic]
packages = ["cosmic-session-git", "cosmic-comp-git", "cosmic-settings-git"]
packages = ["cosmic-session-git", "cosmic-comp-git", "cosmic-settings-git", ...]
# Optional defaults inherited by every member:
# enable_build_from_source = true
```
Expand Down
10 changes: 5 additions & 5 deletions docs/design/07-pipeline-layer.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/design/08-cli-verb-framework.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ spelling per meaning, across every verb:
`state profiles --delete STORE`), with `--yes` to skip its prompt.
`artifact remove --purge` keeps the package-manager sense of also deleting
the managed copy.
- **A negative switch is `--no-<thing>`** (`--no-update`, `--no-interactive`),
- **A negative switch is `--no-<thing>`** (`--no-update`, `--no-review-config`),
or a `--<thing>/--no-<thing>` pair (`BooleanOptionalAction`) when config can
default it either way (`--headers/--no-headers`). Never `--non-…`/`--disable-…`.
- **No two flags differ only by a plural `s`** (`--no-pkg-log`, not
Expand Down
6 changes: 3 additions & 3 deletions docs/design/09-primitives-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Integration sites: `sysforge/verbs/runner.py::run_verb` (universal `<verb>: star

### `prompt.py`

Single shared interactive-prompt helper. Every stage that needs user input goes through `prompt.prompt_text()` (free-form), `prompt.prompt_choice()` (fixed-set), or `prompt.prompt_key()` (single keypress), so the behaviour on empty input, unrecognized input, and EOF is consistent across the codebase. `is_interactive()` wraps `sys.stdin.isatty()` for stages that need to gate prompts on a TTY.
Single shared interactive-prompt helper. Every stage that needs user input goes through `prompt.prompt_text()` (free-form), `prompt.prompt_choice()` (fixed-set), or `prompt.prompt_key()` (single keypress), so the behaviour on empty input, unrecognized input, and EOF is consistent across the codebase. `is_interactive()` wraps `sys.stdin.isatty()` for stages that need to gate prompts on a TTY, and is the only stdin TTY check in the package (`tests/test_unattended.py` fails on another). **Unattended runs** (`3.5.0-F21`): `set_unattended()` is called once by `cli._main` from `cli._resolve_unattended` (`--unattended` > `SYSFORGE_UNATTENDED`, which also turns a configured on off for one run > `[ui] unattended` > off). While set, `is_interactive()` is false even at a terminal and the three helpers take their EOF path without reading (`prompt_text`/`prompt_choice` return `eof_default`, `prompt_key` raises `EOFError`), so the run behaves exactly as one with stdin closed, including at a call site that never checked `is_interactive()`. It changes *whether sysforge may ask*, nothing else: it is not a global `--yes` (a removal still refuses without one; the review gate still refuses an unreviewed change rather than accepting it), and it does not touch the run-level `--interactive` flags of `build`/`update`, which decide whether makepkg and pacman get the terminal. sudo reads the terminal itself, so the EOF path cannot reach its password prompt: `privilege.ensure_credentials()` returns `False` without running the prompting `sudo -v` (cached credentials or `NOPASSWD` still pass the `sudo -n true` probe), and `privilege.privileged_argv()` adds `-n` to every escalation, so one that skipped `authenticate()` fails fast instead of waiting for a password. Refuse/skip messages take their reason from `no_prompt_reason()` (`"unattended run"` or `"no terminal"`). `unattended()` distinguishes a chosen unattended run from a missing TTY, so the kernel stage narrates the skipped kconfig review instead of warning about it.

Key contract:

Expand Down Expand Up @@ -390,7 +390,7 @@ Probes are sub-second: `rust:native` is `rustc --version`, `rust:cross:<target>[

The clang probe additionally checks the `LLVM_LOCKSTEP_SUITE` for a pkgver skew, and reports it **structurally** on `ToolchainCheck.versions` — a tuple of `(label, installed_ver, target_ver)` rows where every version group is stated against the newest installed pkgver (resolved with `vercmp`, falling back to a lexical max if the pacman binary is unavailable, since a display helper must never abort a batch). `render_preflight` turns those rows into an indented `clang/lld: 22.1.5 → 22.1.6` table under the check's detail line, collapsing the already-current group to `llvm/llvm-libs: 22.1.6 (=)`. The probe reports facts; the renderer owns formatting. Checks with nothing version-shaped to say carry an empty `versions` and render exactly as before — a healthy suite is one detail line, not seven rows.

Auto-remediation: failures with `auto_remediable=True` (currently `rustup target add` only) get an interactive y/n prompt via `primitives.prompt.prompt_choice`; on accept the command is executed and the failed probe is re-run. Non-interactive runs (`--no-interactive`, `--noconfirm`, or no TTY) print the fix block and exit 1 instead. Per-package profile-resolution failures are caught and warned — preflight is best-effort and never blocks a build the real wrapper could still succeed at.
Auto-remediation: failures with `auto_remediable=True` (currently `rustup target add` only) get an interactive y/n prompt via `primitives.prompt.prompt_choice`; on accept the command is executed and the failed probe is re-run. A run that may not ask (`--unattended` or no TTY, via `prompt.is_interactive()`) prints the fix block and exits 1 instead. Per-package profile-resolution failures are caught and warned — preflight is best-effort and never blocks a build the real wrapper could still succeed at.

Public API: `collect_required_toolchains(per_pkg, lib32_pkgs, rust_toolchain_pins=None, compilers=None)`, `run_preflight(required)`, `render_preflight(report)`, `auto_remediate(report, *, non_interactive=False)`. Wiring: `update` only, behind `--no-toolchain-preflight`. The companion `primitives.build_diag.diagnose` runs in `makepkg_wrapper.invoke_makepkg` on non-zero exit and matches known failure signatures (E0463 missing std crate, gstreamer PTP-no-rust, meson "Unknown options" → stale build dir, `cuda:host-gcc-too-new` → nvcc rejecting a system gcc newer than the CUDA toolkit supports, `toolchain:lib32-reduced-target` → a link-time `ld.lld: undefined symbol: LLVMInitialize…` against a lib32 libLLVM (matched with an `-m32` / `lib32/libLLVM` signal) — lib32-clang's offload tools referencing all-target init symbols the reduced lib32 libLLVM lacks, pointing at building lib32 with the full target set rather than a spurious version skew; `toolchain:lib32-clang-libgcc` → a lib32 (`-m32`) build where `ld.lld` can't find the 32-bit `libgcc_s` clang implicitly links (`unable to find library`/`cannot find -lgcc_s`, gated on an `-m32`/`lib32` signal so a 64-bit miss doesn't match), which CMake/meson surfaces as the compiler being "broken" even though clang is fine — points at a per-package `--cc gcc --cxx g++` override (gcc `-m32` links its own multilib libgcc); `toolchain:llvm-broken` → a clang/libLLVM mismatch where clang can't run, matched on `undefined symbol: LLVMInitialize…` / `symbol lookup error: …clang` / meson's `Unknown compiler(s): [['clang']]` (the lib32 reduced-target case is matched first and suppresses this generic one); `pkgconfig:version-gate` → meson's canonical `Dependency lookup for <mod> with method 'pkg-config' failed: Invalid version, need '<mod>' ['>= X'] found 'Y'` line, which is accurate but leaves the operator to work out by hand who owns the `.pc` and whether the repos can satisfy the floor at all — the matcher parses module/floor/found, resolves the owner of `<mod>.pc` via `pacman.owners_of` over `/usr/lib`, `/usr/share` and `/usr/lib32` pkgconfig dirs, and reports the repo version (`pacman.get_pacman_sync_version` + `version.vercmp` against the floor) plus whether an AUR `<owner>-git` exists (`aur.aur_info`). Every probe is best-effort — a failure degrades the message, never the diagnosis. It emits **no** `fix_cmd` by design: waiting for the repo bump and adopting the `-git` dep have materially different costs (the latter forfeits the distro's soname-rebuild guarantee, for which sysforge has no general equivalent), and a copy-pasteable command would pre-pick the riskier one; `deps:undeclared` → a dependency the PKGBUILD never declared, which the host carries but a clean (sandboxed) build environment lacks: pkgconf's `Package 'X' not found` (owner of `X.pc` resolved via the same `_pc_owner` probe, or a `pacman -F X.pc` lookup when the host lacks it too) and makepkg's `Cannot find the X package needed to handle Y sources`, naming the package to add to `makedepends` and flagging it as a PKGBUILD defect to report upstream; `cxx:module-identifier` → clang's `module declaration must not come from an #include directive` (P1857: a line-initial `module` identifier, e.g. a `module _module;` member, read as a C++20 module declaration, which GCC still accepts), pointing at the per-package gcc swap) in the captured output and any `meson-logs/meson-log.txt` **or `CMakeFiles/CMakeError.log`** under the build directory; deduped on signature, never masks the real error. The CUDA matcher reads the toolkit's `crt/host_config.h` `#if __GNUC__ > N` gate and the highest installed `/usr/bin/g++-≤N` to emit a concrete `NVCC_APPEND_FLAGS='-ccbin …'` fix. Each `FixSuggestion`'s `signature`/`fix_cmd` is also carried up the exception (`.diagnosis`) and persisted to `build_state.toml`'s `[failures]` table by `sysforge update` (see §`build_state.py`). **Interactive builds** inherit the TTY so makepkg's stdout is never captured; on failure `invoke_makepkg` instead runs `diagnose([], _effective_build_dir(...))` over the side-car logs (resolving `$BUILDDIR/<pkgbase>` when `BUILDDIR` redirects the build out-of-tree) and threads the result through the user-abort RuntimeError, so `state failed` records a real signature rather than "Aborted by user". Log tags: `[PREFLIGHT]`, `[DIAG]`.

Expand Down Expand Up @@ -515,7 +515,7 @@ High-level flow:
4. Import GPG keys via `aur.import_pgp_keys` (bundled `keys/pgp/*.asc` first, keyserver fallback)
5. Run pre-build soname dep analysis
6. If `source_built` or `kernel` mode: extract PKGBUILD flags, write extracted profile, apply patch
7. If `kernel` mode and not `interactive`: patch interactive kconfig targets in `PKGBUILD.sysforge` to `olddefconfig`
7. If `kernel` mode and the config review is off (`interactive=False`, from `resolve_config_review`): patch interactive kconfig targets in `PKGBUILD.sysforge` to `olddefconfig`
8. If `kernel` mode: detect effective CC; if clang, inject `LLVM=1 LLVM_IAS=1` into build env
9. Emit complete temp `makepkg.conf` (merged system conf + profile overrides; kernel mode omits `CFLAGS`/`CXXFLAGS`/`LDFLAGS`/`CPPFLAGS`/`DEBUG_*` profile overrides — system conf values preserved verbatim)
10. Resolve env vars for subprocess injection
Expand Down
3 changes: 1 addition & 2 deletions docs/design/10-flag-profile-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ extends = "bare"
CFLAGS = "-march=native -O2 -pipe"
CXXFLAGS = "$CFLAGS"
LDFLAGS = "-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"
makepkg_flags = ["--noconfirm"]

[profiles.optimized]
extends = "standard"
Expand All @@ -37,7 +36,7 @@ build_mode = "source_built" # the old token "patched_pkgbuild" raises since 4.
extends = "bare"
build_mode = "kernel"
batch = true
makepkg_flags = ["--noconfirm", "-f", "-c"]
makepkg_flags = ["-f", "-c"]
```

### `build_mode` vocabulary
Expand Down
9 changes: 5 additions & 4 deletions docs/design/11-makepkg-wrapper.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ makedep_probe_failed = "warn_and_fallback"

### Interactive mode

`--interactive` on `sysforge build` does two things: it strips `--noconfirm` from the profile's `makepkg_flags`, and it runs makepkg under `pty_runner.run_with_pty(..., interactive=True)`, a two-way proxy in which the pty is makepkg's stdin, stdout, stderr and controlling terminal and the user's keystrokes are copied into it, inside `yield_terminal("child")`, instead of piping output through the line-classification loop. Before `3.2.0-F19` it inherited the parent's stdio outright; the proxy keeps everything that made that worthwhile. Passing every byte straight through is what keeps unbuffered prompts visible — pacman's conflict prompt (`Remove sysforge? [y/N]`) and similar `\r`/no-newline output reaches the terminal immediately rather than sitting in a pipe buffer until the user blindly presses Enter. The tradeoff is that line-based output classification (`failed_stage`, `missing_deps`, `toolchain_mismatch` auto-retry, stdout-match fallback for `AlreadyBuilt`, `captured_output` for `auto_repair`) is still bypassed in this branch, but every line reaches `invoke_makepkg`: the stream is teed (ANSI-stripped) into the per-package log through `log.write_pkg_log`, and on a failure the anchored tail (`select_failure_tail`) is repeated under `[BUILD]` at DEBUG, which lands in the unified run log and the per-package log (`_log_interactive_failure_tail`); exit-code-based detection (`returncode == 13` → `AlreadyBuilt`, `returncode == 8` → install failure) still fires. Useful during development to review makepkg prompts without editing the profile; not appropriate for `update` batch flows, which depend on the classification path and therefore default `interactive=False`.
Whether makepkg may prompt is the run's decision, not the profile's (`3.5.0-F20`): `makepkg_invoke._base_flags` puts `--noconfirm` in front of the profile's `makepkg_flags` for every batch build and leaves it off an interactive one, removing a copy a profile written before that still carries (so it is never doubled and never survives into an interactive run). The build sandbox needs no special case, since `makechrootpkg` passes `--noconfirm` itself. `--interactive` on `sysforge build` does two things: it runs makepkg without `--noconfirm`, and it runs makepkg under `pty_runner.run_with_pty(..., interactive=True)`, a two-way proxy in which the pty is makepkg's stdin, stdout, stderr and controlling terminal and the user's keystrokes are copied into it, inside `yield_terminal("child")`, instead of piping output through the line-classification loop. Before `3.2.0-F19` it inherited the parent's stdio outright; the proxy keeps everything that made that worthwhile. Passing every byte straight through is what keeps unbuffered prompts visible — pacman's conflict prompt (`Remove sysforge? [y/N]`) and similar `\r`/no-newline output reaches the terminal immediately rather than sitting in a pipe buffer until the user blindly presses Enter. The tradeoff is that line-based output classification (`failed_stage`, `missing_deps`, `toolchain_mismatch` auto-retry, stdout-match fallback for `AlreadyBuilt`, `captured_output` for `auto_repair`) is still bypassed in this branch, but every line reaches `invoke_makepkg`: the stream is teed (ANSI-stripped) into the per-package log through `log.write_pkg_log`, and on a failure the anchored tail (`select_failure_tail`) is repeated under `[BUILD]` at DEBUG, which lands in the unified run log and the per-package log (`_log_interactive_failure_tail`); exit-code-based detection (`returncode == 13` → `AlreadyBuilt`, `returncode == 8` → install failure) still fires. Useful during development to review makepkg prompts without editing the profile; not appropriate for `update` batch flows, which depend on the classification path and therefore default `interactive=False`.

### Toolchain-mismatch auto-retry

Expand All @@ -95,8 +95,9 @@ exit 13 and fails hard instead (3.0.0-B9);
`"review-gated"` (kernel stage) preserves the B5 semantics — the skipped build
also skipped the promised in-prepare() kconfig review, so interactive runs get
an install-as-built / rebuild-with-`-f` / abort prompt while unattended runs
proceed. The unattended arbitration (caller interactivity ∧ `--no-interactive`
∧ TTY) lives only in the seam. `makepkg_wrapper`'s own `except AlreadyBuilt`
proceed. The kernel caller passes its one resolved `config_review`
(`resolve_config_review`, which already folds in `review_config`,
`--no-review-config`, `--unattended` and TTY), so the seam re-derives nothing. `makepkg_wrapper`'s own `except AlreadyBuilt`
(manifest capture for renamed builds) is a side-effect that fires regardless of
policy, then re-raises.

Expand Down Expand Up @@ -249,7 +250,7 @@ the full bundle swap (`_reemit_for_swap` → `profile.swap_toolchain`).
[profiles.batch]
extends = "standard"
batch = true
makepkg_flags = ["--noconfirm", "--noprogressbar", "--log", "--cleanbuild"]
makepkg_flags = ["--noprogressbar", "--log", "--cleanbuild"]
clean_builddir = true
```

Expand Down
Loading
Loading