From 786a16b81ab46c55bf134a34262ad8cc8996f7a1 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 11:53:01 -0700 Subject: [PATCH 01/27] docs: RFC for slotted system extensions Adds a 'slotted' scope for sysexts/confexts, stored per A/B volume in a configurable store (default /var/trident/sysexts) and activated via an /etc/extensions symlink written during servicing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 205 ++++++++++++++++++ 1 file changed, 205 insertions(+) create mode 100644 docs/Development/RFCs/0838-slotted-sysexts.md diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md new file mode 100644 index 0000000000..bddbd860b7 --- /dev/null +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -0,0 +1,205 @@ +# 0838 Slotted system extensions + +- Date: 2026-10-08 +- RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) +- Issue: TBD + +## Summary + +Add a `slotted` scope for sysexts/confexts. Trident stores them per A/B volume +under a configurable store (default `/var/trident/sysexts/{a,b}/`) and creates +the activating `/etc/extensions/.raw` symlink during servicing. + +## Motivation + +`systemd-sysext(8)` merges an extension only if its `extension-release` matches +the running OS: `ID=` must match unless `_any`, then `SYSEXT_LEVEL=` if defined, +else `VERSION_ID=`. + +All three directories Trident accepts today (`/etc/extensions`, +`/var/lib/extensions`, `/.extra/sysext`) are outside the partitions an A/B +update replaces. So an OS-pinned extension silently stops merging after the +first update: no error, no failed unit, the content is just gone. + +Azure Container Linux hits this. Its Azure platform extension (guest agent, +Hyper-V daemons, chrony, `oras`) is `VERSION_ID`-pinned on a non-swapped +partition. Losing `oras` also breaks the distro's extension download path. + +The shape that works already exists in the same image: the container runtime +ships as an extension inside `/usr`, so extension and `os-release` swap +together. This proposal gives that property to anyone who cannot re-seal +dm-verity and re-sign the UKI. + +Goals: correct extension version per booted slot; rollback restores the previous +set; no new partitions, no `/usr` change, no re-signing; existing configs +unaffected. + +Non-goals: changing systemd matching semantics; ESP-based extensions. + +## Public API Design + +### Today + +```rust +pub struct Extension { + pub url: Url, + pub sha384: Sha384Hash, + pub path: Option, // absolute, allow-listed dir, *.raw +} +``` + +`path` conflates *where the file is stored* with *what scope it is active in*. +Slotting separates them: storage becomes Trident-owned and derived from the +target volume. + +### Proposed + +```rust +pub struct Extension { + pub url: Url, + pub sha384: Sha384Hash, + + /// Absolute path in the target OS. Only valid when `scope` is `shared`. + pub path: Option, + + #[serde(default)] + pub scope: ExtensionScope, +} + +pub enum ExtensionScope { + /// Stored at `path`, active on both volumes. For `ID=_any` extensions. + #[default] + Shared, + /// Stored per-volume in the extension store, active only on the volume it + /// was installed to. For OS-version-pinned extensions. + Slotted, +} +``` + +Plus one optional field on `os`: + +```rust +/// Root of the per-volume extension store. Default `/var/trident/sysexts`. +/// Must be on storage not replaced by an A/B update, and must not be a +/// systemd-sysext search path. +pub extension_store: Option, +``` + +Configurable because slotting doubles storage, and large extensions (GPU +drivers, ML runtimes) may need a dedicated volume. Confexts use a sibling +`confexts/` under the same root. + +```yaml +os: + extensionStore: /mnt/bigdisk/trident/sysexts # optional + sysexts: + - url: https://example.com/tooling.raw # unchanged, still default + sha384: + - url: https://example.com/platform.raw + sha384: + scope: slotted +``` + +### Validation + +| Condition | Result | +|---|---| +| `scope` absent | `shared`. Identical to today. | +| `scope: shared` + `path` | Existing allow-list check. | +| `scope: slotted` + `path` | Reject. Path is Trident-owned; accepting it would silently ignore it. | +| `scope: slotted`, no `path` | Filename from `url` basename, as today's default. | +| `scope: slotted`, no A/B volumes | Reject at validation, not at provision. | +| `extensionStore` set, no `slotted` entries | Accept, unused. | +| `extensionStore` inside a sysext search path | Reject: both copies would be discovered. | + +### Backward compatibility + +Both fields are `#[serde(default)]`. Existing configs parse and behave +identically. Schema gains two optional properties. + +### Alternatives rejected + +- **Sentinel `path`** (`/var/trident/sysexts//x.raw`): leaks internal + layout, invites a literal `a`/`b`. +- **Separate `os.slottedSysexts` list**: duplicates the type and its + duplicate-hash/path validation. +- **Infer from `extension-release`**: metadata is inside the image, unavailable + at static validation, and removes the operator's choice to slot an `_any` + extension deliberately. + +## Implementation + +``` +/var/trident/sysexts/{a,b}/platform.raw +/var/trident/confexts/{a,b}/ +``` + +Both directories persist across the update, which is what makes rollback a +symlink flip rather than a re-download. + +The extensions subsystem already runs on `CleanInstall` and `AbUpdate` via +`provision()`, documented as "migrate state from A-partition to B-partition (or +vice versa)". + +1. **prepare** — download, verify SHA-384. Unchanged. +2. **provision** — resolve target volume via + `EngineContext::get_ab_update_volume()` (already used for per-slot verity + addons); place into `//`. +3. **activate** — write `/etc/extensions/.raw` → + `//.raw`. `/etc` writes are already supported via + `Subsystem::writable_etc_overlay()`. + +The symlink supplies the `extension-release.NAME`-matching filename; the stored +file may be named freely. Same indirection distributions already use. + +**Shared `/etc`.** The symlink is not slotted, so between servicing and reboot +the running OS points at an extension whose `VERSION_ID` does not match it. This +is inert: already-merged extensions stay merged, the mismatched one is not +merged. + +**Rollback** repoints symlinks at the previous volume's directory. Inverse of +step 3. + +**Ordering.** Symlinks are written before reboot, so they are correct before +`systemd-sysext.service` runs at `Before=sysinit.target`. + +## Testing + +- Clean install: file in correct volume dir, symlink correct, merged after boot. +- A/B update: symlink repoints and **merged content differs from the previous + slot**. A presence-only assertion passes in the inert case and must not be + used. +- Rollback: symlinks and merged content return to the previous slot. +- Validation: each reject row above. +- Compat: config with no `scope` places byte-identically to today. +- Non-default `extensionStore` on a separate mount. + +## Counter-arguments + +- **"Put it in `/usr`."** Strictly better where possible, but requires re-sealing + verity and re-signing the UKI. This targets that gap. +- **"Use the ESP."** `.efi.extra.d/*.sysext.raw` is genuinely slot-scoped, + but ESPs are 128-256 MiB against payloads in the tens to hundreds of MB, and + `systemd-stub` places them in the initrd, not the running system. +- **"Add A/B extension partitions."** Works, but layout changes are breaking + after the first A/B image ships. Directories achieve the same isolation. +- **"Doubles storage."** Yes. Mitigated by `extensionStore`. + +## Open questions + +1. **Does the rollback path invoke subsystem steps?** Largest unknown; gates the + design. +2. **Should Host Status report slotted extensions per volume?** Derivable from + `ab_active_volume` + store, but not inspectable without reading the FS. +3. **Retention.** Nothing prunes a slot directory when an entry is dropped from + the config. Should `provision()` reconcile, and does that break rollback? +4. **SELinux.** Extensions are already documented as unsupported with SELinux + enabled; a Trident-created `/etc` symlink may additionally need a label. +5. **Default path.** `/var/trident/sysexts` vs `/var/lib/trident/sysexts`; the + datastore already lives under `/var/lib/trident/`. + +## Future possibilities + +- Per-volume scoping for other Trident-managed state that is shared today. +- Changing an extension's scope between servicing operations, which currently + strands a copy in the old location. From 9208499ddd3eb3d298710cbcfce55e501d427789 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 11:57:30 -0700 Subject: [PATCH 02/27] docs: resolve rollback question in slotted sysext RFC The A/B rollback path invokes no subsystems: automatic rollback is firmware-driven and validate_boot only reconciles Host Status, while manual_rollback::stage_rollback touches only the pcrlock policy. A symlink written during servicing would therefore survive a rollback pointing at the wrong slot. Activation now resolves at boot from the booted volume, via a single /current flip ordered before systemd-sysext.service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 108 ++++++++++++------ 1 file changed, 76 insertions(+), 32 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index bddbd860b7..8695c3b49a 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -7,8 +7,8 @@ ## Summary Add a `slotted` scope for sysexts/confexts. Trident stores them per A/B volume -under a configurable store (default `/var/trident/sysexts/{a,b}/`) and creates -the activating `/etc/extensions/.raw` symlink during servicing. +under a configurable store (default `/var/trident/sysexts/{a,b}/`) and activates +the correct slot at boot, before `systemd-sysext` merges. ## Motivation @@ -31,8 +31,8 @@ together. This proposal gives that property to anyone who cannot re-seal dm-verity and re-sign the UKI. Goals: correct extension version per booted slot; rollback restores the previous -set; no new partitions, no `/usr` change, no re-signing; existing configs -unaffected. +set without a Trident servicing operation; no new partitions, no `/usr` change, +no re-signing; existing configs unaffected. Non-goals: changing systemd matching semantics; ESP-based extensions. @@ -130,48 +130,87 @@ identically. Schema gains two optional properties. ## Implementation ``` -/var/trident/sysexts/{a,b}/platform.raw -/var/trident/confexts/{a,b}/ +/var/trident/sysexts/ +├── a/platform.raw +├── b/platform.raw +└── current -> a # flipped at boot ``` -Both directories persist across the update, which is what makes rollback a -symlink flip rather than a re-download. +Both slot directories persist across an update, so a rollback needs no +re-download. Confexts use a sibling `confexts/` under the same root. + +### Activation is resolved at boot, not at servicing + +`/etc` is shared between volumes, so an activating symlink written during +servicing would still point at the new slot after a rollback. This cannot be +corrected by an inverse servicing step, because **the A/B rollback path does not +invoke subsystems**: + +- Automatic rollback is firmware-driven. `rollback::validate_boot` only observes + which root actually booted and reconciles Host Status (`spec = spec_old`). +- `manual_rollback::stage_rollback` touches only the pcrlock policy for + `ManualRollbackAbStaged`. Only the *runtime* path calls + `runtime_update::rollback(&mut subsystems, ..)`. + +Activation must therefore key off what booted, not what was staged: + +1. `/etc/extensions/.raw` → `/current/.raw`. Written once at + clean install, never rewritten. +2. `/current` → `a` | `b`, flipped at boot by a Trident unit from the + booted volume. + +One atomic symlink flip activates the whole set, and it is correct after a +rollback Trident never participated in. + +### Servicing flow The extensions subsystem already runs on `CleanInstall` and `AbUpdate` via `provision()`, documented as "migrate state from A-partition to B-partition (or vice versa)". 1. **prepare** — download, verify SHA-384. Unchanged. -2. **provision** — resolve target volume via +2. **provision** — resolve the target volume via `EngineContext::get_ab_update_volume()` (already used for per-slot verity addons); place into `//`. -3. **activate** — write `/etc/extensions/.raw` → - `//.raw`. `/etc` writes are already supported via - `Subsystem::writable_etc_overlay()`. +3. **configure**, clean install only — write the `/etc/extensions` symlinks. + `/etc` writes are already supported via `Subsystem::writable_etc_overlay()`. + +The symlink supplies the `extension-release.NAME`-matching filename; stored +files may be named freely. + +### Boot unit -The symlink supplies the `extension-release.NAME`-matching filename; the stored -file may be named freely. Same indirection distributions already use. +``` +After=local-fs.target # /var mounted, store readable +Before=systemd-sysext.service +``` -**Shared `/etc`.** The symlink is not slotted, so between servicing and reboot -the running OS points at an extension whose `VERSION_ID` does not match it. This -is inert: already-merged extensions stay merged, the mismatched one is not -merged. +`systemd-sysext.service` is `After=local-fs.target Before=sysinit.target`, so +this window exists. It is also gated on +`ConditionDirectoryNotEmpty=|/etc/extensions`, which the install-time symlinks +satisfy. -**Rollback** repoints symlinks at the previous volume's directory. Inverse of -step 3. +This cannot be `trident.service`, which is `WantedBy=multi-user.target` and runs +long after the merge. -**Ordering.** Symlinks are written before reboot, so they are correct before -`systemd-sysext.service` runs at `Before=sysinit.target`. +Slot discovery: compare the booted `/usr` (or root) PARTUUID against the volume +mapping Trident recorded at install. Not the datastore, which is larger +machinery than this needs. See Open Questions. ## Testing -- Clean install: file in correct volume dir, symlink correct, merged after boot. -- A/B update: symlink repoints and **merged content differs from the previous +- Clean install: file in the correct slot dir, `current` and `/etc/extensions` + symlinks correct, merged after boot. +- A/B update: `current` flips and **merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. -- Rollback: symlinks and merged content return to the previous slot. +- Rollback, both automatic and manual: `current` returns to the previous slot + and the previous content is merged, **with no Trident servicing operation in + between**. This is the case the design exists for. +- Boot ordering: the flip lands before `systemd-sysext.service` merges, asserted + on first boot after an update rather than on a steady-state boot. - Validation: each reject row above. -- Compat: config with no `scope` places byte-identically to today. +- Compat: a config with no `scope` places byte-identically to today. - Non-default `extensionStore` on a separate mount. ## Counter-arguments @@ -187,16 +226,21 @@ step 3. ## Open questions -1. **Does the rollback path invoke subsystem steps?** Largest unknown; gates the - design. -2. **Should Host Status report slotted extensions per volume?** Derivable from - `ab_active_volume` + store, but not inspectable without reading the FS. +1. **Slot discovery in the boot unit.** Comparing the booted PARTUUID against a + small install-time mapping file is proposed, but the mapping must survive the + update that rewrites it. Reading the datastore this early is the alternative. +2. **Does the flip need to be a Trident binary at all?** The operation is one + `readlink` and one `symlink`. A generator or a small shipped script may be a + better fit than adding a Trident invocation to early boot. 3. **Retention.** Nothing prunes a slot directory when an entry is dropped from - the config. Should `provision()` reconcile, and does that break rollback? + the config. Should `provision()` reconcile, and does that break the rollback + guarantee that the previous slot's files are still present? 4. **SELinux.** Extensions are already documented as unsupported with SELinux - enabled; a Trident-created `/etc` symlink may additionally need a label. + enabled; Trident-created symlinks in `/etc` may additionally need a label. 5. **Default path.** `/var/trident/sysexts` vs `/var/lib/trident/sysexts`; the datastore already lives under `/var/lib/trident/`. +6. **Host Status.** Should it report slotted extensions per volume? Derivable + from `ab_active_volume` plus the store, but not otherwise inspectable. ## Future possibilities From f4383d0650e16ac26d3ed1caacdef7494cd2f587 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 12:05:08 -0700 Subject: [PATCH 03/27] docs: express the API surface as Host Configuration YAML The Rust types are an implementation detail; the public surface is the Host Configuration document. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 95 ++++++++----------- 1 file changed, 42 insertions(+), 53 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 8695c3b49a..71a0bb5557 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -40,82 +40,71 @@ Non-goals: changing systemd matching semantics; ESP-based extensions. ### Today -```rust -pub struct Extension { - pub url: Url, - pub sha384: Sha384Hash, - pub path: Option, // absolute, allow-listed dir, *.raw -} +```yaml +os: + sysexts: + - url: https://example.com/tooling.raw # http(s) | file | oci + sha384: + path: /var/lib/extensions/tooling.raw # optional; allow-listed dir, *.raw ``` -`path` conflates *where the file is stored* with *what scope it is active in*. +`path` defaults to `/var/lib/extensions/`, and is restricted to +`/etc/extensions`, `/var/lib/extensions` or `/.extra/sysext`. + +It conflates *where the file is stored* with *what scope it is active in*. Slotting separates them: storage becomes Trident-owned and derived from the target volume. ### Proposed -```rust -pub struct Extension { - pub url: Url, - pub sha384: Sha384Hash, - - /// Absolute path in the target OS. Only valid when `scope` is `shared`. - pub path: Option, - - #[serde(default)] - pub scope: ExtensionScope, -} - -pub enum ExtensionScope { - /// Stored at `path`, active on both volumes. For `ID=_any` extensions. - #[default] - Shared, - /// Stored per-volume in the extension store, active only on the volume it - /// was installed to. For OS-version-pinned extensions. - Slotted, -} -``` - -Plus one optional field on `os`: - -```rust -/// Root of the per-volume extension store. Default `/var/trident/sysexts`. -/// Must be on storage not replaced by an A/B update, and must not be a -/// systemd-sysext search path. -pub extension_store: Option, -``` - -Configurable because slotting doubles storage, and large extensions (GPU -drivers, ML runtimes) may need a dedicated volume. Confexts use a sibling -`confexts/` under the same root. +Two new optional fields. ```yaml os: - extensionStore: /mnt/bigdisk/trident/sysexts # optional + # Root of the per-volume extension store. + # Default /var/trident/sysexts. Must be on storage an A/B update does not + # replace, and must not be a systemd-sysext search path. + extensionStore: /mnt/bigdisk/trident/sysexts + sysexts: - - url: https://example.com/tooling.raw # unchanged, still default + # unchanged, and still the default + - url: https://example.com/tooling.raw sha384: + + # new - url: https://example.com/platform.raw sha384: scope: slotted ``` +`scope`: + +| Value | Meaning | +|---|---| +| `shared` (default) | Stored at `path`, active on both volumes. For `ID=_any` extensions. Today's behaviour. | +| `slotted` | Stored per-volume under `extensionStore`, active only on the volume it was installed to. For OS-version-pinned extensions. | + +`extensionStore` is configurable because slotting doubles storage, and large +extensions (GPU drivers, ML runtimes) may need a dedicated volume. Confexts use +a sibling `confexts/` under the same root. + ### Validation | Condition | Result | |---|---| | `scope` absent | `shared`. Identical to today. | -| `scope: shared` + `path` | Existing allow-list check. | -| `scope: slotted` + `path` | Reject. Path is Trident-owned; accepting it would silently ignore it. | -| `scope: slotted`, no `path` | Filename from `url` basename, as today's default. | -| `scope: slotted`, no A/B volumes | Reject at validation, not at provision. | +| `scope: shared` with `path` | Existing allow-list check. | +| `scope: slotted` with `path` | Reject. The path is Trident-owned; accepting it would silently ignore it. | +| `scope: slotted`, no `path` | Filename from the `url` basename, as today's default. | +| `scope: slotted`, no A/B volumes configured | Reject at validation, not at provision. | | `extensionStore` set, no `slotted` entries | Accept, unused. | -| `extensionStore` inside a sysext search path | Reject: both copies would be discovered. | +| `extensionStore` inside a sysext search path | Reject: both slots' copies would be discovered. | ### Backward compatibility -Both fields are `#[serde(default)]`. Existing configs parse and behave -identically. Schema gains two optional properties. +Both fields default, so existing Host Configurations parse and behave +identically. The schema gains two optional properties and no field changes +meaning. ### Alternatives rejected @@ -123,9 +112,9 @@ identically. Schema gains two optional properties. layout, invites a literal `a`/`b`. - **Separate `os.slottedSysexts` list**: duplicates the type and its duplicate-hash/path validation. -- **Infer from `extension-release`**: metadata is inside the image, unavailable - at static validation, and removes the operator's choice to slot an `_any` - extension deliberately. +- **Infer from `extension-release`**: the metadata is inside the image, so it is + unavailable at static validation, and it removes the operator's choice to slot + an `_any` extension deliberately. ## Implementation From 60697c7be19fa473f263d4e6a341774e9aef2b24 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 17:34:07 -0700 Subject: [PATCH 04/27] docs: rewrite 0838 around the validation change The original premise was wrong. Trident does not place extensions on non-slotted storage: validate_extension_images_locations already rejects any extension directory not on an A/B volume. The real cost is that satisfying it requires a dedicated A/B volume pair, which the reference config pays for with four partitions and 2.4 GB, and which a layout whose only A/B volume is a verity-sealed /usr cannot pay at all. The proposal is now to relax that check: slot-scoping as a property of the extension rather than of the partition table, satisfied by an A/B volume as today or by a Trident-owned per-slot store. Activation no longer needs a boot-time unit. The extensions subsystem gains ManualRollbackAb in runs_on, and automatic rollback is corrected at commit via systemd-sysext refresh; version matching keeps the intervening window absent rather than wrong. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 339 ++++++++++-------- 1 file changed, 190 insertions(+), 149 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 71a0bb5557..097bad9203 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,4 +1,4 @@ -# 0838 Slotted system extensions +# 0838 Slotted Extension Storage - Date: 2026-10-08 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) @@ -6,35 +6,58 @@ ## Summary -Add a `slotted` scope for sysexts/confexts. Trident stores them per A/B volume -under a configurable store (default `/var/trident/sysexts/{a,b}/`) and activates -the correct slot at boot, before `systemd-sysext` merges. +Trident requires every extension destination to sit on an A/B volume when A/B is +configured. That is one way to make an extension follow the slot, and it costs a +partition pair per extension directory. -## Motivation +This adds a second. An extension marked `scope: slotted` is stored per volume +under a Trident-owned store, with a symlink at its normal destination. +Slot-scoping becomes a property of the extension rather than of the partition +table. -`systemd-sysext(8)` merges an extension only if its `extension-release` matches -the running OS: `ID=` must match unless `_any`, then `SYSEXT_LEVEL=` if defined, -else `VERSION_ID=`. +## Motivation and Goals -All three directories Trident accepts today (`/etc/extensions`, -`/var/lib/extensions`, `/.extra/sysext`) are outside the partitions an A/B -update replaces. So an OS-pinned extension silently stops merging after the -first update: no error, no failed unit, the content is just gone. +`validate_extension_images_locations` (`config/host/mod.rs:218`) rejects any +extension directory that does not resolve to a device with A/B capabilities, +raising `ExtensionImageNotOnABVolume`. The reasoning is sound: extensions on +shared storage change the running slot immediately and do not roll back. -Azure Container Linux hits this. Its Azure platform extension (guest agent, -Hyper-V daemons, chrony, `oras`) is `VERSION_ID`-pinned on a non-swapped -partition. Losing `oras` also breaks the distro's extension download path. +The cost is that the only way to satisfy it is a dedicated A/B volume pair. +Trident's own reference configuration +(`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) +spends four partitions and 2.4 GB on it: `sysexts-a`/`sysexts-b` at 1 GB each +mounted at `/var/lib/extensions`, and `confexts-a`/`confexts-b` at 200 MB each +at `/var/lib/confexts`. -The shape that works already exists in the same image: the container runtime -ships as an extension inside `/usr`, so extension and `os-release` swap -together. This proposal gives that property to anyone who cannot re-seal -dm-verity and re-sign the UKI. +That price buys only slot-scoping, and it is charged in the currency hardest to +change: -Goals: correct extension version per booted slot; rollback restores the previous -set without a Trident servicing operation; no new partitions, no `/usr` change, -no re-signing; existing configs unaffected. +- **Partitions are fixed at install.** The 1 GB is a guess made before anyone + knows what the host will carry; growing it means repartitioning. +- **Some layouts cannot pay it.** Where the partition table is published, signed + or attested, adding a pair is not a configuration change. A layout whose only + A/B volume is a verity-sealed `/usr` has none to offer on any terms. +- **It is all-or-nothing per directory.** An extension that should follow the + slot and one that should persist cannot share a location. -Non-goals: changing systemd matching semantics; ESP-based extensions. +The check conflates *being slot-scoped*, the property that matters, with *being +on an A/B block device*, one implementation of it. Trident can provide the same +property with directories, on storage the host already has. + +**Goals.** Slot-scoped extensions without a dedicated volume pair; rollback +restores the previous set; shared and slotted coexist in one directory; no new +partitions, no sealed-`/usr` change, no re-signing; existing configurations +unaffected. + +**Non-goals.** Changing `systemd-sysext` matching semantics, which still decides +what merges. Removing the A/B volume route, which stays better where a pair is +affordable, since the filesystem provides the isolation and Trident does +nothing. Extensions on the ESP. + +## Scope + +Sysexts and confexts under `os.sysexts` and `os.confexts`, on clean install, A/B +update and rollback. Runtime update does not change slot and is unaffected. ## Public API Design @@ -48,23 +71,20 @@ os: path: /var/lib/extensions/tooling.raw # optional; allow-listed dir, *.raw ``` -`path` defaults to `/var/lib/extensions/`, and is restricted to -`/etc/extensions`, `/var/lib/extensions` or `/.extra/sysext`. - -It conflates *where the file is stored* with *what scope it is active in*. -Slotting separates them: storage becomes Trident-owned and derived from the -target volume. +`path` is the destination, defaulting to `/var/lib/extensions/{name}.raw`. When +`storage.abUpdate` is configured it must also resolve to an A/B volume. ### Proposed -Two new optional fields. +Two optional fields. `path` keeps one meaning in both cases: where the extension +appears to systemd. ```yaml os: # Root of the per-volume extension store. # Default /var/trident/sysexts. Must be on storage an A/B update does not # replace, and must not be a systemd-sysext search path. - extensionStore: /mnt/bigdisk/trident/sysexts + extensionStore: /mnt/data/trident/sysexts sysexts: # unchanged, and still the default @@ -77,162 +97,183 @@ os: scope: slotted ``` -`scope`: - -| Value | Meaning | -|---|---| -| `shared` (default) | Stored at `path`, active on both volumes. For `ID=_any` extensions. Today's behaviour. | -| `slotted` | Stored per-volume under `extensionStore`, active only on the volume it was installed to. For OS-version-pinned extensions. | +| `scope` | Where the bytes live | What `path` is | +| --- | --- | --- | +| `shared` (default) | At `path`. | The image itself. Today's behaviour. | +| `slotted` | `/{a,b}/{name}.raw` | A symlink to the booted slot's copy. | `extensionStore` is configurable because slotting doubles storage, and large -extensions (GPU drivers, ML runtimes) may need a dedicated volume. Confexts use -a sibling `confexts/` under the same root. +extensions such as GPU drivers may warrant a dedicated volume. Confexts use a +sibling `confexts/` under the same root. + +### The validation change + +`validate_extension_images_locations` is relaxed, not removed. Its requirement +becomes *the destination is slot-scoped*, satisfied either way: + +| Configuration | Rule | +| --- | --- | +| No `abUpdate` | No check, as today. | +| `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | +| `abUpdate`, `scope: slotted` | `path` need not be; the store provides the scoping. | + +This is the substantive behavioural change. Everything else follows from it. ### Validation | Condition | Result | -|---|---| +| --- | --- | | `scope` absent | `shared`. Identical to today. | -| `scope: shared` with `path` | Existing allow-list check. | -| `scope: slotted` with `path` | Reject. The path is Trident-owned; accepting it would silently ignore it. | -| `scope: slotted`, no `path` | Filename from the `url` basename, as today's default. | -| `scope: slotted`, no A/B volumes configured | Reject at validation, not at provision. | +| `scope: slotted`, no `path` | Default destination, as today. | +| `scope: slotted`, no `abUpdate` | Reject, naming the absent configuration. | +| `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | +| `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | | `extensionStore` set, no `slotted` entries | Accept, unused. | -| `extensionStore` inside a sysext search path | Reject: both slots' copies would be discovered. | ### Backward compatibility Both fields default, so existing Host Configurations parse and behave -identically. The schema gains two optional properties and no field changes -meaning. +identically and no field changes meaning. The relaxed check is strictly more +permissive, so anything valid today stays valid. The schema gains two optional +properties. ### Alternatives rejected - **Sentinel `path`** (`/var/trident/sysexts//x.raw`): leaks internal - layout, invites a literal `a`/`b`. -- **Separate `os.slottedSysexts` list**: duplicates the type and its - duplicate-hash/path validation. -- **Infer from `extension-release`**: the metadata is inside the image, so it is - unavailable at static validation, and it removes the operator's choice to slot - an `_any` extension deliberately. + layout, invites a literal `a` or `b`. +- **Separate `os.slottedSysexts` list**: duplicates the type and its duplicate + hash and path validation. +- **Infer from `extension-release`**: inside the image, so unavailable at static + validation, and it removes the operator's choice to slot an `ID=_any` + extension deliberately. ## Implementation ``` /var/trident/sysexts/ ├── a/platform.raw -├── b/platform.raw -└── current -> a # flipped at boot -``` - -Both slot directories persist across an update, so a rollback needs no -re-download. Confexts use a sibling `confexts/` under the same root. - -### Activation is resolved at boot, not at servicing +└── b/platform.raw -`/etc` is shared between volumes, so an activating symlink written during -servicing would still point at the new slot after a rollback. This cannot be -corrected by an inverse servicing step, because **the A/B rollback path does not -invoke subsystems**: - -- Automatic rollback is firmware-driven. `rollback::validate_boot` only observes - which root actually booted and reconciles Host Status (`spec = spec_old`). -- `manual_rollback::stage_rollback` touches only the pcrlock policy for - `ManualRollbackAbStaged`. Only the *runtime* path calls - `runtime_update::rollback(&mut subsystems, ..)`. - -Activation must therefore key off what booted, not what was staged: +/var/lib/extensions/ +├── platform.raw -> /var/trident/sysexts/a/platform.raw # slotted +└── tooling.raw # shared, unchanged +``` -1. `/etc/extensions/.raw` → `/current/.raw`. Written once at - clean install, never rewritten. -2. `/current` → `a` | `b`, flipped at boot by a Trident unit from the - booted volume. +The store sits outside every search path, so only the symlink is discoverable +and exactly one copy is ever a candidate. Both slot directories persist across +an update, so rollback needs no re-download. -One atomic symlink flip activates the whole set, and it is correct after a -rollback Trident never participated in. +Symlinks are followed: `image_discover` opens each search path with +`chase_and_opendir(path, root, CHASE_PREFIX_ROOT, ..)` and stats entries with +`flags = 0` on a running system, with an upstream comment explicitly permitting +symlinks into the search path (systemd `src/shared/discover-image.c`). -### Servicing flow +### Servicing -The extensions subsystem already runs on `CleanInstall` and `AbUpdate` via +The extensions subsystem already runs on clean install and A/B update via `provision()`, documented as "migrate state from A-partition to B-partition (or -vice versa)". - -1. **prepare** — download, verify SHA-384. Unchanged. -2. **provision** — resolve the target volume via - `EngineContext::get_ab_update_volume()` (already used for per-slot verity - addons); place into `//`. -3. **configure**, clean install only — write the `/etc/extensions` symlinks. - `/etc` writes are already supported via `Subsystem::writable_etc_overlay()`. - -The symlink supplies the `extension-release.NAME`-matching filename; stored -files may be named freely. - -### Boot unit - -``` -After=local-fs.target # /var mounted, store readable -Before=systemd-sysext.service -``` - -`systemd-sysext.service` is `After=local-fs.target Before=sysinit.target`, so -this window exists. It is also gated on -`ConditionDirectoryNotEmpty=|/etc/extensions`, which the install-time symlinks -satisfy. - -This cannot be `trident.service`, which is `WantedBy=multi-user.target` and runs -long after the merge. - -Slot discovery: compare the booted `/usr` (or root) PARTUUID against the volume -mapping Trident recorded at install. Not the datastore, which is larger -machinery than this needs. See Open Questions. +vice versa)". For a slotted entry it resolves the target volume with +`EngineContext::get_ab_update_volume()`, already used to select the per-slot +verity addon, writes into `//`, and points the destination +symlink at it. + +### Rollback + +`RUNS_ON_ALL` is `CleanInstall | AbUpdate | RuntimeUpdate` (`engine/mod.rs:63`); +it excludes both rollback types despite the name. The extensions subsystem adds +`ManualRollbackAb` to its `runs_on` and repoints the symlinks at the previous +volume's directory. The files were never removed, so this is a symlink operation +rather than a re-download. `finalize_rollback` already locks `SUBSYSTEMS` for +the runtime path. + +**Automatic rollback is not a servicing operation.** When the firmware falls +back on its own, Trident regains control only at `trident commit`, which is +`WantedBy=multi-user.target` and so runs after `systemd-sysext` merges at +`sysinit.target`. + +The outcome is still safe, because the matching rules that motivate this RFC act +as an interlock. The stale symlink resolves to the other slot's extension, which +declares the other OS version, so systemd declines to merge it. The host comes +up with the extension **absent, not wrong**. `commit` then detects the rollback +through the existing `validate_boot` path, repoints the symlinks and runs +`systemd-sysext refresh`, which is the unit's own `ExecStart` and needs no +reboot. + +Degraded for part of one boot, never serving content from the wrong OS version, +and self-closing. Where that window is unacceptable, a +[health check](../../Reference/Host-Configuration/API-Reference/Health.md) +asserting the extension is merged gates the update before it reaches this state. + +## Relationship to RFC 0789 + +[0789](https://github.com/microsoft/trident/pull/789) carries extension images in +COSI files. The two are orthogonal: 0789 changes where the bytes come from, this +changes where they land and how they follow the slot. Either is useful alone, +and a bundled extension can be slotted. + +Both touch `validate_extension_images_locations`, in opposite directions: 0789 +tightens it into dynamic validation against the storage graph, this adds a +second way to satisfy it. Whichever lands first should leave the check expressed +as *is this destination slot-scoped*, so the other is an added arm rather than a +rewrite. 0789's rollback argument, that extensions revert because the slot's +filesystem reverts, is the A/B volume mechanism described here and holds +unchanged for `scope: shared`. ## Testing -- Clean install: file in the correct slot dir, `current` and `/etc/extensions` - symlinks correct, merged after boot. -- A/B update: `current` flips and **merged content differs from the previous - slot**. A presence-only assertion passes in the inert case and must not be - used. -- Rollback, both automatic and manual: `current` returns to the previous slot - and the previous content is merged, **with no Trident servicing operation in - between**. This is the case the design exists for. -- Boot ordering: the flip lands before `systemd-sysext.service` merges, asserted - on first boot after an update rather than on a steady-state boot. -- Validation: each reject row above. -- Compat: a config with no `scope` places byte-identically to today. -- Non-default `extensionStore` on a separate mount. +- Clean install: file in the correct slot directory, symlink resolving, merged + after boot. +- A/B update: the symlink repoints and **the merged content differs from the + previous slot**. A presence-only assertion passes in the inert case and must + not be used. +- Manual rollback: symlinks and merged content return to the previous slot. +- Automatic rollback: on the fallback boot the extension is absent rather than + stale, and merges after `commit` without a reboot. Both halves asserted; the + first distinguishes safe degradation from serving wrong content. +- Coexistence: a shared and a slotted extension in `/var/lib/extensions`, both + merged, only the slotted one changing across an update. +- Validation: each reject row, plus a shared extension off an A/B volume still + rejected. +- Compatibility: a configuration with no `scope` places byte-identically to + today. A non-default `extensionStore` on a separate mount. ## Counter-arguments -- **"Put it in `/usr`."** Strictly better where possible, but requires re-sealing - verity and re-signing the UKI. This targets that gap. -- **"Use the ESP."** `.efi.extra.d/*.sysext.raw` is genuinely slot-scoped, - but ESPs are 128-256 MiB against payloads in the tens to hundreds of MB, and - `systemd-stub` places them in the initrd, not the running system. -- **"Add A/B extension partitions."** Works, but layout changes are breaking - after the first A/B image ships. Directories achieve the same isolation. -- **"Doubles storage."** Yes. Mitigated by `extensionStore`. +- **"Just add the partitions."** Better where possible. This is for layouts that + cannot, and for sizing decided after install rather than before. +- **"Put it in `/usr`."** Better still, and unavailable without re-sealing + dm-verity and re-signing the UKI. Upstream also forbids extension storage + under `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. +- **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs + are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and + `systemd-stub` places them in the initrd. +- **"Doubles storage."** It does. The A/B volume route doubles it too, with the + allocation fixed at install. +- **"A symlink is weaker than a partition boundary."** Yes, though both are + reachable by anything that can already write the extension image. + +**Prior art.** systemd's versioned directories (`.v/` holding +`_.raw`, highest wins) are the same shape: a directory of +candidates, one selected at discovery. They select by version rather than slot, +so they do not serve this case, but the pattern is systemd's own. ## Open questions -1. **Slot discovery in the boot unit.** Comparing the booted PARTUUID against a - small install-time mapping file is proposed, but the mapping must survive the - update that rewrites it. Reading the datastore this early is the alternative. -2. **Does the flip need to be a Trident binary at all?** The operation is one - `readlink` and one `symlink`. A generator or a small shipped script may be a - better fit than adding a Trident invocation to early boot. -3. **Retention.** Nothing prunes a slot directory when an entry is dropped from - the config. Should `provision()` reconcile, and does that break the rollback - guarantee that the previous slot's files are still present? -4. **SELinux.** Extensions are already documented as unsupported with SELinux - enabled; Trident-created symlinks in `/etc` may additionally need a label. -5. **Default path.** `/var/trident/sysexts` vs `/var/lib/trident/sysexts`; the - datastore already lives under `/var/lib/trident/`. -6. **Host Status.** Should it report slotted extensions per volume? Derivable - from `ab_active_volume` plus the store, but not otherwise inspectable. +1. **Retention.** Nothing prunes a slot directory when an entry leaves the + configuration. Should `provision()` reconcile, and does that conflict with + keeping the previous slot's files for rollback? +2. **Default path.** `/var/trident/sysexts` against `/var/lib/trident/sysexts`; + the datastore already lives under `/var/lib/trident/` (`constants.rs:30`). +3. **Should `commit` refresh unconditionally?** Refreshing only on a detected + rollback is narrower, but an unconditional refresh is cheap and removes a + branch. +4. **Host Status.** Should it report the slotted set per volume? +5. **SELinux.** Extensions are already unsupported with SELinux enforcing. + Slotting does not change that, but Trident-created symlinks may need + labelling, and 0789 proposes moving that check to every servicing type. ## Future possibilities - Per-volume scoping for other Trident-managed state that is shared today. -- Changing an extension's scope between servicing operations, which currently - strands a copy in the old location. +- Changing an extension's scope between operations, which would otherwise strand + a copy in the old location. From 8a86bd3d484a8c96378d1ee42665401841e177a2 Mon Sep 17 00:00:00 2001 From: Paco Date: Mon, 12 Oct 2026 15:40:09 -0700 Subject: [PATCH 05/27] docs: rewrite 0838 around boot-time selection Replaces the previous mechanism. Symlinks are no longer written by the extensions subsystem into /var/lib/extensions during servicing; selection is recomputed every boot into /run/extensions from the slot that actually booted. That removes the ManualRollbackAb runs_on change and the commit-time repair, since nothing persists a decision that a rollback could invalidate. Corrections and additions from review: - Before=systemd-sysext.service is the ordering edge. ensure-sysext.service carries BindsTo= and After=systemd-sysext.service, so it runs after the merge and is the wrong anchor. - WantedBy= plus Before= does not fail closed; the merge unit needs a Requires= drop-in. - Per-file symlinking cannot commit a set atomically. - A store manifest replaces globbing, so absence is distinguishable from loss. - Two-phase rollout, since a rollback slot may predate the loader. - Documents why content matching alone is insufficient, and scopes slotted extensions to /usr and /opt because reordering the merge ahead of an /etc overlay would exhaust the kernel filesystem stacking budget. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 407 +++++++++++++----- 1 file changed, 290 insertions(+), 117 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 097bad9203..29951c3383 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,6 +1,6 @@ # 0838 Slotted Extension Storage -- Date: 2026-10-08 +- Date: 2026-10-12 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) - Issue: TBD @@ -11,9 +11,9 @@ configured. That is one way to make an extension follow the slot, and it costs a partition pair per extension directory. This adds a second. An extension marked `scope: slotted` is stored per volume -under a Trident-owned store, with a symlink at its normal destination. -Slot-scoping becomes a property of the extension rather than of the partition -table. +under a Trident-owned store. A small boot-time component selects the booted +slot's copy into `/run/extensions` before the merge runs. Slot-scoping becomes a +property of the extension rather than of the partition table. ## Motivation and Goals @@ -25,14 +25,12 @@ shared storage change the running slot immediately and do not roll back. The cost is that the only way to satisfy it is a dedicated A/B volume pair. Trident's own reference configuration (`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on it: `sysexts-a`/`sysexts-b` at 1 GB each -mounted at `/var/lib/extensions`, and `confexts-a`/`confexts-b` at 200 MB each -at `/var/lib/confexts`. +spends four partitions and 2.4 GB on it. That price buys only slot-scoping, and it is charged in the currency hardest to change: -- **Partitions are fixed at install.** The 1 GB is a guess made before anyone +- **Partitions are fixed at install.** The size is a guess made before anyone knows what the host will carry; growing it means repartitioning. - **Some layouts cannot pay it.** Where the partition table is published, signed or attested, adding a pair is not a configuration change. A layout whose only @@ -41,23 +39,64 @@ change: slot and one that should persist cannot share a location. The check conflates *being slot-scoped*, the property that matters, with *being -on an A/B block device*, one implementation of it. Trident can provide the same -property with directories, on storage the host already has. +on an A/B block device*, one implementation of it. -**Goals.** Slot-scoped extensions without a dedicated volume pair; rollback -restores the previous set; shared and slotted coexist in one directory; no new -partitions, no sealed-`/usr` change, no re-signing; existing configurations -unaffected. +### Why content matching alone is not sufficient -**Non-goals.** Changing `systemd-sysext` matching semantics, which still decides -what merges. Removing the A/B volume route, which stays better where a pair is -affordable, since the filesystem provides the isolation and Trident does -nothing. Extensions on the ESP. +The obvious alternative is to place both slots' copies in one search directory +and let `systemd-sysext` merge whichever matches the booted OS. Version +mismatches are skipped quietly, so this very nearly works, and on a suitable +platform it would need no new code at all. + +It fails for two independent reasons, either of which is sufficient: + +1. **Deploying one image under two filenames requires systemd v256.** + `path_extract_image_name()` truncates an image name at the first `_` or `+` + (`src/basic/os-util.c`), so `platform_1.raw` and `platform_2.raw` can share a + single inner `extension-release.platform`. That function does not exist + before v256; it was added by commit `37543971` (2024-03-28). On earlier + systemd each copy needs its own `extension-release`, so a signed image must + be repacked per slot. The `user.extension-release.strict` xattr relaxes the + name correspondence but must be set inside the image at build time. +2. **The OS version must actually differ between slots.** Matching uses `ID=`, + then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A + platform whose `VERSION_ID` is coarser than its build cadence, or which can + deploy the same image to both slots, puts identical values in both slots and + both copies then match. + +Both conditions hold on the first intended consumer. Content matching remains +the better mechanism where a platform satisfies both, and this design does not +preclude adopting it there. + +### Goals + +Slot-scoped extensions without a dedicated volume pair; rollback restores the +previous set with no servicing operation; no new partitions, no sealed-`/usr` +change, no re-signing; existing configurations unaffected. + +### Non-goals + +Changing `systemd-sysext` matching semantics. Removing the A/B volume route, +which stays better where a pair is affordable, since the filesystem provides +the isolation and Trident does nothing. Extensions on the ESP. ## Scope -Sysexts and confexts under `os.sysexts` and `os.confexts`, on clean install, A/B -update and rollback. Runtime update does not change slot and is unaffected. +Sysexts and confexts declared under `os.sysexts` and `os.confexts`, on clean +install and A/B update. Runtime update does not change slot and is unaffected. + +**Slotted extensions carry `/usr` and `/opt` content only.** Contributing to +`/etc` is out of scope for this RFC. On a platform that assembles `/etc` as an +overlay whose lowerdir lives inside `/usr`, a sysext can only affect `/etc` if +it is merged before that overlay is built. Reordering the merge ahead of it +would place the `/etc` overlay on top of the `/usr` overlay, consuming the +kernel's entire filesystem stacking budget: `FILESYSTEM_MAX_STACK_DEPTH` is 2 +(`include/linux/fs.h`), enforced in `fs/overlayfs/super.c`, and an overlay's +depth is the maximum of its layers plus one. The `/etc` overlay would reach +depth 2 and any confext merged onto `/etc` afterwards would reach 3 and fail. + +The durable fix is to stop sourcing `/etc` defaults from inside `/usr`, which is +a platform change rather than a Trident one. ## Public API Design @@ -76,14 +115,12 @@ os: ### Proposed -Two optional fields. `path` keeps one meaning in both cases: where the extension -appears to systemd. - ```yaml os: # Root of the per-volume extension store. # Default /var/trident/sysexts. Must be on storage an A/B update does not - # replace, and must not be a systemd-sysext search path. + # replace, must not be a systemd-sysext search path, and must be on a + # filesystem available at local-fs.target. extensionStore: /mnt/data/trident/sysexts sysexts: @@ -97,14 +134,13 @@ os: scope: slotted ``` -| `scope` | Where the bytes live | What `path` is | +| `scope` | Where the bytes live | Where it is activated | | --- | --- | --- | -| `shared` (default) | At `path`. | The image itself. Today's behaviour. | -| `slotted` | `/{a,b}/{name}.raw` | A symlink to the booted slot's copy. | +| `shared` (default) | At `path`. | At `path`. Today's behaviour. | +| `slotted` | `/{a,b}/{name}.raw` | `/run/extensions/`, by the loader | -`extensionStore` is configurable because slotting doubles storage, and large -extensions such as GPU drivers may warrant a dedicated volume. Confexts use a -sibling `confexts/` under the same root. +A slotted extension always surfaces in `/run/extensions`, so `path` does not +apply to it and is rejected. ### The validation change @@ -115,34 +151,36 @@ becomes *the destination is slot-scoped*, satisfied either way: | --- | --- | | No `abUpdate` | No check, as today. | | `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | -| `abUpdate`, `scope: slotted` | `path` need not be; the store provides the scoping. | +| `abUpdate`, `scope: slotted` | `path` need not be; the store plus the loader provide the scoping. | -This is the substantive behavioural change. Everything else follows from it. +This is the substantive behavioural change. It is only safe where the platform +actually provides a loader, so validation must check a **platform capability**, +not merely that A/B is configured. See [Rollout](#rollout). ### Validation | Condition | Result | | --- | --- | | `scope` absent | `shared`. Identical to today. | -| `scope: slotted`, no `path` | Default destination, as today. | +| `scope: slotted` with `path` | Reject. The destination is loader-owned. | | `scope: slotted`, no `abUpdate` | Reject, naming the absent configuration. | +| `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | | `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | | `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| `extensionStore` set, no `slotted` entries | Accept, unused. | +| `extensionStore` not available at `local-fs.target` | Reject: the loader runs before anything later. | +| Two slotted entries resolving to the same basename | Reject: they would collide in `/run/extensions`. | ### Backward compatibility Both fields default, so existing Host Configurations parse and behave -identically and no field changes meaning. The relaxed check is strictly more -permissive, so anything valid today stays valid. The schema gains two optional -properties. +identically. The relaxed check is strictly more permissive, so anything valid +today stays valid. The schema gains two optional properties. ### Alternatives rejected -- **Sentinel `path`** (`/var/trident/sysexts//x.raw`): leaks internal - layout, invites a literal `a` or `b`. -- **Separate `os.slottedSysexts` list**: duplicates the type and its duplicate - hash and path validation. +- **Sentinel `path`** (`//x.raw`): leaks internal layout, invites a + literal `a` or `b`. +- **Separate `os.slottedSysexts` list**: duplicates the type and its validation. - **Infer from `extension-release`**: inside the image, so unavailable at static validation, and it removes the operator's choice to slot an `ID=_any` extension deliberately. @@ -154,126 +192,261 @@ properties. ├── a/platform.raw └── b/platform.raw -/var/lib/extensions/ -├── platform.raw -> /var/trident/sysexts/a/platform.raw # slotted -└── tooling.raw # shared, unchanged +/run/extensions/ +└── platform.raw -> /var/trident/sysexts/a/platform.raw # recreated each boot ``` -The store sits outside every search path, so only the symlink is discoverable -and exactly one copy is ever a candidate. Both slot directories persist across -an update, so rollback needs no re-download. +The store is outside every search path, so its contents are invisible to +`systemd-sysext` until selected. Both slot directories persist across an update, +so rollback needs no re-download. Symlinks are followed: `image_discover` opens each search path with `chase_and_opendir(path, root, CHASE_PREFIX_ROOT, ..)` and stats entries with `flags = 0` on a running system, with an upstream comment explicitly permitting -symlinks into the search path (systemd `src/shared/discover-image.c`). +symlinks into the search path (`src/shared/discover-image.c`). -### Servicing +### Servicing writes the store, and must write it atomically The extensions subsystem already runs on clean install and A/B update via `provision()`, documented as "migrate state from A-partition to B-partition (or vice versa)". For a slotted entry it resolves the target volume with `EngineContext::get_ab_update_volume()`, already used to select the per-slot -verity addon, writes into `//`, and points the destination -symlink at it. - -### Rollback - -`RUNS_ON_ALL` is `CleanInstall | AbUpdate | RuntimeUpdate` (`engine/mod.rs:63`); -it excludes both rollback types despite the name. The extensions subsystem adds -`ManualRollbackAb` to its `runs_on` and repoints the symlinks at the previous -volume's directory. The files were never removed, so this is a symlink operation -rather than a re-download. `finalize_rollback` already locks `SUBSYSTEMS` for -the runtime path. - -**Automatic rollback is not a servicing operation.** When the firmware falls -back on its own, Trident regains control only at `trident commit`, which is -`WantedBy=multi-user.target` and so runs after `systemd-sysext` merges at -`sysinit.target`. - -The outcome is still safe, because the matching rules that motivate this RFC act -as an interlock. The stale symlink resolves to the other slot's extension, which -declares the other OS version, so systemd declines to merge it. The host comes -up with the extension **absent, not wrong**. `commit` then detects the rollback -through the existing `validate_boot` path, repoints the symlinks and runs -`systemd-sysext refresh`, which is the unit's own `ExecStart` and needs no -reboot. - -Degraded for part of one boot, never serving content from the wrong OS version, -and self-closing. Where that window is unacceptable, a -[health check](../../Reference/Host-Configuration/API-Reference/Health.md) -asserting the extension is merged gates the update before it reaches this state. +verity addon, and writes into `//`. + +Placement must be atomic. A malformed `.raw` is not skipped like a version +mismatch: `image_read_metadata` errors propagate in +`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image +tolerance applies, so **one bad image blocks every extension on the host**. +Directory-type images are tolerated; `.raw` are not. + +Trident today falls back to a non-atomic `fs::copy` when `rename` fails across a +filesystem boundary (`crates/trident/src/subsystems/extensions/mod.rs:525-540`). +Staging must move to a temporary file on the destination's own filesystem, +verified and `fsync`ed, then renamed. This is a prerequisite, not a +consequence. + +### Selection happens at boot + +Selection is recomputed on every boot from the slot that actually booted. This +is what makes unsupervised rollback correct: when firmware falls back on its +own, no Trident code runs, so any decision written during servicing would be +stale with nothing to correct it. + +`/run` is tmpfs, so a stale pointer cannot be represented across a boot. + +The loader resolves the booted slot, then exposes the matching slot directory's +contents in `/run/extensions`. Its unit: + +```ini +[Unit] +Description=Select Trident system extensions for the booted slot +DefaultDependencies=no +After=local-fs.target +Before=systemd-sysext.service +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/bin/trident-sysext-loader +[Install] +WantedBy=sysinit.target +``` + +`DefaultDependencies=no` is required; the implicit `After=sysinit.target` would +place it after the merge and create a cycle. + +`Before=systemd-sysext.service` is the correct edge even on platforms that do +not enable that unit directly. Flatcar-derived systems wire +`ensure-sysext.service` into `sysinit.target.wants` instead, but that unit +carries `BindsTo=systemd-sysext.service` and `After=systemd-sysext.service`: it +runs *after* the merge, and exists to `daemon-reload` and restart targets so +that units shipped inside an extension are picked up. Ordering before it would +therefore permit running after the merge. + +This works because systemd evaluates conditions in `unit_start()` +(`src/core/unit.c`), not at transaction creation, so the loader populates +`/run/extensions` before the merge unit's own `ConditionDirectoryNotEmpty=` is +tested. The same fact rules out an `ExecStartPre=` drop-in, since that condition +gates the whole unit and would skip it while the directories are still empty. + +### Failing closed + +`WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states has no effect on +the transaction when it fails, and `Before=` carries no requirement. The unit +above therefore does **not** fail closed on its own: a loader failure would +leave boot to continue with extensions absent or partially present. + +The merge unit must carry `Requires=trident-sysext-loader.service` via a +drop-in. This must be a deliberate requirement edge rather than forcing +emergency mode, because the store is on shared storage and a storage fault must +not be able to fail both slots and defeat A/B rollback. + +### Selection is a set, not a sequence + +Creating N symlinks exposes a prefix of the intended set if it fails partway, +which matters where a driver and its userspace libraries must move together. +The loader must therefore commit the set atomically: prepare the complete set +in a staging directory under `/run` and rename it into place, or select the +whole directory at once. + +Selecting the whole directory is simpler but means everything at that search +path comes from one slot, so a shared extension can no longer sit beside a +slotted one there. That trade should be settled before implementation. + +### Slot detection + +The loader resolves the slot per platform, selected by a `LOADER_MODE` +environment variable so that Trident does not hardcode one platform's +mechanism. The first mode reads a slot label from the kernel command line, +supplied by a per-slot signed boot artefact. + +A label is preferable to a partition UUID because no mapping table has to stay +in sync with a layout that changes. However, a label is an *assertion*: a +packaging error could pair it with the other slot's device. The loader should +cross-check it against the booted `/usr` backing device and refuse to proceed on +a mismatch. + +Platform-specific conditions belong in platform-specific units. A single unit +carrying `ConditionKernelCommandLine=` for one platform's parameter would +prevent every other mode from ever running, since the condition is evaluated +before the binary reads `LOADER_MODE`. + +### The store needs a manifest + +Globbing `*.raw` cannot distinguish "no extensions configured" from "storage +lost", "mount failed" or "update interrupted", and treating all of those as +success produces a quietly extension-less host. + +Each slot directory should carry a manifest naming the expected images and their +hashes. An absent manifest means the feature is not configured; a present +manifest with missing or unexpected content is an error. This also gives +garbage collection something to act on, which globbing does not. + +## Rollout + +The loader ships in `/usr` and therefore rides the A/B swap. A rollback into a +slot whose `/usr` predates the loader has no selector, and every slotted +extension is absent for that boot. + +Enabling `scope: slotted` therefore requires two phases: first deploy a release +carrying the loader and let it reach both slots, then permit slotted +configuration. Validation must verify that capability rather than assume it, +which is why the relaxed check above tests for a loader and not merely for A/B. ## Relationship to RFC 0789 -[0789](https://github.com/microsoft/trident/pull/789) carries extension images in -COSI files. The two are orthogonal: 0789 changes where the bytes come from, this -changes where they land and how they follow the slot. Either is useful alone, -and a bundled extension can be slotted. +[0789](https://github.com/microsoft/trident/pull/789) carries extension images +in COSI files. The two are orthogonal: 0789 changes where the bytes come from, +this changes where they land and how they follow the slot. Either is useful +alone, and a bundled extension can be slotted. Both touch `validate_extension_images_locations`, in opposite directions: 0789 tightens it into dynamic validation against the storage graph, this adds a second way to satisfy it. Whichever lands first should leave the check expressed as *is this destination slot-scoped*, so the other is an added arm rather than a rewrite. 0789's rollback argument, that extensions revert because the slot's -filesystem reverts, is the A/B volume mechanism described here and holds -unchanged for `scope: shared`. +filesystem reverts, is the A/B volume mechanism and holds unchanged for +`scope: shared`. ## Testing -- Clean install: file in the correct slot directory, symlink resolving, merged - after boot. -- A/B update: the symlink repoints and **the merged content differs from the +- Clean install: image in the correct slot directory, `/run/extensions` link + resolving, merged after boot. +- A/B update: the selection repoints and **the merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. -- Manual rollback: symlinks and merged content return to the previous slot. -- Automatic rollback: on the fallback boot the extension is absent rather than - stale, and merges after `commit` without a reboot. Both halves asserted; the - first distinguishes safe degradation from serving wrong content. -- Coexistence: a shared and a slotted extension in `/var/lib/extensions`, both - merged, only the slotted one changing across an update. +- **Unsupervised rollback: with no Trident servicing operation in between**, the + previous slot's content is merged again. This is the case the design exists + for. +- Loader failure: boot does not proceed silently with a partial set. +- Ordering: the selection lands before the merge, asserted on the first boot + after an update rather than on a steady-state boot. +- A malformed image in a slot directory is rejected before it reaches + `/run/extensions`. - Validation: each reject row, plus a shared extension off an A/B volume still rejected. - Compatibility: a configuration with no `scope` places byte-identically to - today. A non-default `extensionStore` on a separate mount. + today. ## Counter-arguments -- **"Just add the partitions."** Better where possible. This is for layouts that - cannot, and for sizing decided after install rather than before. +- **"Just add the partitions."** Better where possible: the filesystem provides + the isolation, Trident does nothing at boot, and capacity is contained. This + is for layouts that cannot. Directory slotting gives equivalent naming, not + equivalent failure containment: a store on shared storage can fill and affect + the running slot, which a dedicated pair cannot. +- **"Do it in the initrd."** Flatcar does exactly this and it is the stronger + position in principle, since the merge then precedes unit enumeration and the + `/etc` overlay. It requires rebuilding and re-signing the UKI, which is a + delivery cost rather than a correctness argument, and it does not fit a + platform that cannot modify its initrd. Worth revisiting for extensions needed + before `sysinit.target`, which this design cannot serve. - **"Put it in `/usr`."** Better still, and unavailable without re-sealing - dm-verity and re-signing the UKI. Upstream also forbids extension storage - under `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. + dm-verity and re-signing. Upstream also forbids extension storage under + `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. - **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and `systemd-stub` places them in the initrd. - **"Doubles storage."** It does. The A/B volume route doubles it too, with the allocation fixed at install. -- **"A symlink is weaker than a partition boundary."** Yes, though both are - reachable by anything that can already write the extension image. -**Prior art.** systemd's versioned directories (`.v/` holding -`_.raw`, highest wins) are the same shape: a directory of -candidates, one selected at discovery. They select by version rather than slot, -so they do not serve this case, but the pattern is systemd's own. +### Rejected mechanisms + +- **`.v` versioned directories.** They select the highest version and stop, with + no fallback when that image fails OS matching, so they cannot hold per-slot + copies. Verified empirically; Flatcar reached the same conclusion. +- **`systemd-tmpfiles`.** No specifier identifies the booted device, partition + or slot, symlink targets are literal text plus specifier expansion, and the + main pass is ordered after the merge. +- **A systemd generator.** Generators run sandboxed with most of the filesystem + read-only, and `/var` is not mounted when they run. +- **Repair after boot.** Any agent running at `multi-user.target` is far too + late to influence the merge. + +### Prior art + +Three shipping systems converge on recomputing the selection at boot rather than +persisting it: + +- **Flatcar**, whose layout is closest to the first consumer, stamps the version + into the filename on shared storage and rewrites the activating symlink in the + initrd from the booted `/usr`'s `os-release`. +- **ChromeOS** stores `dlc_a`/`dlc_b` on shared stateful storage and derives the + slot from the booted partition number on every service start. It also clears + the inactive slot's validity marker *before* writing and re-sets it after, so + a rollback into a half-written slot finds an honest failure rather than a + plausible file. That is worth adopting. +- **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` + and symlinks the winner into `/run/extensions`. + +The counter-example is instructive: Fedora CoreOS activates sysexts statically +across deployments, and its documentation records that after a major rebase the +extensions silently stop matching and are not loaded. ## Open questions -1. **Retention.** Nothing prunes a slot directory when an entry leaves the - configuration. Should `provision()` reconcile, and does that conflict with - keeping the previous slot's files for rollback? -2. **Default path.** `/var/trident/sysexts` against `/var/lib/trident/sysexts`; - the datastore already lives under `/var/lib/trident/` (`constants.rs:30`). -3. **Should `commit` refresh unconditionally?** Refreshing only on a detected - rollback is narrower, but an unconditional refresh is cheap and removes a - branch. -4. **Host Status.** Should it report the slotted set per volume? -5. **SELinux.** Extensions are already unsupported with SELinux enforcing. - Slotting does not change that, but Trident-created symlinks may need +1. **Whole-directory selection or a prepared set?** Both commit atomically. The + former is simpler; the latter allows shared and slotted extensions to share a + search path. +2. **Manifest format and whether it is Trident-internal or part of the API.** +3. **Retention.** Nothing prunes a slot directory when an entry leaves the + configuration. Reconciling in `provision()` conflicts with keeping the + previous slot's files for rollback. +4. **Default store path.** `/var/trident/sysexts` against + `/var/lib/trident/sysexts`; the datastore already lives under the latter. +5. **Confexts.** The same mechanism applies with different search paths, but + `/etc` content is out of scope here, so whether confexts are in scope at all + needs deciding. +6. **SELinux.** Extensions are already unsupported with SELinux enforcing. + Slotting does not change that, but loader-created symlinks in `/run` need labelling, and 0789 proposes moving that check to every servicing type. +7. **Measured boot.** The selected image is outside the UKI's measurements, so + identical measurements can yield different `/usr` contents. Whether that is + acceptable depends on the platform's attestation claims. ## Future possibilities +- **Initrd-phase selection**, for extensions that must be merged before unit + enumeration or before an `/etc` overlay is assembled. +- **Content matching** on platforms with systemd v256 or later and a + build-granular `VERSION_ID`, which needs no loader at all. - Per-volume scoping for other Trident-managed state that is shared today. -- Changing an extension's scope between operations, which would otherwise strand - a copy in the old location. From 2d58c9f333c7da6096a99398f511f458b159262c Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 13 Oct 2026 16:15:22 -0700 Subject: [PATCH 06/27] docs: unify sysext and /etc selection in one initrd component Both halves resolve the booted slot and expose that slot's content before a consumer reads it, and both must act before the /etc overlay is mounted in the initrd. /run is bind-moved across switch-root, so the sysext symlinks written there remain valid for the later merge in the real root. Resolving the slot once means the two halves cannot disagree about which slot they came from, and it removes the unit-ordering, condition and fail-closed problems a real-root systemd unit carries. /etc payloads are confext-format images mounted as additional lowerdirs on the existing overlay, so /etc stays mutable on systemd 255. Keeping the confext format rather than inventing a tarball means the artefact is already what native systemd-confext consumes on 256 or later; Azure Linux 4.0 ships 258.4. Payloads must declare ID=_any, because native mounting enforces extension-release matching that the custom mount bypasses, and a concrete ID would silently stop merging at the transition. Adds a change inventory across the new component, Trident and ACL with T-shirt estimates for design and implementation, plus suggested sequencing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 485 ++++++++++-------- 1 file changed, 281 insertions(+), 204 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 29951c3383..d2ac694860 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,6 +1,6 @@ # 0838 Slotted Extension Storage -- Date: 2026-10-12 +- Date: 2026-10-13 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) - Issue: TBD @@ -11,24 +11,25 @@ configured. That is one way to make an extension follow the slot, and it costs a partition pair per extension directory. This adds a second. An extension marked `scope: slotted` is stored per volume -under a Trident-owned store. A small boot-time component selects the booted -slot's copy into `/run/extensions` before the merge runs. Slot-scoping becomes a -property of the extension rather than of the partition table. +under a Trident-owned store. A small component resolves the booted slot early in +boot and exposes that slot's content before anything consumes it. Slot-scoping +becomes a property of the extension rather than of the partition table. + +The same component serves `/etc` layers, which otherwise have no viable +mechanism on systemd 255. ## Motivation and Goals `validate_extension_images_locations` (`config/host/mod.rs:218`) rejects any -extension directory that does not resolve to a device with A/B capabilities, -raising `ExtensionImageNotOnABVolume`. The reasoning is sound: extensions on -shared storage change the running slot immediately and do not roll back. +extension directory that does not resolve to a device with A/B capabilities. +The reasoning is sound: extensions on shared storage change the running slot +immediately and do not roll back. The cost is that the only way to satisfy it is a dedicated A/B volume pair. Trident's own reference configuration (`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on it. - -That price buys only slot-scoping, and it is charged in the currency hardest to -change: +spends four partitions and 2.4 GB on it. That price buys only slot-scoping, and +it is charged in the currency hardest to change: - **Partitions are fixed at install.** The size is a guess made before anyone knows what the host will carry; growing it means repartitioning. @@ -46,57 +47,145 @@ on an A/B block device*, one implementation of it. The obvious alternative is to place both slots' copies in one search directory and let `systemd-sysext` merge whichever matches the booted OS. Version mismatches are skipped quietly, so this very nearly works, and on a suitable -platform it would need no new code at all. +platform it needs no new code at all. -It fails for two independent reasons, either of which is sufficient: +It fails for two independent reasons, either sufficient on its own: 1. **Deploying one image under two filenames requires systemd v256.** `path_extract_image_name()` truncates an image name at the first `_` or `+` - (`src/basic/os-util.c`), so `platform_1.raw` and `platform_2.raw` can share a - single inner `extension-release.platform`. That function does not exist - before v256; it was added by commit `37543971` (2024-03-28). On earlier - systemd each copy needs its own `extension-release`, so a signed image must - be repacked per slot. The `user.extension-release.strict` xattr relaxes the - name correspondence but must be set inside the image at build time. + (`src/basic/os-util.c`), letting `platform_1.raw` and `platform_2.raw` share + one inner `extension-release.platform`. That function does not exist before + v256; it arrived in commit `37543971` (2024-03-28). Earlier systemd requires + each copy to carry its own `extension-release`, so a signed image must be + repacked per slot. 2. **The OS version must actually differ between slots.** Matching uses `ID=`, then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A platform whose `VERSION_ID` is coarser than its build cadence, or which can - deploy the same image to both slots, puts identical values in both slots and - both copies then match. + deploy the same image to both slots, puts identical values in both and both + copies match. Both conditions hold on the first intended consumer. Content matching remains the better mechanism where a platform satisfies both, and this design does not -preclude adopting it there. +preclude adopting it there. See [Future possibilities](#future-possibilities). ### Goals Slot-scoped extensions without a dedicated volume pair; rollback restores the -previous set with no servicing operation; no new partitions, no sealed-`/usr` -change, no re-signing; existing configurations unaffected. +previous set with no servicing operation; a workable `/etc` story on systemd +255; no new partitions, no sealed-`/usr` change, no modification to systemd. ### Non-goals Changing `systemd-sysext` matching semantics. Removing the A/B volume route, -which stays better where a pair is affordable, since the filesystem provides -the isolation and Trident does nothing. Extensions on the ESP. +which stays better where a pair is affordable. Extensions on the ESP. ## Scope -Sysexts and confexts declared under `os.sysexts` and `os.confexts`, on clean +Sysexts, confexts and `/etc` layers declared in the Host Configuration, on clean install and A/B update. Runtime update does not change slot and is unaffected. -**Slotted extensions carry `/usr` and `/opt` content only.** Contributing to -`/etc` is out of scope for this RFC. On a platform that assembles `/etc` as an -overlay whose lowerdir lives inside `/usr`, a sysext can only affect `/etc` if -it is merged before that overlay is built. Reordering the merge ahead of it -would place the `/etc` overlay on top of the `/usr` overlay, consuming the -kernel's entire filesystem stacking budget: `FILESYSTEM_MAX_STACK_DEPTH` is 2 -(`include/linux/fs.h`), enforced in `fs/overlayfs/super.c`, and an overlay's -depth is the maximum of its layers plus one. The `/etc` overlay would reach -depth 2 and any confext merged onto `/etc` afterwards would reach 3 and fail. +## Design + +### Selection is recomputed every boot, never persisted + +When firmware falls back to the previous slot on its own, no Trident code runs. +Anything written during servicing would be stale with nothing to correct it, so +the selection must be derived from what actually booted. + +Verified: Trident's A/B rollback path invokes no subsystems. +`rollback::validate_boot` only reconciles Host Status, and +`manual_rollback::stage_rollback` touches only the pcrlock policy. Only the +runtime path passes `&mut subsystems`. + +### Both halves run in the initrd -The durable fix is to stop sourcing `/etc` defaults from inside `/usr`, which is -a platform change rather than a Trident one. +The `/etc` half has no choice. On a platform that assembles `/etc` as an overlay +whose lowerdir lives inside `/usr`, anything contributing to `/etc` must act +before that overlay is mounted, which happens in the initrd. + +The sysext half can then ride along, because `/run` is bind-moved across +switch-root (`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). +Symlinks written to `/run/extensions` in the initrd are still present when +`systemd-sysext` merges later in the real root. Their targets must be written as +post-switch-root paths, so they are dangling in the initrd and correct +afterwards. + +Unifying is not merely possible, it is better. The slot is resolved **once**, so +the sysext set and the `/etc` layer cannot disagree about which slot they came +from. It also removes a large amount of accidental complexity that a real-root +systemd unit would carry: + +| Problem with a real-root unit | In the initrd | +| --- | --- | +| `WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states cannot fail the transaction, so the unit does not fail closed | No unit; failing in dracut is native | +| Ordering anchor is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong edge | Nothing to order against | +| A platform-specific `ConditionKernelCommandLine=` would block every other mode from running | No unit condition | +| Unit enablement needs a preset or scriptlet | No unit to enable | +| Store must be available at `local-fs.target` | `/sysroot` is mounted by definition | + +A real-root unit remains the fallback for platforms that cannot modify their +initrd. The mode selector therefore chooses *where* the loader runs as well as +how it detects the slot. + +### Why `/etc` payloads are confext images + +`systemd-confext` is the natural mechanism and is unusable here: on systemd 255 +it mounts `/etc` read-only. Mutable overlays arrived in v256; Flatcar's backport +is 1310 lines across five files, and modifying systemd is out of scope. + +The resolution is to keep the **format** and replace the **mount**. Payloads are +confext-format DDIs, carrying +`/etc/extension-release.d/extension-release.` (`src/basic/os-util.c`). +The loader mounts them read-only and adds them as additional lowerdirs to the +existing `/etc` overlay, whose upperdir is unchanged, so `/etc` stays mutable. + +This is preferred over tarballs or plain directory trees because: + +- **It is forward-compatible.** The payload is already a valid confext. When a + platform reaches systemd 256 or later, the identical artefact is handed to + native `systemd-confext --mutable=yes` and the custom mounting is deleted. The + custom mounting is a bridge, not a fork. Azure Linux 4.0 ships systemd 258.4, + so this transition is concrete rather than hypothetical. + + Two things do not change at that transition, and both matter. The loader + itself remains, because systemd has no concept of A/B slots: *selection* stays + the platform's job, and only *mounting* becomes native. And native mounting + reintroduces `extension-release` matching, which the custom lowerdir mount + bypasses entirely. +- **It reuses everything.** Same build and signing pipeline as sysexts, same + Trident download and hash-verification path, verity available by DDI + convention. +- **It is atomic.** A loop mount either appears or does not. A tarball + extraction can half-fail, and would also pay extraction cost on every boot. +- **SELinux labels travel in the filesystem image**, rather than in tar metadata. + +#### Payloads must declare `ID=_any` + +Because slot selection is the platform's job and not systemd's, the payload must +not additionally be subject to OS-version matching. A payload declaring a +concrete `ID=` would merge under the custom mount, which ignores matching, and +then silently stop merging the day a platform switches to native confext +mounting. That is the failure mode this RFC exists to prevent, reappearing at +the point of transition. + +Declaring `ID=_any` bypasses matching on every systemd version. Selection +remains entirely with the loader, which is what slot-scoping requires anyway, +and the move to native mounting changes no artefact. Deciding this late would +mean rebuilding every payload at transition time, so it is specified here. + +Adding lowerdirs does not increase filesystem stacking depth, because depth is +the maximum of the layers plus one and every lowerdir sits on a plain +filesystem. This matters: *reordering* the sysext merge ahead of the `/etc` +overlay would instead place `/etc` on top of the `/usr` overlay, reaching depth +2, and any confext merged afterwards would reach 3 and fail. +`FILESYSTEM_MAX_STACK_DEPTH` is 2 (`include/linux/fs.h`), enforced in +`fs/overlayfs/super.c`. + +### Selection is a set, not a sequence + +Creating N symlinks exposes a prefix of the intended set if it fails partway, +which matters where a driver and its userspace libraries must move together. +The loader prepares the complete set and commits it with a single rename. ## Public API Design @@ -118,10 +207,9 @@ os: ```yaml os: # Root of the per-volume extension store. - # Default /var/trident/sysexts. Must be on storage an A/B update does not - # replace, must not be a systemd-sysext search path, and must be on a - # filesystem available at local-fs.target. - extensionStore: /mnt/data/trident/sysexts + # Default /var/trident/extensions. Must be on storage an A/B update does not + # replace, and must not be a systemd-sysext search path. + extensionStore: /mnt/data/trident/extensions sysexts: # unchanged, and still the default @@ -132,15 +220,26 @@ os: - url: https://example.com/platform.raw sha384: scope: slotted + + # new: confext-format images contributed as /etc overlay layers + etcLayers: + - url: https://example.com/platform-etc.raw + sha384: + scope: slotted ``` | `scope` | Where the bytes live | Where it is activated | | --- | --- | --- | | `shared` (default) | At `path`. | At `path`. Today's behaviour. | -| `slotted` | `/{a,b}/{name}.raw` | `/run/extensions/`, by the loader | +| `slotted` | `/{a,b}/` | `/run/extensions`, or as an `/etc` lowerdir | + +A slotted extension always surfaces in a loader-owned location, so `path` does +not apply to it and is rejected. -A slotted extension always surfaces in `/run/extensions`, so `path` does not -apply to it and is rejected. +`etcLayers` is a separate list rather than a `scope` on `os.confexts`, because +the two are activated by different mechanisms and only one of them is +`systemd-confext`. If a platform later gains mutable confext support, entries +can migrate to `os.confexts` unchanged. ### The validation change @@ -153,9 +252,8 @@ becomes *the destination is slot-scoped*, satisfied either way: | `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | | `abUpdate`, `scope: slotted` | `path` need not be; the store plus the loader provide the scoping. | -This is the substantive behavioural change. It is only safe where the platform -actually provides a loader, so validation must check a **platform capability**, -not merely that A/B is configured. See [Rollout](#rollout). +This is only safe where the platform actually provides a loader, so validation +must test a **platform capability**, not merely that A/B is configured. ### Validation @@ -167,49 +265,38 @@ not merely that A/B is configured. See [Rollout](#rollout). | `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | | `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | | `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| `extensionStore` not available at `local-fs.target` | Reject: the loader runs before anything later. | -| Two slotted entries resolving to the same basename | Reject: they would collide in `/run/extensions`. | +| Two slotted entries resolving to the same basename | Reject: they would collide. | +| `etcLayers` entry without confext metadata | Reject at deploy time, after mount. | ### Backward compatibility -Both fields default, so existing Host Configurations parse and behave +All new fields default, so existing Host Configurations parse and behave identically. The relaxed check is strictly more permissive, so anything valid -today stays valid. The schema gains two optional properties. - -### Alternatives rejected - -- **Sentinel `path`** (`//x.raw`): leaks internal layout, invites a - literal `a` or `b`. -- **Separate `os.slottedSysexts` list**: duplicates the type and its validation. -- **Infer from `extension-release`**: inside the image, so unavailable at static - validation, and it removes the operator's choice to slot an `ID=_any` - extension deliberately. +today stays valid. The schema gains three optional properties. ## Implementation ``` -/var/trident/sysexts/ -├── a/platform.raw -└── b/platform.raw - -/run/extensions/ -└── platform.raw -> /var/trident/sysexts/a/platform.raw # recreated each boot +/ +├── a/ +│ ├── manifest.json +│ ├── sysext/platform.raw +│ └── etc/platform-etc.raw +└── b/ ... + +/run/extensions/platform.raw -> /a/sysext/platform.raw +/etc overlay lowerdir list += ``` The store is outside every search path, so its contents are invisible to `systemd-sysext` until selected. Both slot directories persist across an update, so rollback needs no re-download. -Symlinks are followed: `image_discover` opens each search path with -`chase_and_opendir(path, root, CHASE_PREFIX_ROOT, ..)` and stats entries with -`flags = 0` on a running system, with an upstream comment explicitly permitting -symlinks into the search path (`src/shared/discover-image.c`). - -### Servicing writes the store, and must write it atomically +### Servicing writes the store, atomically The extensions subsystem already runs on clean install and A/B update via `provision()`, documented as "migrate state from A-partition to B-partition (or -vice versa)". For a slotted entry it resolves the target volume with +vice versa)". It resolves the target volume with `EngineContext::get_ab_update_volume()`, already used to select the per-slot verity addon, and writes into `//`. @@ -221,116 +308,106 @@ Directory-type images are tolerated; `.raw` are not. Trident today falls back to a non-atomic `fs::copy` when `rename` fails across a filesystem boundary (`crates/trident/src/subsystems/extensions/mod.rs:525-540`). -Staging must move to a temporary file on the destination's own filesystem, -verified and `fsync`ed, then renamed. This is a prerequisite, not a -consequence. - -### Selection happens at boot - -Selection is recomputed on every boot from the slot that actually booted. This -is what makes unsupervised rollback correct: when firmware falls back on its -own, no Trident code runs, so any decision written during servicing would be -stale with nothing to correct it. - -`/run` is tmpfs, so a stale pointer cannot be represented across a boot. - -The loader resolves the booted slot, then exposes the matching slot directory's -contents in `/run/extensions`. Its unit: - -```ini -[Unit] -Description=Select Trident system extensions for the booted slot -DefaultDependencies=no -After=local-fs.target -Before=systemd-sysext.service -[Service] -Type=oneshot -RemainAfterExit=yes -ExecStart=/usr/bin/trident-sysext-loader -[Install] -WantedBy=sysinit.target -``` - -`DefaultDependencies=no` is required; the implicit `After=sysinit.target` would -place it after the merge and create a cycle. - -`Before=systemd-sysext.service` is the correct edge even on platforms that do -not enable that unit directly. Flatcar-derived systems wire -`ensure-sysext.service` into `sysinit.target.wants` instead, but that unit -carries `BindsTo=systemd-sysext.service` and `After=systemd-sysext.service`: it -runs *after* the merge, and exists to `daemon-reload` and restart targets so -that units shipped inside an extension are picked up. Ordering before it would -therefore permit running after the merge. +This is a prerequisite fix, not a consequence of this design. -This works because systemd evaluates conditions in `unit_start()` -(`src/core/unit.c`), not at transaction creation, so the loader populates -`/run/extensions` before the merge unit's own `ConditionDirectoryNotEmpty=` is -tested. The same fact rules out an `ExecStartPre=` drop-in, since that condition -gates the whole unit and would skip it while the directories are still empty. - -### Failing closed - -`WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states has no effect on -the transaction when it fails, and `Before=` carries no requirement. The unit -above therefore does **not** fail closed on its own: a loader failure would -leave boot to continue with extensions absent or partially present. - -The merge unit must carry `Requires=trident-sysext-loader.service` via a -drop-in. This must be a deliberate requirement edge rather than forcing -emergency mode, because the store is on shared storage and a storage fault must -not be able to fail both slots and defeat A/B rollback. - -### Selection is a set, not a sequence +### The store carries a manifest -Creating N symlinks exposes a prefix of the intended set if it fails partway, -which matters where a driver and its userspace libraries must move together. -The loader must therefore commit the set atomically: prepare the complete set -in a staging directory under `/run` and rename it into place, or select the -whole directory at once. +Globbing `*.raw` cannot distinguish "no extensions configured" from "storage +lost", "mount failed" or "update interrupted", and treating all of those as +success produces a quietly extension-less host. -Selecting the whole directory is simpler but means everything at that search -path comes from one slot, so a shared extension can no longer sit beside a -slotted one there. That trade should be settled before implementation. +Each slot directory carries a manifest naming the expected images and hashes. An +absent manifest means the feature is not configured; a present manifest with +missing or unexpected content is an error. This also gives garbage collection +something to act on. ### Slot detection -The loader resolves the slot per platform, selected by a `LOADER_MODE` -environment variable so that Trident does not hardcode one platform's -mechanism. The first mode reads a slot label from the kernel command line, -supplied by a per-slot signed boot artefact. +The loader resolves the slot per platform, selected by a mode setting so that +Trident does not hardcode one platform's mechanism. The first mode reads a slot +label from the kernel command line, supplied by a per-slot signed boot artefact. A label is preferable to a partition UUID because no mapping table has to stay in sync with a layout that changes. However, a label is an *assertion*: a -packaging error could pair it with the other slot's device. The loader should -cross-check it against the booted `/usr` backing device and refuse to proceed on -a mismatch. +packaging error could pair it with the other slot's device. The loader +cross-checks it against the booted `/usr` backing device and refuses to proceed +on a mismatch. + +## Change inventory + +Sizes are T-shirt estimates covering design and implementation separately. Risk +reflects how much is unknown rather than how much work there is. + +### New component + +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; logic is pure given cmdline, store and target, so unit-testable without root or a VM | +| `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | +| Manifest format and verification | M | S | Low | Decide internal versus API-visible | + +### Trident + +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| **Atomic extension placement** | S | S | Low | Prerequisite bug fix; live defect today, independent of this RFC | +| `scope` and `extensionStore` API | S | S | Low | Additive, both default | +| `etcLayers` API | M | S | Med | New concept; shape depends on confext migration path | +| Relax `validate_extension_images_locations` | M | S | Med | Must test loader capability, not just A/B | +| Subsystem writes slotted entries per volume | S | M | Low | `provision()` already has the target volume | +| Manifest generation | S | S | Low | | +| Package loader and dracut module in the RPM | S | S | Low | `%files acl` already ships a second binary | +| Schema regeneration | S | S | Low | Mechanical | + +### ACL + +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| New dracut module to install and run the loader | S | S | Low | `dracut-acl-selinux-toggle` is an existing precedent | +| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount itself is one line, validation is the work | +| Order the loader before the `/etc` overlay mount | S | S | Low | Same initrd window as `initrd-setup-root` | +| Review `overlaycleanup()` interaction | M | S | Med | It compares only against the distro lower layer, so extension-provided defaults will not auto-prune | +| Package catalog entry | S | S | Low | | +| Initrd size and UKI rebuild | S | S | Med | Re-signing already happens every build; size growth is the open part | + +### Not affected + +| Component | Why | +| --- | --- | +| systemd / Azure Linux | Nothing. Avoiding a systemd change is the point of this route | +| AgentBaker, AKS RP | No interaction | +| Image Customizer | None unless RFC 0789 lands, and then only at the shared validation | -Platform-specific conditions belong in platform-specific units. A single unit -carrying `ConditionKernelCommandLine=` for one platform's parameter would -prevent every other mode from ever running, since the condition is evaluated -before the binary reads `LOADER_MODE`. +### Cross-cutting -### The store needs a manifest +| Change | Design | Impl | Risk | Notes | +| --- | :-: | :-: | :-: | --- | +| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot; see [Rollout](#rollout) | +| Boot and rollback end-to-end tests | M | L | High | Unsupervised rollback cannot be asserted from unit tests | +| SELinux policy | M | M | High | Extensions are already unsupported with SELinux enforcing | -Globbing `*.raw` cannot distinguish "no extensions configured" from "storage -lost", "mount failed" or "update interrupted", and treating all of those as -success produces a quietly extension-less host. +### Suggested sequencing -Each slot directory should carry a manifest naming the expected images and their -hashes. An absent manifest means the feature is not configured; a present -manifest with missing or unexpected content is an error. This also gives -garbage collection something to act on, which globbing does not. +1. **Atomic placement fix.** Standalone, lands independently, worth doing + regardless of this RFC. +2. Loader crate and dracut module, sysext half only, behind no API change. +3. Trident API and the validation relaxation. +4. `/etc` layer half, including the `bootengine` patch. Separable; ships after. +5. Rollout gating, once the per-slot-initrd question is settled. ## Rollout -The loader ships in `/usr` and therefore rides the A/B swap. A rollback into a -slot whose `/usr` predates the loader has no selector, and every slotted -extension is absent for that boot. +If the initrd is shared between slots, loader capability is never +slot-dependent and no staged rollout is required. + +If it is per-slot, a rollback into a slot whose initrd predates the loader has +no selector, and every slotted extension is absent for that boot. Enabling +`scope: slotted` then requires two phases: deploy a release carrying the loader +and let it reach both slots, then permit slotted configuration. Validation must +verify that capability rather than assume it. -Enabling `scope: slotted` therefore requires two phases: first deploy a release -carrying the loader and let it reach both slots, then permit slotted -configuration. Validation must verify that capability rather than assume it, -which is why the relaxed check above tests for a loader and not merely for A/B. +This is an open question, listed below, and it changes the work in the table +above. ## Relationship to RFC 0789 @@ -343,25 +420,22 @@ Both touch `validate_extension_images_locations`, in opposite directions: 0789 tightens it into dynamic validation against the storage graph, this adds a second way to satisfy it. Whichever lands first should leave the check expressed as *is this destination slot-scoped*, so the other is an added arm rather than a -rewrite. 0789's rollback argument, that extensions revert because the slot's -filesystem reverts, is the A/B volume mechanism and holds unchanged for -`scope: shared`. +rewrite. ## Testing -- Clean install: image in the correct slot directory, `/run/extensions` link - resolving, merged after boot. +- Clean install: image in the correct slot directory, selection resolving, + merged after boot. - A/B update: the selection repoints and **the merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. -- **Unsupervised rollback: with no Trident servicing operation in between**, the +- **Unsupervised rollback, with no Trident servicing operation in between**: the previous slot's content is merged again. This is the case the design exists for. +- `/etc` layer: a slotted layer is visible in `/etc`, `/etc` remains writable, + and a write to a file the layer provides persists across reboot. - Loader failure: boot does not proceed silently with a partial set. -- Ordering: the selection lands before the merge, asserted on the first boot - after an update rather than on a steady-state boot. -- A malformed image in a slot directory is rejected before it reaches - `/run/extensions`. +- A malformed image in a slot directory is rejected before it is selected. - Validation: each reject row, plus a shared extension off an A/B volume still rejected. - Compatibility: a configuration with no `scope` places byte-identically to @@ -370,22 +444,19 @@ filesystem reverts, is the A/B volume mechanism and holds unchanged for ## Counter-arguments - **"Just add the partitions."** Better where possible: the filesystem provides - the isolation, Trident does nothing at boot, and capacity is contained. This - is for layouts that cannot. Directory slotting gives equivalent naming, not + the isolation, nothing runs at boot, and capacity is contained. This is for + layouts that cannot. Directory slotting gives equivalent naming, not equivalent failure containment: a store on shared storage can fill and affect the running slot, which a dedicated pair cannot. -- **"Do it in the initrd."** Flatcar does exactly this and it is the stronger - position in principle, since the merge then precedes unit enumeration and the - `/etc` overlay. It requires rebuilding and re-signing the UKI, which is a - delivery cost rather than a correctness argument, and it does not fit a - platform that cannot modify its initrd. Worth revisiting for extensions needed - before `sysinit.target`, which this design cannot serve. - **"Put it in `/usr`."** Better still, and unavailable without re-sealing dm-verity and re-signing. Upstream also forbids extension storage under `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. - **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and `systemd-stub` places them in the initrd. +- **"Backport mutable confexts."** It exists, and it is 1310 lines across five + files of someone else's component. This route reaches the same place without + carrying that, and converges on native confexts later at zero payload cost. - **"Doubles storage."** It does. The A/B volume route doubles it too, with the allocation fixed at install. @@ -399,6 +470,8 @@ filesystem reverts, is the A/B volume mechanism and holds unchanged for main pass is ordered after the merge. - **A systemd generator.** Generators run sandboxed with most of the filesystem read-only, and `/var` is not mounted when they run. +- **Reordering the sysext merge ahead of the `/etc` overlay.** Exhausts the + filesystem stacking budget and breaks confexts outright. - **Repair after boot.** Any agent running at `multi-user.target` is far too late to influence the merge. @@ -409,12 +482,13 @@ persisting it: - **Flatcar**, whose layout is closest to the first consumer, stamps the version into the filename on shared storage and rewrites the activating symlink in the - initrd from the booted `/usr`'s `os-release`. + initrd from the booted `/usr`'s `os-release`. It has since moved extension + mounting wholesale into the initrd and deleted its late-merge workaround. - **ChromeOS** stores `dlc_a`/`dlc_b` on shared stateful storage and derives the - slot from the booted partition number on every service start. It also clears - the inactive slot's validity marker *before* writing and re-sets it after, so - a rollback into a half-written slot finds an honest failure rather than a - plausible file. That is worth adopting. + slot from the booted partition number on every service start. It clears the + inactive slot's validity marker *before* writing and re-sets it after, so a + rollback into a half-written slot finds an honest failure rather than a + plausible file. Worth adopting. - **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` and symlinks the winner into `/run/extensions`. @@ -424,29 +498,32 @@ extensions silently stop matching and are not loaded. ## Open questions -1. **Whole-directory selection or a prepared set?** Both commit atomically. The +1. **Is the initrd per-slot or shared?** Decides whether a staged rollout is + needed at all, and changes the estimates above. +2. **Whole-directory selection or a prepared set?** Both commit atomically. The former is simpler; the latter allows shared and slotted extensions to share a search path. -2. **Manifest format and whether it is Trident-internal or part of the API.** -3. **Retention.** Nothing prunes a slot directory when an entry leaves the +3. **Manifest format**, and whether it is Trident-internal or API-visible. +4. **Retention.** Nothing prunes a slot directory when an entry leaves the configuration. Reconciling in `provision()` conflicts with keeping the previous slot's files for rollback. -4. **Default store path.** `/var/trident/sysexts` against - `/var/lib/trident/sysexts`; the datastore already lives under the latter. -5. **Confexts.** The same mechanism applies with different search paths, but - `/etc` content is out of scope here, so whether confexts are in scope at all - needs deciding. -6. **SELinux.** Extensions are already unsupported with SELinux enforcing. - Slotting does not change that, but loader-created symlinks in `/run` need - labelling, and 0789 proposes moving that check to every servicing type. -7. **Measured boot.** The selected image is outside the UKI's measurements, so +5. **Default store path.** `/var/trident/extensions` against + `/var/lib/trident/extensions`; the datastore already lives under the latter. +6. **Measured boot.** The selected image is outside the UKI's measurements, so identical measurements can yield different `/usr` contents. Whether that is acceptable depends on the platform's attestation claims. +7. **SELinux.** Extensions are already unsupported with SELinux enforcing, which + is a precondition this RFC does not resolve. ## Future possibilities -- **Initrd-phase selection**, for extensions that must be merged before unit - enumeration or before an `/etc` overlay is assembled. +- **Native confext mounting.** On systemd 256 or later, the loader places the + selected payload in a confext search path and `systemd-confext --mutable=yes` + mounts it, replacing the custom lowerdir handling. The loader still performs + selection, and the payloads do not change provided they declare `ID=_any`. + Azure Linux 4.0 ships systemd 258.4. - **Content matching** on platforms with systemd v256 or later and a build-granular `VERSION_ID`, which needs no loader at all. +- **Initrd-phase sysext merging**, as Flatcar now does, for extensions that must + be present before unit enumeration. - Per-volume scoping for other Trident-managed state that is shared today. From 710d82772ef107e2d312169f798f98db4daa48fc Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 14 Oct 2026 10:23:11 -0700 Subject: [PATCH 07/27] docs: rebase 0838 on RFC 0789 as a dependency 0789 was rewritten and now names this RFC in three places: the optional path discussion, destination validation, and an open question asking how a bundled extension requests slotted placement. This RFC now answers that question rather than sitting beside it. The common case is a COSI-bundled extension that is slot-scoped, since an extension shipped with the OS image is already tied to it. An independently hosted extension placed per slot is the unusual combination. Removes what 0789 delivers: the invented manifest.json is replaced by 0789's Deployed Extension Inventory, the store is keyed by (kind, id) instead of filenames, and the effective set, resolved-placement validation, atomic staging and SELinux check relocation are referenced rather than restated. Adds scope: shared|slotted on both the COSI ExtensionImage and the Host Configuration Extension, spelled identically so one concept has one name, and mutually exclusive with path so placement never changes implicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 432 +++++++++--------- 1 file changed, 216 insertions(+), 216 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index d2ac694860..efbc05877b 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,35 +1,45 @@ # 0838 Slotted Extension Storage -- Date: 2026-10-13 +- Date: 2026-10-14 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) -- Issue: TBD +- Builds on: [RFC 0789](https://github.com/microsoft/trident/pull/789) ## Summary -Trident requires every extension destination to sit on an A/B volume when A/B is -configured. That is one way to make an extension follow the slot, and it costs a -partition pair per extension directory. +[RFC 0789](https://github.com/microsoft/trident/pull/789) carries extension +images inside the COSI, so an OS-coupled extension ships and rolls out with the +OS. It leaves one question open: such an extension still has to land somewhere +that follows the A/B slot, and today the only way to satisfy that is a dedicated +A/B partition pair. -This adds a second. An extension marked `scope: slotted` is stored per volume -under a Trident-owned store. A small component resolves the booted slot early in -boot and exposes that slot's content before anything consumes it. Slot-scoping -becomes a property of the extension rather than of the partition table. - -The same component serves `/etc` layers, which otherwise have no viable +This RFC answers it. A bundled extension can request Trident-managed, per-slot +storage, and a boot-time component selects the booted slot's copy before +anything consumes it. It also supplies `/etc` layers, which have no viable mechanism on systemd 255. +An extension bundled in the COSI is already tied to the OS image it shipped +with. Slotted placement is the natural destination for it, and the two features +are expected to be used together in the common case. An independently hosted +extension fetched over OCI and placed per slot is the unusual combination, not +the default. + ## Motivation and Goals -`validate_extension_images_locations` (`config/host/mod.rs:218`) rejects any -extension directory that does not resolve to a device with A/B capabilities. -The reasoning is sound: extensions on shared storage change the running slot -immediately and do not roll back. +0789 establishes that OS-coupled extensions belong in the COSI. It also +establishes, in [Destination +Validation](https://github.com/microsoft/trident/pull/789), that a bundled image +written to shared storage changes the running slot immediately and defeats +rollback, so placement must be slot-scoped. -The cost is that the only way to satisfy it is a dedicated A/B volume pair. -Trident's own reference configuration +Trident has exactly one mechanism for that today, and +`validate_extension_images_locations` (`config/host/mod.rs:218`) enforces it: the +destination must resolve to a device with A/B capabilities. The only way to +satisfy it is a dedicated A/B volume pair. Trident's own reference configuration (`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on it. That price buys only slot-scoping, and -it is charged in the currency hardest to change: +spends four partitions and 2.4 GB on exactly this. + +That price buys only slot-scoping, and it is charged in the currency hardest to +change: - **Partitions are fixed at install.** The size is a guess made before anyone knows what the host will carry; growing it means repartitioning. @@ -44,7 +54,7 @@ on an A/B block device*, one implementation of it. ### Why content matching alone is not sufficient -The obvious alternative is to place both slots' copies in one search directory +A tempting alternative is to place both slots' copies in one search directory and let `systemd-sysext` merge whichever matches the booted OS. Version mismatches are skipped quietly, so this very nearly works, and on a suitable platform it needs no new code at all. @@ -53,11 +63,10 @@ It fails for two independent reasons, either sufficient on its own: 1. **Deploying one image under two filenames requires systemd v256.** `path_extract_image_name()` truncates an image name at the first `_` or `+` - (`src/basic/os-util.c`), letting `platform_1.raw` and `platform_2.raw` share - one inner `extension-release.platform`. That function does not exist before - v256; it arrived in commit `37543971` (2024-03-28). Earlier systemd requires - each copy to carry its own `extension-release`, so a signed image must be - repacked per slot. + (`src/basic/os-util.c`), letting two copies share one inner + `extension-release`. That function arrived in commit `37543971` (2024-03-28). + Earlier systemd requires each copy to carry its own, so a signed image must + be repacked per slot. 2. **The OS version must actually differ between slots.** Matching uses `ID=`, then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A platform whose `VERSION_ID` is coarser than its build cadence, or which can @@ -65,24 +74,44 @@ It fails for two independent reasons, either sufficient on its own: copies match. Both conditions hold on the first intended consumer. Content matching remains -the better mechanism where a platform satisfies both, and this design does not -preclude adopting it there. See [Future possibilities](#future-possibilities). +the better mechanism where a platform satisfies both; see +[Future possibilities](#future-possibilities). ### Goals -Slot-scoped extensions without a dedicated volume pair; rollback restores the -previous set with no servicing operation; a workable `/etc` story on systemd -255; no new partitions, no sealed-`/usr` change, no modification to systemd. +Give bundled extensions a slot-scoped destination without a dedicated volume +pair; restore the previous set on rollback with no servicing operation; provide +a workable `/etc` story on systemd 255; require no new partitions, no +sealed-`/usr` change and no modification to systemd. ### Non-goals -Changing `systemd-sysext` matching semantics. Removing the A/B volume route, -which stays better where a pair is affordable. Extensions on the ESP. +Changing `systemd-sysext` matching semantics. Replacing the A/B volume route, +which stays better where a pair is affordable. Extensions on the ESP. Anything +0789 already covers, including the effective set, the deployed inventory and +extension-only updates. + +## Dependencies + +**RFC 0789 is a prerequisite.** This RFC adds a placement policy to the model +0789 defines and does not restate it. In particular it relies on: + +| From 0789 | Used here as | +| --- | --- | +| `extensions.sysexts` / `extensions.confexts` in COSI 1.3 | The delivery mechanism for slotted content | +| `(kind, id)` as stable extension identity | The store key, instead of a filename | +| Deployed Extension Inventory in Host Status | The record of what is deployed per slot; this RFC adds no separate manifest | +| Effective set computed from COSI plus Host Configuration | The set the loader selects from | +| Resolved-placement destination validation | The hook where the slotted arm is added | +| Atomic staging on the destination filesystem | A prerequisite shared by both RFCs | +| SELinux check moved into validation for every servicing type | Inherited unchanged | -## Scope +0789's prerequisites apply transitively, including the `set_up_extensions` +replacement bug and durable staged payloads. -Sysexts, confexts and `/etc` layers declared in the Host Configuration, on clean -install and A/B update. Runtime update does not change slot and is unaffected. +Bundled extensions under +[disk streaming](../../Explanation/Disk-Streaming.md) are refused by 0789; that +restriction is inherited. ## Design @@ -90,7 +119,7 @@ install and A/B update. Runtime update does not change slot and is unaffected. When firmware falls back to the previous slot on its own, no Trident code runs. Anything written during servicing would be stale with nothing to correct it, so -the selection must be derived from what actually booted. +the selection must derive from what actually booted. Verified: Trident's A/B rollback path invokes no subsystems. `rollback::validate_boot` only reconciles Host Status, and @@ -99,33 +128,31 @@ runtime path passes `&mut subsystems`. ### Both halves run in the initrd -The `/etc` half has no choice. On a platform that assembles `/etc` as an overlay -whose lowerdir lives inside `/usr`, anything contributing to `/etc` must act -before that overlay is mounted, which happens in the initrd. +The `/etc` half has no choice. Where `/etc` is assembled as an overlay whose +lowerdir lives inside `/usr`, anything contributing to it must act before that +overlay is mounted, which happens in the initrd. -The sysext half can then ride along, because `/run` is bind-moved across -switch-root (`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). -Symlinks written to `/run/extensions` in the initrd are still present when -`systemd-sysext` merges later in the real root. Their targets must be written as -post-switch-root paths, so they are dangling in the initrd and correct -afterwards. +The sysext half rides along, because `/run` is bind-moved across switch-root +(`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). Symlinks +written to `/run/extensions` in the initrd are still present when +`systemd-sysext` merges in the real root. Their targets are written as +post-switch-root paths, so they are dangling in the initrd and correct after. -Unifying is not merely possible, it is better. The slot is resolved **once**, so -the sysext set and the `/etc` layer cannot disagree about which slot they came -from. It also removes a large amount of accidental complexity that a real-root -systemd unit would carry: +Unifying is better than running them separately: the slot is resolved **once**, +so the sysext set and the `/etc` layer cannot disagree about which slot they +came from. It also removes complexity a real-root systemd unit would carry: | Problem with a real-root unit | In the initrd | | --- | --- | | `WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states cannot fail the transaction, so the unit does not fail closed | No unit; failing in dracut is native | -| Ordering anchor is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong edge | Nothing to order against | -| A platform-specific `ConditionKernelCommandLine=` would block every other mode from running | No unit condition | +| Ordering is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong anchor | Nothing to order against | +| A platform-specific `ConditionKernelCommandLine=` would block every other mode | No unit condition | | Unit enablement needs a preset or scriptlet | No unit to enable | | Store must be available at `local-fs.target` | `/sysroot` is mounted by definition | A real-root unit remains the fallback for platforms that cannot modify their -initrd. The mode selector therefore chooses *where* the loader runs as well as -how it detects the slot. +initrd, so the mode selector chooses *where* the loader runs as well as how it +detects the slot. ### Why `/etc` payloads are confext images @@ -133,51 +160,47 @@ how it detects the slot. it mounts `/etc` read-only. Mutable overlays arrived in v256; Flatcar's backport is 1310 lines across five files, and modifying systemd is out of scope. -The resolution is to keep the **format** and replace the **mount**. Payloads are -confext-format DDIs, carrying -`/etc/extension-release.d/extension-release.` (`src/basic/os-util.c`). -The loader mounts them read-only and adds them as additional lowerdirs to the -existing `/etc` overlay, whose upperdir is unchanged, so `/etc` stays mutable. - -This is preferred over tarballs or plain directory trees because: - -- **It is forward-compatible.** The payload is already a valid confext. When a - platform reaches systemd 256 or later, the identical artefact is handed to - native `systemd-confext --mutable=yes` and the custom mounting is deleted. The - custom mounting is a bridge, not a fork. Azure Linux 4.0 ships systemd 258.4, - so this transition is concrete rather than hypothetical. - - Two things do not change at that transition, and both matter. The loader - itself remains, because systemd has no concept of A/B slots: *selection* stays - the platform's job, and only *mounting* becomes native. And native mounting - reintroduces `extension-release` matching, which the custom lowerdir mount - bypasses entirely. +Keep the **format** and replace the **mount**. Payloads are confext-format DDIs +carrying `/etc/extension-release.d/extension-release.` +(`src/basic/os-util.c`). The loader mounts them read-only and adds them as +additional lowerdirs to the existing `/etc` overlay, whose upperdir is +unchanged, so `/etc` stays mutable. + +Preferred over tarballs or plain trees because: + +- **It is forward-compatible.** The payload is already a valid confext. Azure + Linux 4.0 ships systemd 258.4, where the identical artefact is handed to + native `systemd-confext --mutable=yes` and the custom mounting is deleted. + Two things do not change at that transition: the loader remains, because + systemd has no concept of A/B slots and selection stays the platform's job; + and native mounting reintroduces `extension-release` matching that the custom + lowerdir mount bypasses. - **It reuses everything.** Same build and signing pipeline as sysexts, same - Trident download and hash-verification path, verity available by DDI - convention. + 0789 delivery path, verity available by DDI convention. - **It is atomic.** A loop mount either appears or does not. A tarball - extraction can half-fail, and would also pay extraction cost on every boot. -- **SELinux labels travel in the filesystem image**, rather than in tar metadata. + extraction can half-fail and pays extraction cost on every boot. +- **SELinux labels travel in the filesystem image**, not in tar metadata. #### Payloads must declare `ID=_any` Because slot selection is the platform's job and not systemd's, the payload must not additionally be subject to OS-version matching. A payload declaring a -concrete `ID=` would merge under the custom mount, which ignores matching, and -then silently stop merging the day a platform switches to native confext -mounting. That is the failure mode this RFC exists to prevent, reappearing at -the point of transition. +concrete `ID=` would merge under the custom mount, which ignores matching, then +silently stop merging the day a platform switches to native confext mounting. +That is the failure mode this RFC exists to prevent, reappearing at the point of +transition. -Declaring `ID=_any` bypasses matching on every systemd version. Selection -remains entirely with the loader, which is what slot-scoping requires anyway, -and the move to native mounting changes no artefact. Deciding this late would -mean rebuilding every payload at transition time, so it is specified here. +`ID=_any` bypasses matching on every systemd version. Selection remains with the +loader, which is what slot-scoping requires anyway, and the move to native +mounting changes no artefact. Deciding this late would mean rebuilding every +payload, so it is specified here. 0789 already states that `ID=_any` must not be +warned on, since it declares compatibility rather than delivery method. Adding lowerdirs does not increase filesystem stacking depth, because depth is the maximum of the layers plus one and every lowerdir sits on a plain -filesystem. This matters: *reordering* the sysext merge ahead of the `/etc` -overlay would instead place `/etc` on top of the `/usr` overlay, reaching depth -2, and any confext merged afterwards would reach 3 and fail. +filesystem. *Reordering* the sysext merge ahead of the `/etc` overlay would +instead place `/etc` on top of the `/usr` overlay, reaching depth 2, and any +confext merged afterwards would reach 3 and fail. `FILESYSTEM_MAX_STACK_DEPTH` is 2 (`include/linux/fs.h`), enforced in `fs/overlayfs/super.c`. @@ -189,20 +212,23 @@ The loader prepares the complete set and commits it with a single rename. ## Public API Design -### Today +### Answering 0789's open question -```yaml -os: - sysexts: - - url: https://example.com/tooling.raw # http(s) | file | oci - sha384: - path: /var/lib/extensions/tooling.raw # optional; allow-listed dir, *.raw -``` +0789 asks how a COSI entry requests slotted placement, and states that omitting +`path` must not opt in. The answer is an explicit `scope`, spelled identically +on both sources so one concept has one name. + +**COSI metadata**, extending 0789's `ExtensionImage`: -`path` is the destination, defaulting to `/var/lib/extensions/{name}.raw`. When -`storage.abUpdate` is configured it must also resolve to an A/B volume. +```json +{ + "id": "gpu-driver", + "image": { "path": "images/extensions/gpu-driver.rawzst", "...": "..." }, + "scope": "slotted" +} +``` -### Proposed +**Host Configuration**, extending `Extension`: ```yaml os: @@ -212,16 +238,13 @@ os: extensionStore: /mnt/data/trident/extensions sysexts: - # unchanged, and still the default - - url: https://example.com/tooling.raw + - url: https://example.com/tooling.raw # unchanged, still the default sha384: - - # new - url: https://example.com/platform.raw sha384: scope: slotted - # new: confext-format images contributed as /etc overlay layers + # confext-format images contributed as /etc overlay layers etcLayers: - url: https://example.com/platform-etc.raw sha384: @@ -231,29 +254,32 @@ os: | `scope` | Where the bytes live | Where it is activated | | --- | --- | --- | | `shared` (default) | At `path`. | At `path`. Today's behaviour. | -| `slotted` | `/{a,b}/` | `/run/extensions`, or as an `/etc` lowerdir | +| `slotted` | `/{a,b}/`, keyed by `(kind, id)` | `/run/extensions`, or as an `/etc` lowerdir | -A slotted extension always surfaces in a loader-owned location, so `path` does -not apply to it and is rejected. +`scope` and `path` are mutually exclusive: a slotted extension surfaces in a +loader-owned location, so a caller-supplied destination would be silently +ignored. This satisfies 0789's requirement that placement never change +implicitly. -`etcLayers` is a separate list rather than a `scope` on `os.confexts`, because -the two are activated by different mechanisms and only one of them is -`systemd-confext`. If a platform later gains mutable confext support, entries -can migrate to `os.confexts` unchanged. +`etcLayers` is a separate list rather than a `scope` on `os.confexts` because +the two are activated by different mechanisms, only one of which is +`systemd-confext`. The COSI side needs no equivalent split: a confext entry with +`scope: slotted` is an `/etc` layer. ### The validation change `validate_extension_images_locations` is relaxed, not removed. Its requirement -becomes *the destination is slot-scoped*, satisfied either way: +becomes *the destination is slot-scoped*, satisfied by either mechanism. This is +the arm 0789's Destination Validation anticipates: | Configuration | Rule | | --- | --- | | No `abUpdate` | No check, as today. | -| `abUpdate`, `scope: shared` | `path` must be on an A/B volume. Unchanged. | -| `abUpdate`, `scope: slotted` | `path` need not be; the store plus the loader provide the scoping. | +| `abUpdate`, `scope: shared` | Destination must be on an A/B volume. Unchanged. | +| `abUpdate`, `scope: slotted` | Satisfied by the store, **only when loader capability is present**. | -This is only safe where the platform actually provides a loader, so validation -must test a **platform capability**, not merely that A/B is configured. +0789 states the same constraint from its side: slotted storage qualifies only +with loader capability, and omitting `path` must not bypass the A/B check. ### Validation @@ -265,67 +291,59 @@ must test a **platform capability**, not merely that A/B is configured. | `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | | `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | | `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| Two slotted entries resolving to the same basename | Reject: they would collide. | | `etcLayers` entry without confext metadata | Reject at deploy time, after mount. | +| `etcLayers` entry without `ID=_any` | Reject; see above. | + +Duplicate `(kind, id)` and destination collisions are already covered by 0789 +and are not restated. ### Backward compatibility -All new fields default, so existing Host Configurations parse and behave -identically. The relaxed check is strictly more permissive, so anything valid -today stays valid. The schema gains three optional properties. +All new fields default, so existing Host Configurations and 1.3 COSI files +without `scope` behave identically. The relaxed check is strictly more +permissive, so anything valid today stays valid. ## Implementation ``` / ├── a/ -│ ├── manifest.json -│ ├── sysext/platform.raw -│ └── etc/platform-etc.raw +│ ├── sysext/.raw +│ └── etc/.raw └── b/ ... -/run/extensions/platform.raw -> /a/sysext/platform.raw -/etc overlay lowerdir list += +/run/extensions/.raw -> /a/sysext/.raw +/etc overlay lowerdir list += .raw> ``` -The store is outside every search path, so its contents are invisible to -`systemd-sysext` until selected. Both slot directories persist across an update, -so rollback needs no re-download. +Keyed by `(kind, id)` from 0789 rather than by filename, so replacement and +comparison use the same identity the inventory does. The store is outside every +search path, so its contents are invisible to `systemd-sysext` until selected. +Both slot directories persist across an update, so rollback needs no +re-download. -### Servicing writes the store, atomically +### Servicing -The extensions subsystem already runs on clean install and A/B update via -`provision()`, documented as "migrate state from A-partition to B-partition (or -vice versa)". It resolves the target volume with -`EngineContext::get_ab_update_volume()`, already used to select the per-slot -verity addon, and writes into `//`. - -Placement must be atomic. A malformed `.raw` is not skipped like a version -mismatch: `image_read_metadata` errors propagate in -`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image -tolerance applies, so **one bad image blocks every extension on the host**. -Directory-type images are tolerated; `.raw` are not. +0789's extensions subsystem already computes the effective set and places +images. For a slotted entry the destination becomes `//`, with +the target volume from `EngineContext::get_ab_update_volume()`, already used to +select the per-slot verity addon. -Trident today falls back to a non-atomic `fs::copy` when `rename` fails across a -filesystem boundary (`crates/trident/src/subsystems/extensions/mod.rs:525-540`). -This is a prerequisite fix, not a consequence of this design. +What is deployed per slot is recorded in 0789's Deployed Extension Inventory. +This RFC adds no separate manifest: the inventory already carries kind, ID, +name, resolved placement, source and source hash, which is what the loader needs +to distinguish "nothing configured" from "storage lost". -### The store carries a manifest - -Globbing `*.raw` cannot distinguish "no extensions configured" from "storage -lost", "mount failed" or "update interrupted", and treating all of those as -success produces a quietly extension-less host. - -Each slot directory carries a manifest naming the expected images and hashes. An -absent manifest means the feature is not configured; a present manifest with -missing or unexpected content is an error. This also gives garbage collection -something to act on. +Atomic placement is a shared prerequisite. A malformed `.raw` is not skipped like +a version mismatch: `image_read_metadata` errors propagate in +`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image +tolerance applies, so one bad image blocks every extension on the host. ### Slot detection -The loader resolves the slot per platform, selected by a mode setting so that -Trident does not hardcode one platform's mechanism. The first mode reads a slot -label from the kernel command line, supplied by a per-slot signed boot artefact. +The loader resolves the slot per platform, selected by a mode setting so Trident +does not hardcode one platform's mechanism. The first mode reads a slot label +from the kernel command line, supplied by a per-slot signed boot artefact. A label is preferable to a partition UUID because no mapping table has to stay in sync with a layout that changes. However, a label is an *assertion*: a @@ -335,27 +353,26 @@ on a mismatch. ## Change inventory -Sizes are T-shirt estimates covering design and implementation separately. Risk -reflects how much is unknown rather than how much work there is. +T-shirt estimates covering design and implementation separately. Risk reflects +how much is unknown rather than how much work there is. Rows delivered by 0789 +are excluded. ### New component | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; logic is pure given cmdline, store and target, so unit-testable without root or a VM | +| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | | `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | -| Manifest format and verification | M | S | Low | Decide internal versus API-visible | ### Trident | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| **Atomic extension placement** | S | S | Low | Prerequisite bug fix; live defect today, independent of this RFC | -| `scope` and `extensionStore` API | S | S | Low | Additive, both default | -| `etcLayers` API | M | S | Med | New concept; shape depends on confext migration path | -| Relax `validate_extension_images_locations` | M | S | Med | Must test loader capability, not just A/B | -| Subsystem writes slotted entries per volume | S | M | Low | `provision()` already has the target volume | -| Manifest generation | S | S | Low | | +| `scope` on COSI `ExtensionImage` and HC `Extension` | S | S | Low | Additive; answers 0789's open question | +| `extensionStore` and `etcLayers` API | M | S | Med | `etcLayers` shape depends on the confext migration path | +| Relax `validate_extension_images_locations` | M | S | Med | Adds the slotted arm to 0789's resolved-placement validation | +| Slotted destination in the extensions subsystem | S | S | Low | 0789 already resolves placement; this adds one destination policy | +| Loader capability detection | M | S | Med | Needed by both RFCs' validation | | Package loader and dracut module in the RPM | S | S | Low | `%files acl` already ships a second binary | | Schema regeneration | S | S | Low | Mechanical | @@ -364,7 +381,7 @@ reflects how much is unknown rather than how much work there is. | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | | New dracut module to install and run the loader | S | S | Low | `dracut-acl-selinux-toggle` is an existing precedent | -| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount itself is one line, validation is the work | +| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount is one line, validation is the work | | Order the loader before the `/etc` overlay mount | S | S | Low | Same initrd window as `initrd-setup-root` | | Review `overlaycleanup()` interaction | M | S | Med | It compares only against the distro lower layer, so extension-provided defaults will not auto-prune | | Package catalog entry | S | S | Low | | @@ -376,25 +393,28 @@ reflects how much is unknown rather than how much work there is. | --- | --- | | systemd / Azure Linux | Nothing. Avoiding a systemd change is the point of this route | | AgentBaker, AKS RP | No interaction | -| Image Customizer | None unless RFC 0789 lands, and then only at the shared validation | +| Image Customizer | Covered by 0789, which already requires a writer emitting the section | ### Cross-cutting | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot; see [Rollout](#rollout) | +| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot | | Boot and rollback end-to-end tests | M | L | High | Unsupervised rollback cannot be asserted from unit tests | | SELinux policy | M | M | High | Extensions are already unsupported with SELinux enforcing | ### Suggested sequencing -1. **Atomic placement fix.** Standalone, lands independently, worth doing - regardless of this RFC. +1. 0789 steps 1 and 2: prerequisites, COSI 1.3, and reading and installing + bundled extensions. 2. Loader crate and dracut module, sysext half only, behind no API change. -3. Trident API and the validation relaxation. +3. `scope` plumbing and the validation relaxation. 4. `/etc` layer half, including the `bootengine` patch. Separable; ships after. 5. Rollout gating, once the per-slot-initrd question is settled. +The atomic-placement fix is a prerequisite of both RFCs and should land first +and independently; it is a live defect today. + ## Rollout If the initrd is shared between slots, loader capability is never @@ -404,42 +424,26 @@ If it is per-slot, a rollback into a slot whose initrd predates the loader has no selector, and every slotted extension is absent for that boot. Enabling `scope: slotted` then requires two phases: deploy a release carrying the loader and let it reach both slots, then permit slotted configuration. Validation must -verify that capability rather than assume it. - -This is an open question, listed below, and it changes the work in the table -above. - -## Relationship to RFC 0789 - -[0789](https://github.com/microsoft/trident/pull/789) carries extension images -in COSI files. The two are orthogonal: 0789 changes where the bytes come from, -this changes where they land and how they follow the slot. Either is useful -alone, and a bundled extension can be slotted. - -Both touch `validate_extension_images_locations`, in opposite directions: 0789 -tightens it into dynamic validation against the storage graph, this adds a -second way to satisfy it. Whichever lands first should leave the check expressed -as *is this destination slot-scoped*, so the other is an added arm rather than a -rewrite. +verify that capability rather than assume it, which is also what 0789 requires +of its destination validation. ## Testing -- Clean install: image in the correct slot directory, selection resolving, - merged after boot. +Beyond 0789's coverage: + +- A bundled extension with `scope: slotted`: placed in the correct slot + directory, selected at boot, merged. - A/B update: the selection repoints and **the merged content differs from the previous slot**. A presence-only assertion passes in the inert case and must not be used. - **Unsupervised rollback, with no Trident servicing operation in between**: the previous slot's content is merged again. This is the case the design exists for. -- `/etc` layer: a slotted layer is visible in `/etc`, `/etc` remains writable, - and a write to a file the layer provides persists across reboot. +- `/etc` layer: visible in `/etc`, `/etc` remains writable, and a write to a + file the layer provides persists across reboot. - Loader failure: boot does not proceed silently with a partial set. -- A malformed image in a slot directory is rejected before it is selected. - Validation: each reject row, plus a shared extension off an A/B volume still - rejected. -- Compatibility: a configuration with no `scope` places byte-identically to - today. + rejected, and `scope: slotted` rejected without loader capability. ## Counter-arguments @@ -457,8 +461,6 @@ rewrite. - **"Backport mutable confexts."** It exists, and it is 1310 lines across five files of someone else's component. This route reaches the same place without carrying that, and converges on native confexts later at zero payload cost. -- **"Doubles storage."** It does. The A/B volume route doubles it too, with the - allocation fixed at install. ### Rejected mechanisms @@ -466,14 +468,11 @@ rewrite. no fallback when that image fails OS matching, so they cannot hold per-slot copies. Verified empirically; Flatcar reached the same conclusion. - **`systemd-tmpfiles`.** No specifier identifies the booted device, partition - or slot, symlink targets are literal text plus specifier expansion, and the - main pass is ordered after the merge. + or slot, and the main pass is ordered after the merge. - **A systemd generator.** Generators run sandboxed with most of the filesystem read-only, and `/var` is not mounted when they run. - **Reordering the sysext merge ahead of the `/etc` overlay.** Exhausts the filesystem stacking budget and breaks confexts outright. -- **Repair after boot.** Any agent running at `multi-user.target` is far too - late to influence the merge. ### Prior art @@ -492,8 +491,8 @@ persisting it: - **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` and symlinks the winner into `/run/extensions`. -The counter-example is instructive: Fedora CoreOS activates sysexts statically -across deployments, and its documentation records that after a major rebase the +Fedora CoreOS is the counter-example: it activates sysexts statically across +deployments, and its documentation records that after a major rebase the extensions silently stop matching and are not loaded. ## Open questions @@ -503,25 +502,26 @@ extensions silently stop matching and are not loaded. 2. **Whole-directory selection or a prepared set?** Both commit atomically. The former is simpler; the latter allows shared and slotted extensions to share a search path. -3. **Manifest format**, and whether it is Trident-internal or API-visible. -4. **Retention.** Nothing prunes a slot directory when an entry leaves the - configuration. Reconciling in `provision()` conflicts with keeping the - previous slot's files for rollback. -5. **Default store path.** `/var/trident/extensions` against +3. **Retention.** Nothing prunes a slot directory when an entry leaves the + configuration. Reconciling during provisioning conflicts with keeping the + previous slot's files for rollback. 0789's inventory gives this something to + act on, but the policy is undecided. +4. **Default store path.** `/var/trident/extensions` against `/var/lib/trident/extensions`; the datastore already lives under the latter. +5. **Does a slotted extension need an extension-only update path?** 0789's + Runtime Update path writes to the running slot, which is in tension with + slot-scoped placement. 6. **Measured boot.** The selected image is outside the UKI's measurements, so - identical measurements can yield different `/usr` contents. Whether that is - acceptable depends on the platform's attestation claims. -7. **SELinux.** Extensions are already unsupported with SELinux enforcing, which - is a precondition this RFC does not resolve. + identical measurements can yield different `/usr` contents. +7. **SELinux.** Extensions are already unsupported with SELinux enforcing, a + precondition neither RFC resolves. ## Future possibilities - **Native confext mounting.** On systemd 256 or later, the loader places the selected payload in a confext search path and `systemd-confext --mutable=yes` - mounts it, replacing the custom lowerdir handling. The loader still performs - selection, and the payloads do not change provided they declare `ID=_any`. - Azure Linux 4.0 ships systemd 258.4. + mounts it. The loader still performs selection, and payloads do not change + provided they declare `ID=_any`. Azure Linux 4.0 ships systemd 258.4. - **Content matching** on platforms with systemd v256 or later and a build-granular `VERSION_ID`, which needs no loader at all. - **Initrd-phase sysext merging**, as Flatcar now does, for extensions that must From d424a2432037378aeac0d9d2d93d3b3e5f837238 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 14 Oct 2026 11:17:33 -0700 Subject: [PATCH 08/27] docs: avoid 'commit' for the loader's atomic activation 'commit' already names the post-boot servicing operation in Trident, so using it for the loader publishing its symlink set is a confusing collision. The loader has no relationship to that operation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index efbc05877b..b21c2f42c1 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -208,7 +208,8 @@ confext merged afterwards would reach 3 and fail. Creating N symlinks exposes a prefix of the intended set if it fails partway, which matters where a driver and its userspace libraries must move together. -The loader prepares the complete set and commits it with a single rename. +The loader prepares the complete set out of the way and activates it with a +single rename, so the set becomes visible all at once or not at all. ## Public API Design @@ -361,7 +362,7 @@ are excluded. | Change | Design | Impl | Risk | Notes | | --- | :-: | :-: | :-: | --- | -| `trident-sysext-loader` crate, slot detection and set commit | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | +| `trident-sysext-loader` crate, slot detection and atomic set activation | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | | `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | ### Trident @@ -499,7 +500,7 @@ extensions silently stop matching and are not loaded. 1. **Is the initrd per-slot or shared?** Decides whether a staged rollout is needed at all, and changes the estimates above. -2. **Whole-directory selection or a prepared set?** Both commit atomically. The +2. **Whole-directory selection or a prepared set?** Both activate atomically. The former is simpler; the latter allows shared and slotted extensions to share a search path. 3. **Retention.** Nothing prunes a slot directory when an entry leaves the From b43c3932e124098f24d8372da13e3af739f6777b Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 13:41:39 -0700 Subject: [PATCH 09/27] RFC: refresh managed extensions design Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-cosi-anatomy.svg | 96 ++ .../RFCs/0838-managed-extensions-overview.svg | 66 + .../0838-slot-loader-manifest.schema.json | 49 + docs/Development/RFCs/0838-slotted-sysexts.md | 1114 ++++++++++------- 4 files changed, 873 insertions(+), 452 deletions(-) create mode 100644 docs/Development/RFCs/0838-cosi-anatomy.svg create mode 100644 docs/Development/RFCs/0838-managed-extensions-overview.svg create mode 100644 docs/Development/RFCs/0838-slot-loader-manifest.schema.json diff --git a/docs/Development/RFCs/0838-cosi-anatomy.svg b/docs/Development/RFCs/0838-cosi-anatomy.svg new file mode 100644 index 0000000000..6ec2853103 --- /dev/null +++ b/docs/Development/RFCs/0838-cosi-anatomy.svg @@ -0,0 +1,96 @@ + + + Exploded view of the PoC COSI archive and its nested filesystem images + The tar starts with cosi-marker and metadata.json, + immediately followed by primary GPT data, then seven source partition images. + Two new extension image members come last. Metadata describes their paths, + sizes and compressed hashes. Expanding the ESP reveals the UKI and slot addons. + Expanding the sysext and confext members reveals separate squashfs filesystems + containing usr content and etc defaults. These extension members are not disk + partitions and are not nested inside the usr image. + + + + + + + + + b.cosi + 12 tar members, shown in archive order + + 1 cosi-marker + + 2 metadata.json + COSI 1.3: layout, references, hashes, sizes + + 3 images/..._gpt.raw.zst + + SOURCE DISK-REGION MEMBERS + ..._1.raw.zst ESP (boot files) + ..._2.raw.zst /usr filesystem + ..._3.raw.zst /usr verity hashes + ..._4.raw.zst USR-B source region + ..._5.raw.zst HASH-B source region + ..._6.raw.zst OEM + ..._7.raw.zst ROOT + Numbers describe the source disk layout. + + images/extensions/ (new tar members) + + trident-poc.raw.zst + Sysext: 415 B compressed → 4 KiB raw + + trident-poc-etc.raw.zst + Confext: 409 B compressed → 4 KiB raw + Schematic, not proportional to byte sizes. + + Metadata is the map + disk.gptRegions[] → GPT + partition members + images[] → filesystem and verity roles + extensions.{sysexts,confexts}[] → payload DDIs + Each descriptor: path, sizes, compressed SHA-384 + Also: architecture, os-release, bootloader and package metadata + + + Open ..._1.raw.zst: Zstd → FAT ESP image + /EFI/Linux/vmlinuz-6.6.157.1-1.azl3.efi + That UKI contains the kernel + initrd, including our loader. + /acl/uki-addons/slot-{a,b}.addon.efi + Boot files inside a partition image, not loose tar entries. + + + Open trident-poc.raw.zst: Zstd → squashfs DDI + /usr/lib/extension-release.d/extension-release.trident-poc + ID=_any · SYSEXT_ID=trident-poc + /usr/share/trident-poc/version → "slot-b" + The .raw is a filesystem image; these paths are inside it. + + + Open trident-poc-etc.raw.zst: Zstd → squashfs DDI + /etc/extension-release.d/extension-release.trident-poc-etc + ID=_any · CONFEXT_ID=trident-poc-etc + /etc/trident-poc/version → "slot-b" + /etc/trident-poc/user-override → "factory-b" + + Source partition numbers are not target slot choices; Trident maps data to the inactive pair. + Extension payloads are extra tar members, not new disk partitions. + diff --git a/docs/Development/RFCs/0838-managed-extensions-overview.svg b/docs/Development/RFCs/0838-managed-extensions-overview.svg new file mode 100644 index 0000000000..f185481e82 --- /dev/null +++ b/docs/Development/RFCs/0838-managed-extensions-overview.svg @@ -0,0 +1,66 @@ + + + COSI delivery, per-slot storage and two boot-time activation paths + Trident replaces the inactive slot's extension files in the + shared store, preserving the active slot. At each boot the initrd loader reads acl.slot + and that slot's slot-loader.json. It mounts the selected + confext as a lower layer in the writable etc overlay, and creates sysext links in run. + After switch-root, native systemd-sysext merges the selected system extension into usr. + + + + + + + + DELIVERY AND STAGING + + 1. COSI package + OS + extension images + + + 2. Trident update + Verify; replace target-slot content + + + /var/lib/trident/slots/{a,b}/ + slot-loader.json + sysexts/ + confexts/ + ON EVERY BOOT, INCLUDING ROLLBACK + + acl.slot=a|b + From the per-slot UKI addon + + + 3. trident-slot-loader + Reads acl.slot and slot-loader.json + + Read the selected slot's payloads + + + + 4. Read-only confext defaults + Extra lowers; factory defaults and user upper retained + First: initrd assembles writable /etc + + 5. Native /usr activation + /run/extensions links survive switch-root + Later: systemd-sysext merges /usr + diff --git a/docs/Development/RFCs/0838-slot-loader-manifest.schema.json b/docs/Development/RFCs/0838-slot-loader-manifest.schema.json new file mode 100644 index 0000000000..809a306c06 --- /dev/null +++ b/docs/Development/RFCs/0838-slot-loader-manifest.schema.json @@ -0,0 +1,49 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Trident Slot Loader Manifest v1", + "type": "object", + "additionalProperties": true, + "required": ["version"], + "properties": { + "version": { + "type": "integer", + "const": 1 + }, + "sysexts": { + "type": "array", + "default": [], + "uniqueItems": true, + "items": { + "$ref": "#/$defs/extension" + } + }, + "confexts": { + "type": "array", + "default": [], + "uniqueItems": true, + "items": { + "$ref": "#/$defs/extension" + } + } + }, + "$defs": { + "extension": { + "type": "object", + "additionalProperties": true, + "required": ["name"], + "properties": { + "name": { + "description": "Filename stem shared by .raw and extension-release..", + "type": "string", + "minLength": 1, + "not": { + "anyOf": [ + {"enum": [".", ".."]}, + {"pattern": "[/\\\\\\u0000-\\u001f\\u007f]"} + ] + } + } + } + } + } +} diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index b21c2f42c1..c08123e3d8 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,530 +1,740 @@ -# 0838 Slotted Extension Storage +# 0838 Trident-Managed Extensions -- Date: 2026-10-14 +- Date: 2026-10-15 - RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) -- Builds on: [RFC 0789](https://github.com/microsoft/trident/pull/789) +- Status: Proposal. ## Summary -[RFC 0789](https://github.com/microsoft/trident/pull/789) carries extension -images inside the COSI, so an OS-coupled extension ships and rolls out with the -OS. It leaves one question open: such an extension still has to land somewhere -that follows the A/B slot, and today the only way to satisfy that is a dedicated -A/B partition pair. +Provide an **A/B update path for sysexts on ACL**. Image builders bundle their +sysexts in the update COSI so Trident can stage, activate and roll them back +with the OS, rather than requiring a separate stack updater. An opt-in Host +Configuration policy controls storage and activation for sysexts and confexts, +while preserving shared user state. -This RFC answers it. A bundled extension can request Trident-managed, per-slot -storage, and a boot-time component selects the booted slot's copy before -anything consumes it. It also supplies `/etc` layers, which have no viable -mechanism on systemd 255. +## Motivation and Goals -An extension bundled in the COSI is already tied to the OS image it shipped -with. Slotted placement is the natural destination for it, and the two features -are expected to be used together in the common case. An independently hosted -extension fetched over OCI and placed per slot is the unusual combination, not -the default. +### Background -## Motivation and Goals +Systemd **system extensions (sysexts)** supply `/usr` and `/opt` content; +**configuration extensions (confexts)** supply `/etc` content. Here they are +Discoverable Disk Images (DDIs): mountable filesystem images containing those +files and compatibility metadata. -0789 establishes that OS-coupled extensions belong in the COSI. It also -establishes, in [Destination -Validation](https://github.com/microsoft/trident/pull/789), that a bundled image -written to shared storage changes the running slot immediately and defeats -rollback, so placement must be slot-scoped. - -Trident has exactly one mechanism for that today, and -`validate_extension_images_locations` (`config/host/mod.rs:218`) enforces it: the -destination must resolve to a device with A/B capabilities. The only way to -satisfy it is a dedicated A/B volume pair. Trident's own reference configuration -(`tests/e2e_tests/trident_configurations/root-verity/trident-config.yaml`) -spends four partitions and 2.4 GB on exactly this. - -That price buys only slot-scoping, and it is charged in the currency hardest to -change: - -- **Partitions are fixed at install.** The size is a guess made before anyone - knows what the host will carry; growing it means repartitioning. -- **Some layouts cannot pay it.** Where the partition table is published, signed - or attested, adding a pair is not a configuration change. A layout whose only - A/B volume is a verity-sealed `/usr` has none to offer on any terms. -- **It is all-or-nothing per directory.** An extension that should follow the - slot and one that should persist cannot share a location. - -The check conflates *being slot-scoped*, the property that matters, with *being -on an A/B block device*, one implementation of it. - -### Why content matching alone is not sufficient - -A tempting alternative is to place both slots' copies in one search directory -and let `systemd-sysext` merge whichever matches the booted OS. Version -mismatches are skipped quietly, so this very nearly works, and on a suitable -platform it needs no new code at all. - -It fails for two independent reasons, either sufficient on its own: - -1. **Deploying one image under two filenames requires systemd v256.** - `path_extract_image_name()` truncates an image name at the first `_` or `+` - (`src/basic/os-util.c`), letting two copies share one inner - `extension-release`. That function arrived in commit `37543971` (2024-03-28). - Earlier systemd requires each copy to carry its own, so a signed image must - be repacked per slot. -2. **The OS version must actually differ between slots.** Matching uses `ID=`, - then `SYSEXT_LEVEL=` when defined on both sides, otherwise `VERSION_ID=`. A - platform whose `VERSION_ID` is coarser than its build cadence, or which can - deploy the same image to both slots, puts identical values in both and both - copies match. - -Both conditions hold on the first intended consumer. Content matching remains -the better mechanism where a platform satisfies both; see -[Future possibilities](#future-possibilities). +A **Composable OS Image (COSI)** is a tar of OS partition images and metadata. +RFC 0789 proposes COSI 1.3 support for bundled extensions: optional metadata arrays +`extensions.sysexts` and `extensions.confexts` reference compressed DDIs by ID, +location, size and hash. OS and extensions can then ship as one artifact; +Trident can also fetch extensions listed separately in Host Configuration. +See the [example COSI cutaway](./0838-cosi-anatomy.svg). -### Goals +#### ACL storage and update behavior + +Azure Container Linux (ACL) already uses sysexts for containerd and OEM +integration and is expected to rely more on them. + +Only the `/usr` filesystem is A/B-paired and dm-verity-protected, with paired +hash partitions; **all mutable filesystems are shared**. ESP, OEM and ROOT are +unpaired, and `/etc`, `/var` and `/opt` live on shared ROOT. + +Images embedded in `/usr` follow its slot. For images on shared storage, +changing the stored files affects both slots; switching `/usr` does not undo +those changes. -Give bundled extensions a slot-scoped destination without a dedicated volume -pair; restore the previous set on rollback with no servicing operation; provide -a workable `/etc` story on systemd 255; require no new partitions, no -sealed-`/usr` change and no modification to systemd. +### Problem Statement -### Non-goals +Users build custom images on ACL, often adding stack files or preloaded +sysexts on mutable storage. ACL's A/B update replaces the core OS, not those +additions. Their stack therefore needs an out-of-band updater even though it +already ships in their image. -Changing `systemd-sysext` matching semantics. Replacing the A/B volume route, -which stays better where a pair is affordable. Extensions on the ESP. Anything -0789 already covers, including the effective set, the deployed inventory and -extension-only updates. +### Goals + +- Update and roll back ACL's OS and declared sysexts together, using one COSI + instead of an out-of-band stack updater. +- Activate the matching extension set before consumers run on every boot, + including rollback. +- Preserve user state while confext defaults follow the OS slot. + +## Scope + +**ACL is the initial design and implementation target** because its sealed +A/B `/usr` and shared mutable storage require explicit slot selection. +Support for other platforms is future work. The public API avoids ACL-specific +settings so it can be reused later. + +The mode selects how Trident consumes COSI-bundled extensions. This RFC +develops `managed` (alias `acl`) for ACL; `simple` and `native` describe direct +placement on writable A/B filesystems. + +Explicit Host Configuration `os.sysexts` and `os.confexts` declarations remain +honored in every mode. They retain existing placement behavior except in +`managed`/`acl`, which owns the combined COSI and Host Configuration set. + +### Requirements + +- Select bundled-extension handling through one public mode. +- A published store must contain valid manifests for every bootable slot; + images without a store need no initialization. +- Replace the inactive slot's managed extension set on reuse; do not retain + older sets for that slot. +- Preserve each platform's filesystem mutability and user-state persistence. +- Support ACL's shared storage without new partitions or systemd source changes. +- Keep the policy reusable on platforms with writable A/B storage. + +### Out of Scope + +- Image Customizer's managed-content authoring/bundling API and build design + (a required separate proposal; see [Dependencies](#dependencies)). +- Relocating the Trident-owned store; see [Future Possibilities](#future-possibilities). +- Updating arbitrary files on shared mutable storage. +- Runtime Update for managed extension/policy changes in the initial release. +- New image-less updates, initrd-scoped sysext content or ESP payload storage. +- Bundled disk streaming; 0789's exclusion remains. + +### Exit Criteria + +- A COSI file carrying extensions completes an ACL A/B update with the OS, + sysexts and confexts staged for the same target slot. +- Update, commit, health-check rollback and bootloader fallback expose matching + slot content before consumers run, without partial activation or late repair. +- Repeated slot reuse removes obsolete extension files without changing the + active slot or accumulating older sets. +- The public API, safe adoption and tests below are delivered; + user `/etc` edits and SELinux Enforcing are preserved. ## Dependencies -**RFC 0789 is a prerequisite.** This RFC adds a placement policy to the model -0789 defines and does not restate it. In particular it relies on: +- **[RFC 0789](https://github.com/microsoft/trident/pull/789):** COSI extension + metadata contract and reader/writer support. +- **Image Customizer (IC):** a separate, out-of-scope API and build design for + declaring extensions, maintaining the store and manifests, and bundling + extensions into COSIs. Required for the end-to-end image-building workflow; + Trident accepts a conforming COSI from any producer. -| From 0789 | Used here as | -| --- | --- | -| `extensions.sysexts` / `extensions.confexts` in COSI 1.3 | The delivery mechanism for slotted content | -| `(kind, id)` as stable extension identity | The store key, instead of a filename | -| Deployed Extension Inventory in Host Status | The record of what is deployed per slot; this RFC adds no separate manifest | -| Effective set computed from COSI plus Host Configuration | The set the loader selects from | -| Resolved-placement destination validation | The hook where the slotted arm is added | -| Atomic staging on the destination filesystem | A prerequisite shared by both RFCs | -| SELinux check moved into validation for every servicing type | Inherited unchanged | +## Implementation + +### Design + +This section describes `managed`/`acl` on ACL. + +Trident keeps the old and new extension files side by side. An update prepares +the inactive slot's files without changing the running system. At boot, +`trident-slot-loader` selects the files for the OS slot that actually booted. +Rollback uses the same selection process. + +![Managed extension delivery and activation on ACL](./0838-managed-extensions-overview.svg) + +#### Where the files live + +The [store](#managed-content-store) is fixed at `/var/lib/trident/slots`. Its `a/` and `b/` directories +hold each slot's sysexts and confexts on shared ROOT. These are separate +directories, not new partitions. If A is running, an update prepares B's set +while retaining A's files. **Reusing B replaces its entire managed extension +set**, just as staging a new OS replaces B's partition contents. Keep one set +per slot, not a history of sets within each slot. + +The [staged update](#trident-staging) takes effect on reboot into the target OS slot. + +#### Booting after an update + +The **[Slot Loader Manifest](#slot-loader-manifest)** tells the loader exactly which managed extensions +belong to the booted slot. +It is created when the store is initialized, during +[image construction](#base-image-construction-and-ic-integration) or first +servicing, and replaced during update staging. No additional COSI format change +is required. + +After reading `acl.slot`, the loader opens +`/sysroot/var/lib/trident/slots//slot-loader.json` in the initrd. +It uses this file, not Trident's SQLite database or a directory scan. + +The loader [returns success without activation](#loader-startup-behavior) when +the store is absent or a valid manifest selects no images. +A loader failure on an uncommitted update boot triggers +[diagnostics and reboot](#update-boot-failure-recovery) into the committed OS. + +#### Writable `/etc` + +User edits in the writable **upper** take precedence over [extension defaults](#confext-handling), +which take precedence over factory defaults. Changing OS slots changes the +extension defaults but keeps that shared upper, so user edits survive. +The native sysext merge still runs after `/etc` assembly. + +### Component deep dives -0789's prerequisites apply transitively, including the `set_up_extensions` -replacement bug and durable staged payloads. +#### Managed content store -Bundled extensions under -[disk streaming](../../Explanation/Disk-Streaming.md) are refused by 0789; that -restriction is inherited. +The managed content store is persistent storage for Trident-managed content +associated with an OS slot. In this design, it holds each slot's sysext/confext +`.raw` images and Slot Loader Manifest (`slot-loader.json`). The initial ACL +implementation uses only `/var/lib/trident/slots`, with separate `a/` and `b/` +directories on shared ROOT. -## Design +The path must resolve to ACL's shared ROOT filesystem. ROOT must be mounted +before the loader runs and writable during staging. Target-slot selection +comes from the host's A/B configuration, not source COSI partition numbers. -### Selection is recomputed every boot, never persisted +#### Base Image Construction and IC Integration -When firmware falls back to the previous slot on its own, no Trident code runs. -Anything written during servicing would be stale with nothing to correct it, so -the selection must derive from what actually booted. +ACL images may omit the managed content store entirely. **If a store is +present, it must contain valid manifests for every bootable slot**, even if +they select no images. `{"version": 1}` is a valid empty manifest. Boot must +not depend on Trident servicing or datastore initialization. -Verified: Trident's A/B rollback path invokes no subsystems. -`rollback::validate_boot` only reconciles Host Status, and -`manual_rollback::stage_rollback` touches only the pcrlock policy. Only the -runtime path passes `&mut subsystems`. +IC must support adding, replacing and removing managed payloads while +publishing their files and manifests together. Its API remains a +[separate proposal](#dependencies). -### Both halves run in the initrd +Integrating `trident offline-initialize` into ACL builds remains an independent +goal. A store/manifest initialization extension could let ACL scripts and IC +discover managed slot payloads in the image filesystem, validate them and write +manifests without requiring a servicing datastore or adopting unrelated +factory/operator extensions. -The `/etc` half has no choice. Where `/etc` is assembled as an overlay whose -lowerdir lives inside `/usr`, anything contributing to it must act before that -overlay is mounted, which happens in the initrd. +Any build-time initialization must finish before publishing the image. The +loader must not invoke initialization to repair a missing manifest. -The sysext half rides along, because `/run` is bind-moved across switch-root -(`src/shared/switch-root.c`: `{ "/run", MS_BIND, MS_BIND|MS_REC }`). Symlinks -written to `/run/extensions` in the initrd are still present when -`systemd-sysext` merges in the real root. Their targets are written as -post-switch-root paths, so they are dangling in the initrd and correct after. +#### Trident staging -Unifying is better than running them separately: the slot is resolved **once**, -so the sysext set and the `/etc` layer cannot disagree about which slot they -came from. It also removes complexity a real-root systemd unit would carry: +This sequence applies to `managed`/`acl`. Direct-placement modes follow the +[mode policies](#host-configuration) instead. -| Problem with a real-root unit | In the initrd | +Use the normal A/B lifecycle checks before modifying the target. If an +update has booted B but B is not committed, A is still the fallback and must +not be overwritten. + +1. Collect the COSI and Host Configuration extensions. Validate their identities, + paths and whether the host can activate them. + Each boot path's loader must understand its slot's manifest version. +2. Select the inactive slot. Remove any old Slot Loader Manifest and synchronize + its directory before clearing the Trident-owned extension directory, + including interrupted-staging leftovers. +3. Reserve space to stage the new extension files. Write temporary `.raw` files + inside the cleared directory, outside systemd's search paths. Verify hashes, + filesystem structure and extension compatibility. Reject invalid images with + an invalid-input error; flush valid files and directory entries to disk. +4. Build the manifest from the incoming extensions, not the old list. Once every + image is ready, write and synchronize `slot-loader.json.tmp`, atomically + rename it to `slot-loader.json`, then synchronize the slot directory. + +Boot reads only `slot-loader.json`; a leftover temporary file is never used. + +Finalization retains Trident's existing one-shot boot handoff. Before arming +the candidate, it verifies that persistent boot defaults still select the +committed OS and that the candidate supports the recovery path below. + +#### Trident Slot Loader + +##### Loader startup behavior + +| Condition | Result | | --- | --- | -| `WantedBy=` creates a `Wants=`, which `systemd.unit(5)` states cannot fail the transaction, so the unit does not fail closed | No unit; failing in dracut is native | -| Ordering is subtle: `ensure-sysext.service` carries `BindsTo=` and `After=systemd-sysext.service`, so it runs *after* the merge and is the wrong anchor | Nothing to order against | -| A platform-specific `ConditionKernelCommandLine=` would block every other mode | No unit condition | -| Unit enablement needs a preset or scriptlet | No unit to enable | -| Store must be available at `local-fs.target` | `/sysroot` is mounted by definition | - -A real-root unit remains the fallback for platforms that cannot modify their -initrd, so the mode selector chooses *where* the loader runs as well as how it -detects the slot. - -### Why `/etc` payloads are confext images - -`systemd-confext` is the natural mechanism and is unusable here: on systemd 255 -it mounts `/etc` read-only. Mutable overlays arrived in v256; Flatcar's backport -is 1310 lines across five files, and modifying systemd is out of scope. - -Keep the **format** and replace the **mount**. Payloads are confext-format DDIs -carrying `/etc/extension-release.d/extension-release.` -(`src/basic/os-util.c`). The loader mounts them read-only and adds them as -additional lowerdirs to the existing `/etc` overlay, whose upperdir is -unchanged, so `/etc` stays mutable. - -Preferred over tarballs or plain trees because: - -- **It is forward-compatible.** The payload is already a valid confext. Azure - Linux 4.0 ships systemd 258.4, where the identical artefact is handed to - native `systemd-confext --mutable=yes` and the custom mounting is deleted. - Two things do not change at that transition: the loader remains, because - systemd has no concept of A/B slots and selection stays the platform's job; - and native mounting reintroduces `extension-release` matching that the custom - lowerdir mount bypasses. -- **It reuses everything.** Same build and signing pipeline as sysexts, same - 0789 delivery path, verity available by DDI convention. -- **It is atomic.** A loop mount either appears or does not. A tarball - extraction can half-fail and pays extraction cost on every boot. -- **SELinux labels travel in the filesystem image**, not in tar metadata. - -#### Payloads must declare `ID=_any` - -Because slot selection is the platform's job and not systemd's, the payload must -not additionally be subject to OS-version matching. A payload declaring a -concrete `ID=` would merge under the custom mount, which ignores matching, then -silently stop merging the day a platform switches to native confext mounting. -That is the failure mode this RFC exists to prevent, reappearing at the point of -transition. - -`ID=_any` bypasses matching on every systemd version. Selection remains with the -loader, which is what slot-scoping requires anyway, and the move to native -mounting changes no artefact. Deciding this late would mean rebuilding every -payload, so it is specified here. 0789 already states that `ID=_any` must not be -warned on, since it declares compatibility rather than delivery method. - -Adding lowerdirs does not increase filesystem stacking depth, because depth is -the maximum of the layers plus one and every lowerdir sits on a plain -filesystem. *Reordering* the sysext merge ahead of the `/etc` overlay would -instead place `/etc` on top of the `/usr` overlay, reaching depth 2, and any -confext merged afterwards would reach 3 and fail. -`FILESYSTEM_MAX_STACK_DEPTH` is 2 (`include/linux/fs.h`), enforced in -`fs/overlayfs/super.c`. - -### Selection is a set, not a sequence - -Creating N symlinks exposes a prefix of the intended set if it fails partway, -which matters where a driver and its userspace libraries must move together. -The loader prepares the complete set out of the way and activates it with a -single rename, so the set becomes visible all at once or not at all. +| Store directory absent | Exit successfully (`0`); no activation. | +| Store present, manifest absent | Return nonzero; activation fails. | +| Valid manifest selects no images | Exit successfully (`0`); no activation. | +| Other lookup, parsing or selected-image errors | Return nonzero. | -## Public API Design +Check store presence before slot detection. Successful early exits do not +create or replace runtime links or mounts. `{"version": 1}` is a valid empty +manifest; unreadable or malformed files are not. +Nonzero results follow the [update-boot failure policy](#update-boot-failure-recovery). + +##### Loader configuration + +ACL packaging supplies fixed loader settings: + +```ini +LOADER_MODE=acl +TRIDENT_SYSEXT_STORE=/sysroot/var/lib/trident/slots +TRIDENT_SYSEXT_LINK_STORE=/var/lib/trident/slots +``` -### Answering 0789's open question +Both paths name the same store: `/sysroot/...` is the initrd's view; +`/var/lib/...` is the symlink destination after switching to the real root. +These settings are private. `LOADER_MODE` controls slot detection, not service +timing. -0789 asks how a COSI entry requests slotted placement, and states that omitting -`path` must not opt in. The answer is an explicit `scope`, spelled identically -on both sources so one concept has one name. +This RFC proposes a private kernel command-line input, +`trident.update_boot=1`, for an uncommitted A/B update boot. Absence means a +normal boot. It is boot configuration, not a manifest field or Host +Configuration option, so a missing or malformed manifest cannot hide it. -**COSI metadata**, extending 0789's `ExtensionImage`: +Trident stages a removable, prebuilt `trident-update.addon.efi` on the +candidate UKI's `.extra.d/` only, following the image's normal signing policy. +Factory and fallback entries do not carry it. Commit durably removes that copy +before any persistent boot-order/default promotion; subsequent boots +therefore use the normal failure policy. + +##### Slot Loader Manifest + +The [JSON Schema](./0838-slot-loader-manifest.schema.json) defines format v1 +as UTF-8 JSON: ```json { - "id": "gpu-driver", - "image": { "path": "images/extensions/gpu-driver.rawzst", "...": "..." }, - "scope": "slotted" + "version": 1, + "sysexts": [{"name": "runtime"}], + "confexts": [{"name": "runtime-defaults"}] } ``` -**Host Configuration**, extending `Extension`: +| Field | Required | Contract | +| --- | --- | --- | +| `version` | Yes | Integer `1`: file-format version, not an update or generation counter. | +| `sysexts`, `confexts` | No | Arrays of image objects; omitted arrays are treated as `[]`. Names must be unique per kind. | +| `name` | Yes, per item | Filename stem shared by the image `.raw` and metadata `extension-release.`. No directory separators, control characters, `.` or `..`. | + +Unknown fields are ignored in both the manifest and image objects. Compatible +optional fields can be added within v1; changes that old loaders must understand +require a new format version. + +An entry named `runtime` resolves to `/sysexts/runtime.raw` or +`/confexts/runtime.raw`, according to its array, relative to the store. +There are no caller-supplied image paths. + +Sysexts retain native systemd ordering. Trident writes confexts highest-priority +first using +[systemd's version-aware extension-name ordering](https://github.com/systemd/systemd/blob/v255/src/sysext/sysext.c#L869-L892); +the loader uses that array order for `/etc` lowerdirs, where the first wins. + +The loader rejects unsupported versions, missing required fields, invalid known +fields, duplicate known JSON keys and duplicate names within either kind, even if +ignored fields differ. It requires readable, regular image files confined to +the selected slot's directory and applies the +compatibility checks below before activation. Symlinks, missing image files and invalid +metadata fail activation; none trigger a fallback to directory scanning. + +##### Selecting extensions at boot + +1. **Existing:** ACL mounts ROOT in the initrd. +2. **New:** `trident-slot-loader.service` runs after that mount. The loader reads + `acl.slot=a|b`, loads the slot's manifest, validates its images and prepares + `/run/extensions` links and confext lowerdirs. Missing, invalid or duplicate + slot values are errors. +3. **Integration change:** the existing `initrd-setup-root.service` gains a + required dependency on the loader and consumes its lowerdir list when + creating the existing writable `/etc` overlay. +4. **Existing:** `/run` survives switch-root, and `systemd-sysext.service` merges + the sysexts in the real root. Afterwards, `ensure-sysext.service` reloads unit + discovery and restarts targets; it does not perform the merge itself. + +On rollback, the booted slot selects its old set before consumers run. No later +`trident commit` repair is needed. Sysext content is not available before the +native merge in step 4. + +Loader failure must stop root setup; activation failure must prevent a healthy +commit. Use required service dependencies, not `Wants=` alone or conditions that +quietly skip the loader. Preserve another component's activation entries or +reject the conflict; the exact shared-directory publication protocol remains open. + +##### Update-boot failure recovery + +The loader unit uses `Restart=no` and gains +`OnFailure=trident-slot-loader-recovery.service`. This new initrd-only recovery +unit uses `ConditionKernelCommandLine=trident.update_boot=1` and +`ConditionPathExists=/etc/initrd-release`, with `DefaultDependencies=no`. +It has no dependency on `/etc` assembly or Trident's datastore and runs with +a bounded timeout. Nonzero exits, crashes and timeouts take this path, not +just errors returned by the loader's main function. + +The handler captures diagnostics, then requests a reboot through systemd. +`SuccessAction=reboot` and `FailureAction=reboot` ensure that capture failure +or timeout does not leave the candidate stuck. It never re-arms the one-shot +entry or guesses the other slot: the consumed one-shot handoff returns the next boot +to the committed default. Normal boots keep the existing fatal-error behavior +without automatic reboot, preventing a fallback boot from entering a loop. + +Before reboot, the handler overwrites a bounded, human-readable record at +`/sysroot/var/lib/trident/diagnostics/slot-loader-last-failure.log` and +synchronizes it. This is shared ROOT, outside the slot directory that staging +clears; the old OS reads it at `/var/lib/trident/diagnostics/slot-loader-last-failure.log`. +Record the boot ID, attempted slot when known, failure phase, service result +and error reason, not manifest or image contents. Failure to persist the record +is reported to the journal/console but does not prevent reboot. If the loader +crashed before reporting a reason, capture its unit result and journal tail. + +Use leveled stderr diagnostics (slot, manifest path, store, activation method and +image errors), captured by the journal/console. Return nonzero on failure. +Install the loader's dracut module before normal UKI build and signing. + +##### Confext handling + +The loader mounts selected confext images read-only below +`/run/trident-slot-loader/etc/`. It publishes their `/etc` subdirectories as a +colon-separated lowerdir list in `/run/trident-slot-loader/etc-lowerdirs`, in +manifest order, only after all selected mounts are ready. + +The loader reuses confext images, not systemd 255's native `systemd-confext` +merge, which would make `/etc` read-only. It does not create the final overlay; +ACL Bootengine consumes the list. With no confexts, there are no extra lowerdirs. + +#### ACL Bootengine + +ACL already combines factory defaults with a persistent writable directory +using overlayfs. This RFC requires two integration changes: + +- The existing `initrd-setup-root.service` gains `Requires=` and `After=` + dependencies on `trident-slot-loader.service`. +- The existing `initrd-setup-root` command reads the published lowerdir list + before its overlay mount and prepends those paths to the factory lower. + +The resulting mount keeps the existing upper and work directories: + +```text +lowerdir=:/sysroot/usr/share/distro/etc +upperdir=/sysroot/etc +workdir=/sysroot/.etc-work +``` + +A missing or empty list leaves the existing factory-only lower unchanged. +Bootengine does not modify a live mount or copy defaults into the writable +directory. Each confext supplies a directory to the same overlay, not another +overlay mount. + +`overlaycleanup()` currently compares user files only with factory defaults. +How cleanup handles changed or removed extension defaults and deletion markers +(whiteouts) remains an open policy decision. + +### Safety checks + +| Area | Required behavior | Failure behavior | +| --- | --- | --- | +| Mode and layout | Validate selected inputs and placement before writing. `managed`/`acl` initially requires ACL A/B servicing and compatible boot components. | Reject unsupported layouts; never silently switch modes. Intentional skips are reported. | +| Native placement | Bundled `simple` and `native` images go only to writable A/B filesystems. Resolve the actual backing filesystem, including an overlay's writable upper. | Reject an unsafe explicit destination or an exhausted native candidate list before staging. | +| Ownership and cleanup | Clear only the inactive managed slot tree. Direct placement changes only Trident-tracked files, not entire native directories. Reject aliases, overlapping paths, traversal and untrusted writers. | Preserve the running/fallback slot and unrelated data; refuse unsafe cleanup. | +| Image validity | Validate delivery hashes and standard [`extension-release` compatibility](https://www.freedesktop.org/software/systemd/man/latest/os-release.html): distro, level/version, architecture and scope. `ID=_any` is allowed, not required. | Reject invalid selected images. Custom mounting does not bypass these checks. | +| Publication and activation | Keep temporary images undiscoverable and publish a complete managed manifest last. A valid empty manifest or absent store is a no-op; a missing manifest in an existing store is not. | Block root setup and healthy commit after activation failure; apply the update-boot recovery policy when marked. | +| Update recovery | Verify the committed default, one-shot support, recovery unit and marker addon before activation. Remove the marker before persistent boot promotion. | Refuse an unsafe handoff. Pending-boot failures record diagnostics and reboot once; normal-boot failures do not loop. | +| Writable-store trust | Delivery checks do not add boot-time hashes or authenticate the manifest. Signing, integrity and measurement remain platform policies; sealed `/usr` attestation does not cover the store. | Do not claim inherited trust. Loss of shared storage can affect both slots; A/B cannot restore it, and boot must not require a network download. | +| SELinux | Preserve Enforcing mode and correct payload/mount labels. | Surface access or labeling errors; never disable checks or exempt COSI inputs. | + +## Public API Design + +### Host Configuration ```yaml +image: + url: https://example.com/acl.cosi + sha384: os: - # Root of the per-volume extension store. - # Default /var/trident/extensions. Must be on storage an A/B update does not - # replace, and must not be a systemd-sysext search path. - extensionStore: /mnt/data/trident/extensions - - sysexts: - - url: https://example.com/tooling.raw # unchanged, still the default - sha384: - - url: https://example.com/platform.raw - sha384: - scope: slotted - - # confext-format images contributed as /etc overlay layers - etcLayers: - - url: https://example.com/platform-etc.raw - sha384: - scope: slotted + managedExtensions: acl ``` -| `scope` | Where the bytes live | Where it is activated | -| --- | --- | --- | -| `shared` (default) | At `path`. | At `path`. Today's behaviour. | -| `slotted` | `/{a,b}/`, keyed by `(kind, id)` | `/run/extensions`, or as an `/etc` lowerdir | +| Field | Type | Default | Meaning | +| --- | --- | --- | --- | +| `os.managedExtensions` | Enum | `disabled` | Bundled-extension intake and placement policy. | + +The proposed names distinguish filesystem-level slotting (`native`) from +Trident-managed logical slots on shared storage (`managed`). `acl` is an alias +for `managed`, not a separate mode. Aliases are normalized before comparison +and serialized as the canonical value, so changing only the spelling does not +trigger servicing. Boolean values are not mode values. + +| Value | COSI-bundled extensions | +| --- | --- | +| `disabled` (default) | Ignore bundled extensions: do not extract, verify or deploy their payloads. OS-image processing and explicit Host Configuration extensions are unchanged. | +| `simple`\* | Place an image at its requested `path` only when that destination is writable and A/B-slotted. Entries without `path` are skipped with a diagnostic; an invalid or unsafe explicit path fails validation. | +| `native`\* | Consume all bundled images. For each kind, choose the first writable, A/B-slotted native directory from the ordered list below. Incoming destination hints do not choose the location. | +| `managed` (alias `acl`) | Consume all bundled images and own their store, lifecycle and activation, overriding requested destinations. Trident supplies logical slot separation on shared storage; no writable filesystem-level A/B store is required. The initial ACL store is fixed at `/var/lib/trident/slots`. | -`scope` and `path` are mutually exclusive: a slotted extension surfaces in a -loader-owned location, so a caller-supplied destination would be silently -ignored. This satisfies 0789's requirement that placement never change -implicitly. +\* Future developments, not part of the initial implementation. -`etcLayers` is a separate list rather than a `scope` on `os.confexts` because -the two are activated by different mechanisms, only one of which is -`systemd-confext`. The COSI side needs no equivalent split: a confext entry with -`scope: slotted` is an `/etc` layer. +The mode applies to both sysexts and confexts. In `managed`/`acl`, explicit +Host Configuration URL/OCI images join the owned set and use managed +destinations; in other modes their existing explicit placement behavior is +retained. Unrelated factory/operator extensions are not adopted. -### The validation change +Ignoring bundled payloads does not waive normal COSI metadata/OS-image +validation. Intentionally ignored members are not reported as unused OS images. +An empty `managed`/`acl` set publishes an empty target manifest. -`validate_extension_images_locations` is relaxed, not removed. Its requirement -becomes *the destination is slot-scoped*, satisfied by either mechanism. This is -the arm 0789's Destination Validation anticipates: +### Native mode directory selection -| Configuration | Rule | +| Kind | Candidate order | | --- | --- | -| No `abUpdate` | No check, as today. | -| `abUpdate`, `scope: shared` | Destination must be on an A/B volume. Unchanged. | -| `abUpdate`, `scope: slotted` | Satisfied by the store, **only when loader capability is present**. | +| Sysext | `/var/lib/extensions`, then `/etc/extensions` | +| Confext | `/var/lib/confexts`, then `/usr/lib/confexts`, then `/usr/local/lib/confexts` | + +The primary candidates retain Trident's existing default directories. +Ephemeral `/run` locations and initrd-scoped `/.extra/sysext` are not candidates. +An eligible directory exists or can be created on the target slot's writable +A/B filesystem without breaking image protection. Evaluate both kinds before +writing; if a consumed kind has no eligible destination, fail the update. +`native` never falls back to `managed` implicitly. On ACL, shared ROOT and +sealed `/usr` provide no eligible native destination. + +`simple` and `native` only place files in the target filesystem if it is +A/B-slotted at a partition level. The image's existing boot setup activates +them: standard OS discovery for +`native`, or image-provided configuration for `simple`. Trident Slot Loader +does not activate these files. + +### Identity and layout + +For consumed images, use `(kind, id)` for update identity and verify it against +the DDI's `SYSEXT_ID` or `CONFEXT_ID`. Derive `.raw` from +`extension-release.`; ID and filename need not be equal. + +```text +/var/lib/trident/slots/{a,b}/{sysexts,confexts}/.raw +``` + +Record both ID and name. Reject duplicate IDs per kind and conflicting discovery +names/destinations, including existing activation entries. No source wins a +collision silently. In `managed`/`acl`, each slot's directory is exclusively +Trident-owned and is cleared when that slot is reused. The direct modes track +their deployed files without owning the surrounding native directory. -0789 states the same constraint from its side: slotted storage qualifies only -with loader capability, and omitting `path` must not bypass the A/B check. +## Testing and Metrics ### Validation -| Condition | Result | -| --- | --- | -| `scope` absent | `shared`. Identical to today. | -| `scope: slotted` with `path` | Reject. The destination is loader-owned. | -| `scope: slotted`, no `abUpdate` | Reject, naming the absent configuration. | -| `scope: slotted`, no loader capability | Reject. See [Rollout](#rollout). | -| `extensionStore` inside a search path | Reject: both slots' copies would be discovered. | -| `extensionStore` on an A/B volume | Reject: it must outlive the slot it describes. | -| `etcLayers` entry without confext metadata | Reject at deploy time, after mount. | -| `etcLayers` entry without `ID=_any` | Reject; see above. | +A proof of concept combining the core managed-extension, A/B update and +writable `/etc` mechanisms was successfully executed on ACL. -Duplicate `(kind, id)` and destination collisions are already covered by 0789 -and are not restated. +### Required tests -### Backward compatibility +For the initial ACL implementation, cover: -All new fields default, so existing Host Configurations and 1.3 COSI files -without `scope` behave identically. The relaxed check is strictly more -permissive, so anything valid today stays valid. +#### API and intake -## Implementation +- `managed`/`acl` equivalence and canonical serialization; `disabled` ignores + bundled payloads without unused-image errors. Unsupported modes and Boolean + inputs fail, while explicit URL/OCI behavior remains unchanged by default. +- Both input sources, destination overrides, IDs distinct from filenames, + renames and cross-source collisions; unrelated extensions remain untouched. +- Compressed COSI and raw Host Configuration hashes, malformed images and + extension compatibility failures are checked before publication. -``` -/ -├── a/ -│ ├── sysext/.raw -│ └── etc/.raw -└── b/ ... - -/run/extensions/.raw -> /a/sysext/.raw -/etc overlay lowerdir list += .raw> -``` +#### Store lifetime and publication + +- Fixed ROOT placement and mount ordering; refuse writes through aliases, + traversal or unowned paths, or cleanup of a still-needed fallback slot. +- Repeated A/B reuse, additions/removals, empty sets, retries and mode changes: + only the target changes, space stays bounded by two slot sets, and retained + rollback data remains intact. Non-owned activation entries are preserved or + rejected. +- Interrupt initial store creation, staging and manifest publication: + retained slots remain bootable, temporary manifests are never consumed, + and no partial set becomes active. -Keyed by `(kind, id)` from 0789 rather than by filename, so replacement and -comparison use the same identity the inventory does. The store is outside every -search path, so its contents are invisible to `systemd-sysext` until selected. -Both slot directories persist across an update, so rollback needs no -re-download. +#### Loader and ACL integration -### Servicing +- Manifest versions/types and required fields, duplicate known keys/names + (including different ignored fields), unknown optional fields, unsafe names + and symlinks; omitted arrays and version-only manifests are valid. +- No store or a valid empty set exits successfully without links or mounts. + A present store with missing/malformed/unreadable metadata or selected images + fails rather than activating a partial set. +- ROOT/loader/Bootengine ordering, factory-only fallback for missing/empty + lowerdir lists, and blocked root setup after loader failure. +- Multiple `/etc` layers, precedence, copy-up, whiteouts and removal; user edits + survive update and rollback. Assert visible content, not just file existence. -0789's extensions subsystem already computes the effective set and places -images. For a slotted entry the destination becomes `//`, with -the target volume from `EngineContext::get_ab_update_volume()`, already used to -select the per-slot verity addon. +#### Update failure and rollback -What is deployed per slot is recorded in 0789's Deployed Extension Inventory. -This RFC adds no separate manifest: the inventory already carries kind, ID, -name, resolved placement, source and source hash, which is what the loader needs -to distinguish "nothing configured" from "storage lost". +- Loader errors, crashes and timeouts return a pending boot to the committed + OS without a loop; normal no-ops do not reboot. Include bootloader fallback, + failed commit and power loss in both slot directions. +- Marker cleanup precedes persistent boot promotion; cleanup failure preserves + the previous default. +- Failure records survive outside the reused slot. Read-only ROOT, capture + errors and timeouts do not prevent the rollback reboot. -Atomic placement is a shared prerequisite. A malformed `.raw` is not skipped like -a version mismatch: `image_read_metadata` errors propagate in -`image_discover_and_read_metadata` (`src/sysext/sysext.c`) before any per-image -tolerance applies, so one bad image blocks every extension on the host. +#### Image construction and compatibility -### Slot detection +- Base/customized images boot and reboot before datastore initialization, + with absent, empty or populated stores. Construction edits keep payloads and + manifests consistent without adopting unrelated images. +- Old/new loader, manifest, boot-context and Host Configuration reader + combinations preserve supported rollback paths; unsafe adoption is refused. -The loader resolves the slot per platform, selected by a mode setting so Trident -does not hardcode one platform's mechanism. The first mode reads a slot label -from the kernel command line, supplied by a per-slot signed boot artefact. +#### Performance -A label is preferable to a partition UUID because no mapping table has to stay -in sync with a layout that changes. However, a label is an *assertion*: a -packaging error could pair it with the other slot's device. The loader -cross-checks it against the booted `/usr` backing device and refuses to proceed -on a mismatch. +- Matched cold/warm benchmarks for the scenarios and budgets below, including + image-count scaling and end-to-end critical-path measurements. -## Change inventory +### Metrics -T-shirt estimates covering design and implementation separately. Risk reflects -how much is unknown rather than how much work there is. Rows delivered by 0789 -are excluded. +| Scenario | Provisional p95 goal | What is measured | +| --- | --- | --- | +| No-op: absent store or valid empty manifest | Less than 5 ms added latency | Service/process startup and lookup or empty-manifest handling; no links or mounts. | +| Sysext-only | Less than 10 ms added latency | All loader work before the existing native sysext merge. The unchanged OS merge is not charged as new loader work. | +| Confext-only or mixed | Less than 10 ms coordination time, plus measured image preparation | Separate startup, selection and publication from image mounting/validation. Report the full time until `/etc` is ready; its total budget is TBD after representative measurements. | + +Confext handling owns work that systemd would normally perform, so its total +time is inherently higher than selection alone and is not subject to the +sysext-only 10 ms limit. Do not report coordination time as the whole impact. + +Compare matched ACL boots with equivalent hardware, kernel, services and +payloads. Measure added boot-critical-path delay, not just process CPU time; +report cold/warm median and p95, preparation time and image counts. The no-op +5 ms goal is a starting budget, and none of these targets is yet demonstrated. -### New component +Record staging, activation and rollback outcomes with slot, selected image names, +activation method and failure reason in servicing logs and boot diagnostics. +Correlate a recovered update failure with its persistent record's boot ID. +Success requires matching visible content and no healthy commit after activation +failure. -| Change | Design | Impl | Risk | Notes | -| --- | :-: | :-: | :-: | --- | -| `trident-sysext-loader` crate, slot detection and atomic set activation | S | M | Low | std plus `anyhow`; pure given cmdline, store and inventory, so unit-testable without root or a VM | -| `/etc` layer mounting and lowerdir emission | M | M | Med | Loop-mount in the initrd; interaction with the existing overlay needs care | +## Servicing -### Trident +### Introduction and opt-in -| Change | Design | Impl | Risk | Notes | -| --- | :-: | :-: | :-: | --- | -| `scope` on COSI `ExtensionImage` and HC `Extension` | S | S | Low | Additive; answers 0789's open question | -| `extensionStore` and `etcLayers` API | M | S | Med | `etcLayers` shape depends on the confext migration path | -| Relax `validate_extension_images_locations` | M | S | Med | Adds the slotted arm to 0789's resolved-placement validation | -| Slotted destination in the extensions subsystem | S | S | Low | 0789 already resolves placement; this adds one destination policy | -| Loader capability detection | M | S | Med | Needed by both RFCs' validation | -| Package loader and dracut module in the RPM | S | S | Low | `%files acl` already ships a second binary | -| Schema regeneration | S | S | Low | Mechanical | +The default is no new behavior: omitted means `disabled`, existing OS +servicing and explicit URL/OCI declarations are unchanged, and bundled images +are not adopted. Images without a store boot normally without Trident +initialization. -### ACL +Customers opt in with `os.managedExtensions: managed` or its `acl` alias: -| Change | Design | Impl | Risk | Notes | -| --- | :-: | :-: | :-: | --- | -| New dracut module to install and run the loader | S | S | Low | `dracut-acl-selinux-toggle` is an existing precedent | -| Patch `bootengine` to add `/etc` lowerdirs | S | M | Med | Third patch alongside two already carried; the mount is one line, validation is the work | -| Order the loader before the `/etc` overlay mount | S | S | Low | Same initrd window as `initrd-setup-root` | -| Review `overlaycleanup()` interaction | M | S | Med | It compares only against the distro lower layer, so extension-provided defaults will not auto-prune | -| Package catalog entry | S | S | Low | | -| Initrd size and UKI rebuild | S | S | Med | Re-signing already happens every build; size growth is the open part | +1. Supply an ACL image with loader, Bootengine and recovery support. Preflight + checks the target and retained rollback paths. +2. On first use, publish the store only after retained slots have valid + manifests, empty where appropriate. +3. Stage the target and attempt a one-shot boot. The committed OS stays the + fallback until health checks and commit succeed. -### Not affected +### Compatibility policy -| Component | Why | +Keep the loader and its boot contracts small, stable and long-lived. + +| Interface | Compatibility approach | | --- | --- | -| systemd / Azure Linux | Nothing. Avoiding a systemd change is the point of this route | -| AgentBaker, AKS RP | No interaction | -| Image Customizer | Covered by 0789, which already requires a writer emitting the section | - -### Cross-cutting - -| Change | Design | Impl | Risk | Notes | -| --- | :-: | :-: | :-: | --- | -| Rollout and capability gating | M | S | Med | Depends on whether the initrd is per-slot | -| Boot and rollback end-to-end tests | M | L | High | Unsupervised rollback cannot be asserted from unit tests | -| SELinux policy | M | M | High | Extensions are already unsupported with SELinux enforcing | - -### Suggested sequencing - -1. 0789 steps 1 and 2: prerequisites, COSI 1.3, and reading and installing - bundled extensions. -2. Loader crate and dracut module, sysext half only, behind no API change. -3. `scope` plumbing and the validation relaxation. -4. `/etc` layer half, including the `bootengine` patch. Separable; ships after. -5. Rollout gating, once the per-slot-initrd question is settled. - -The atomic-placement fix is a prerequisite of both RFCs and should land first -and independently; it is a live defect today. - -## Rollout - -If the initrd is shared between slots, loader capability is never -slot-dependent and no staged rollout is required. - -If it is per-slot, a rollback into a slot whose initrd predates the loader has -no selector, and every slotted extension is absent for that boot. Enabling -`scope: slotted` then requires two phases: deploy a release carrying the loader -and let it reach both slots, then permit slotted configuration. Validation must -verify that capability rather than assume it, which is also what 0789 requires -of its destination validation. - -## Testing - -Beyond 0789's coverage: - -- A bundled extension with `scope: slotted`: placed in the correct slot - directory, selected at boot, merged. -- A/B update: the selection repoints and **the merged content differs from the - previous slot**. A presence-only assertion passes in the inert case and must - not be used. -- **Unsupervised rollback, with no Trident servicing operation in between**: the - previous slot's content is merged again. This is the case the design exists - for. -- `/etc` layer: visible in `/etc`, `/etc` remains writable, and a write to a - file the layer provides persists across reboot. -- Loader failure: boot does not proceed silently with a partial set. -- Validation: each reject row, plus a shared extension off an A/B volume still - rejected, and `scope: slotted` rejected without loader capability. - -## Counter-arguments - -- **"Just add the partitions."** Better where possible: the filesystem provides - the isolation, nothing runs at boot, and capacity is contained. This is for - layouts that cannot. Directory slotting gives equivalent naming, not - equivalent failure containment: a store on shared storage can fill and affect - the running slot, which a dedicated pair cannot. -- **"Put it in `/usr`."** Better still, and unavailable without re-sealing - dm-verity and re-signing. Upstream also forbids extension storage under - `/usr`: a `lowerdir=` that is a child of another fails with `-ELOOP`. -- **"Use the ESP."** Genuinely slot-scoped via `.efi.extra.d/`, but ESPs - are 128 to 256 MiB against payloads in the tens to hundreds of megabytes, and - `systemd-stub` places them in the initrd. -- **"Backport mutable confexts."** It exists, and it is 1310 lines across five - files of someone else's component. This route reaches the same place without - carrying that, and converges on native confexts later at zero payload cost. - -### Rejected mechanisms - -- **`.v` versioned directories.** They select the highest version and stop, with - no fallback when that image fails OS matching, so they cannot hold per-slot - copies. Verified empirically; Flatcar reached the same conclusion. -- **`systemd-tmpfiles`.** No specifier identifies the booted device, partition - or slot, and the main pass is ordered after the merge. -- **A systemd generator.** Generators run sandboxed with most of the filesystem - read-only, and `/var` is not mounted when they run. -- **Reordering the sysext merge ahead of the `/etc` overlay.** Exhausts the - filesystem stacking budget and breaks confexts outright. - -### Prior art - -Three shipping systems converge on recomputing the selection at boot rather than -persisting it: - -- **Flatcar**, whose layout is closest to the first consumer, stamps the version - into the filename on shared storage and rewrites the activating symlink in the - initrd from the booted `/usr`'s `os-release`. It has since moved extension - mounting wholesale into the initrd and deleted its late-merge workaround. -- **ChromeOS** stores `dlc_a`/`dlc_b` on shared stateful storage and derives the - slot from the booted partition number on every service start. It clears the - inactive slot's validity marker *before* writing and re-sets it after, so a - rollback into a half-written slot finds an honest failure rather than a - plausible file. Worth adopting. -- **Fedora's `sysexts-manager`** filters candidates by the running `VERSION_ID` - and symlinks the winner into `/run/extensions`. - -Fedora CoreOS is the counter-example: it activates sysexts statically across -deployments, and its documentation records that after a major rebase the -extensions silently stop matching and are not loaded. - -## Open questions - -1. **Is the initrd per-slot or shared?** Decides whether a staged rollout is - needed at all, and changes the estimates above. -2. **Whole-directory selection or a prepared set?** Both activate atomically. The - former is simpler; the latter allows shared and slotted extensions to share a - search path. -3. **Retention.** Nothing prunes a slot directory when an entry leaves the - configuration. Reconciling during provisioning conflicts with keeping the - previous slot's files for rollback. 0789's inventory gives this something to - act on, but the policy is undecided. -4. **Default store path.** `/var/trident/extensions` against - `/var/lib/trident/extensions`; the datastore already lives under the latter. -5. **Does a slotted extension need an extension-only update path?** 0789's - Runtime Update path writes to the running slot, which is in tension with - slot-scoped placement. -6. **Measured boot.** The selected image is outside the UKI's measurements, so - identical measurements can yield different `/usr` contents. -7. **SELinux.** Extensions are already unsupported with SELinux enforcing, a - precondition neither RFC resolves. - -## Future possibilities - -- **Native confext mounting.** On systemd 256 or later, the loader places the - selected payload in a confext search path and `systemd-confext --mutable=yes` - mounts it. The loader still performs selection, and payloads do not change - provided they declare `ID=_any`. Azure Linux 4.0 ships systemd 258.4. -- **Content matching** on platforms with systemd v256 or later and a - build-granular `VERSION_ID`, which needs no loader at all. -- **Initrd-phase sysext merging**, as Flatcar now does, for extensions that must - be present before unit enumeration. -- Per-volume scoping for other Trident-managed state that is shared today. +| Host Configuration | The enum allows future modes without changing the field shape. Initially support `disabled` and `managed`/`acl`; reject unsupported values. Canonicalize aliases and omit defaults from serialized state. | +| Loader and manifest | New loaders retain older released format readers. Old loaders ignore optional unknown fields within supported versions; incompatible semantics need a new version that they reject. | +| Per-slot state | Write each slot's manifest in a format its loader supports. Preserve the retained slot's files and format; never convert it to a new-only version. | +| Boot and store contracts | Keep paths, update markers and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | + +Rollback needs compatible Trident state readers as well as a compatible +loader: older strict `Os` readers may reject new fields or values. + +### Preserving rollback capability + +**TBD:** the exact compatibility matrix and admission/upgrade checks that prove +rollback remains usable. Prefer an exceptionally stable loader, maintained +backward compatibility and automated old/new boot tests. Block changes whose +retained rollback path is incompatible. + +### Policy changes + +Mode and managed-set changes initially use A/B servicing; alias spelling alone +does not trigger one. Preserve retained slots until the transition completes. +Normal inactive-slot reuse replaces its old set rather than retaining history. + +If an owned store already exists and the target mode is not `managed`/`acl`, +clear the reused target's owned tree and publish an empty manifest. This avoids +stale activation without consuming disabled bundles or wiping native +directories. Migrate legacy aliases safely: deleting them can break rollback, +while leaving them can shadow `/run`. Refuse unsafe transitions and leave +unrelated entries untouched. + +Reject unsupported Runtime Update requests, without promotion or partial application. +Explicit Host Configuration inputs keep 0789's servicing constraints; +`simple`/`native` remain deferred. + +## Implementation Plan + +Initial delivery is `managed`/`acl` on ACL, with `disabled` as the default +for bundled extensions. `simple`, `native`, configurable stores and native +mutable-confext migration remain future work. ACL integration of +`trident offline-initialize` is an independent goal, not a prerequisite for +unmanaged-image boot. + +RFC 0789 reader/writer support is a prerequisite; its shared work and the +separate IC authoring API are not estimated here. Sizes below are relative +design/implementation effort (S/M/L), not elapsed-time commitments. + +| Phase | Work and completion gate | Size | Main risk | +| --- | --- | :-: | --- | +| 1. Contracts and compatibility | Freeze the initial API, manifest and boot interfaces; define the rollback compatibility matrix and admission rules. | M/M | High: retained readers | +| 2. Managed staging and lifecycle | Validate inputs; implement fixed-ROOT placement, atomic first-use publication, inactive-slot replacement and safe disable transitions. | M/L | High: ownership and interrupted writes | +| 3. Loader and ACL integration | Implement selection, links and confext preparation; integrate Bootengine ordering/lowerdirs and package the dracut assets before UKI signing. | M/M | High: early-boot integration | +| 4. Update-boot recovery | Add the candidate marker, one-shot fallback, persistent diagnostics and marker removal before commit promotion; prove fallback cannot loop. | M/M | High: commit and failure ordering | +| 5. Rollout and fault validation | Apply target/retained-reader preflight and execute the grouped functional, interruption, compatibility and rollback tests. | L/L | High: rollback viability | +| 6. Performance validation | Measure no-op and sysext-only budgets plus complete confext cost; publish cold/warm results and settle the confext total budget. | S/M | Medium: hardware and image-count variance | + +Production enablement requires passing the initial test areas, demonstrated +rollback compatibility and published measurements against the scenario budgets. +Unmet performance goals need explicit review; the compatibility-enforcement +TBD must be resolved before that gate. No new COSI fields, systemd source +changes or cluster-control-plane integration are required. + +## Counter-Arguments + +### Drawbacks + +- The shared store is a failure domain for both slots and needs space for both + slot sets. +- The loader and versioned manifests add early-boot and rollout complexity. +- Confext layering requires `/etc` reconciliation and integrity policies. + +### Alternatives + +- **Compatibility-based selection:** versions may match, and portable images may + be compatible with both slots. +- **Version selection (`.v` or "newest" links):** selects a version, not the + booted OS slot. +- **New dedicated A/B extension partitions:** valid, but require reserving disk + capacity for both slots upfront. Shared ROOT lets the two sets use available + filesystem space as needed. +- **ESP payload storage:** unnecessary when the existing ROOT can hold the images. +- **Systemd backports or changing ACL's merge order:** neither is required by + this design. + +## Open Questions + +1. Shared activation-path publication and legacy alias migration. +2. `/etc` reconciliation and integrity/attestation policy. +3. Whether managed sysext-only Runtime Update belongs in the first release; + this proposal starts with A/B servicing. +4. How rollback compatibility is admitted and enforced across loader, manifest, + boot-context and servicing-state reader versions. + +## Future Possibilities + +### Configurable store location + +A configurable `os.managedContentStore` path could support other storage +layouts later. It would require early-mount dependencies, compatible loader +configuration and rollback-safe migration of both slot sets. The initial ACL +implementation supports only `/var/lib/trident/slots` on ROOT. + +### Direct placement modes + +Future `simple` and `native` implementations need placement tests for pathless +entries, unsafe explicit paths, candidate ordering/exhaustion, writable +A/B-partition detection and direct activation without the loader. + +### Native mutable-confext integration + +Newer ACL can replace custom mounting with native mutable confext integration +without changing the API or payload format. Verify precedence, upper state, +labels, compatibility and ordering; version alone is insufficient. + +### Runtime updates + +Managed sysext-only Runtime Update can follow; ACL cannot replace its `/etc` +lowerdirs live. + +### Btrfs-backed managed store + +On Btrfs-backed storage, the store could use a pair of subvolumes, one per +slot, to simplify inactive-slot replacement without changing paths or the +public API. + +### Other platforms + +Non-ACL backends can reuse the policy once validated against their storage +and activation models. + +A future `managed` backend could prepare links in writable A/B extension +search directories during staging. Switching OS slots then selects those +links without a loader. The owned images may live on shared or slotted +storage, but the links must still resolve after reboot. + +Choose separately for sysexts and confexts. Native confext integration can use +slotted links or boot-selected `/run/confexts` entries if it preserves the +platform's `/etc` mutability and user-state persistence. Prefer existing +writable A/B directories in a fixed, platform-defined order. From f191b6b6b04aa572822858198ca7c35d0c5c255b Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 14:07:58 -0700 Subject: [PATCH 10/27] RFC: group supporting assets and flag review sections Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-cosi-anatomy.svg | 96 ------------------- docs/Development/RFCs/0838-slotted-sysexts.md | 15 ++- .../0838-managed-extensions-overview.svg | 0 .../0838-slot-loader-manifest.schema.json | 0 4 files changed, 12 insertions(+), 99 deletions(-) delete mode 100644 docs/Development/RFCs/0838-cosi-anatomy.svg rename docs/Development/RFCs/{ => 0838-slotted-sysexts}/0838-managed-extensions-overview.svg (100%) rename docs/Development/RFCs/{ => 0838-slotted-sysexts}/0838-slot-loader-manifest.schema.json (100%) diff --git a/docs/Development/RFCs/0838-cosi-anatomy.svg b/docs/Development/RFCs/0838-cosi-anatomy.svg deleted file mode 100644 index 6ec2853103..0000000000 --- a/docs/Development/RFCs/0838-cosi-anatomy.svg +++ /dev/null @@ -1,96 +0,0 @@ - - - Exploded view of the PoC COSI archive and its nested filesystem images - The tar starts with cosi-marker and metadata.json, - immediately followed by primary GPT data, then seven source partition images. - Two new extension image members come last. Metadata describes their paths, - sizes and compressed hashes. Expanding the ESP reveals the UKI and slot addons. - Expanding the sysext and confext members reveals separate squashfs filesystems - containing usr content and etc defaults. These extension members are not disk - partitions and are not nested inside the usr image. - - - - - - - - - b.cosi - 12 tar members, shown in archive order - - 1 cosi-marker - - 2 metadata.json - COSI 1.3: layout, references, hashes, sizes - - 3 images/..._gpt.raw.zst - - SOURCE DISK-REGION MEMBERS - ..._1.raw.zst ESP (boot files) - ..._2.raw.zst /usr filesystem - ..._3.raw.zst /usr verity hashes - ..._4.raw.zst USR-B source region - ..._5.raw.zst HASH-B source region - ..._6.raw.zst OEM - ..._7.raw.zst ROOT - Numbers describe the source disk layout. - - images/extensions/ (new tar members) - - trident-poc.raw.zst - Sysext: 415 B compressed → 4 KiB raw - - trident-poc-etc.raw.zst - Confext: 409 B compressed → 4 KiB raw - Schematic, not proportional to byte sizes. - - Metadata is the map - disk.gptRegions[] → GPT + partition members - images[] → filesystem and verity roles - extensions.{sysexts,confexts}[] → payload DDIs - Each descriptor: path, sizes, compressed SHA-384 - Also: architecture, os-release, bootloader and package metadata - - - Open ..._1.raw.zst: Zstd → FAT ESP image - /EFI/Linux/vmlinuz-6.6.157.1-1.azl3.efi - That UKI contains the kernel + initrd, including our loader. - /acl/uki-addons/slot-{a,b}.addon.efi - Boot files inside a partition image, not loose tar entries. - - - Open trident-poc.raw.zst: Zstd → squashfs DDI - /usr/lib/extension-release.d/extension-release.trident-poc - ID=_any · SYSEXT_ID=trident-poc - /usr/share/trident-poc/version → "slot-b" - The .raw is a filesystem image; these paths are inside it. - - - Open trident-poc-etc.raw.zst: Zstd → squashfs DDI - /etc/extension-release.d/extension-release.trident-poc-etc - ID=_any · CONFEXT_ID=trident-poc-etc - /etc/trident-poc/version → "slot-b" - /etc/trident-poc/user-override → "factory-b" - - Source partition numbers are not target slot choices; Trident maps data to the inactive pair. - Extension payloads are extra tar members, not new disk partitions. - diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index c08123e3d8..bfa137c623 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -26,7 +26,6 @@ RFC 0789 proposes COSI 1.3 support for bundled extensions: optional metadata arr `extensions.sysexts` and `extensions.confexts` reference compressed DDIs by ID, location, size and hash. OS and extensions can then ship as one artifact; Trident can also fetch extensions listed separately in Host Configuration. -See the [example COSI cutaway](./0838-cosi-anatomy.svg). #### ACL storage and update behavior @@ -123,7 +122,7 @@ the inactive slot's files without changing the running system. At boot, `trident-slot-loader` selects the files for the OS slot that actually booted. Rollback uses the same selection process. -![Managed extension delivery and activation on ACL](./0838-managed-extensions-overview.svg) +![Managed extension delivery and activation on ACL](./0838-slotted-sysexts/0838-managed-extensions-overview.svg) #### Where the files live @@ -268,7 +267,7 @@ therefore use the normal failure policy. ##### Slot Loader Manifest -The [JSON Schema](./0838-slot-loader-manifest.schema.json) defines format v1 +The [JSON Schema](./0838-slotted-sysexts/0838-slot-loader-manifest.schema.json) defines format v1 as UTF-8 JSON: ```json @@ -495,6 +494,8 @@ writable `/etc` mechanisms was successfully executed on ACL. ### Required tests +*WIP: TODO: finalize* + For the initial ACL implementation, cover: #### API and intake @@ -598,6 +599,8 @@ Customers opt in with `os.managedExtensions: managed` or its `acl` alias: ### Compatibility policy +*WIP: TODO: finalize* + Keep the loader and its boot contracts small, stable and long-lived. | Interface | Compatibility approach | @@ -612,6 +615,8 @@ loader: older strict `Os` readers may reject new fields or values. ### Preserving rollback capability +*WIP: TODO: finalize* + **TBD:** the exact compatibility matrix and admission/upgrade checks that prove rollback remains usable. Prefer an exceptionally stable loader, maintained backward compatibility and automated old/new boot tests. Block changes whose @@ -619,6 +624,8 @@ retained rollback path is incompatible. ### Policy changes +*WIP: TODO: finalize* + Mode and managed-set changes initially use A/B servicing; alias spelling alone does not trigger one. Preserve retained slots until the transition completes. Normal inactive-slot reuse replaces its old set rather than retaining history. @@ -685,6 +692,8 @@ changes or cluster-control-plane integration are required. ## Open Questions +*WIP: TODO: finalize* + 1. Shared activation-path publication and legacy alias migration. 2. `/etc` reconciliation and integrity/attestation policy. 3. Whether managed sysext-only Runtime Update belongs in the first release; diff --git a/docs/Development/RFCs/0838-managed-extensions-overview.svg b/docs/Development/RFCs/0838-slotted-sysexts/0838-managed-extensions-overview.svg similarity index 100% rename from docs/Development/RFCs/0838-managed-extensions-overview.svg rename to docs/Development/RFCs/0838-slotted-sysexts/0838-managed-extensions-overview.svg diff --git a/docs/Development/RFCs/0838-slot-loader-manifest.schema.json b/docs/Development/RFCs/0838-slotted-sysexts/0838-slot-loader-manifest.schema.json similarity index 100% rename from docs/Development/RFCs/0838-slot-loader-manifest.schema.json rename to docs/Development/RFCs/0838-slotted-sysexts/0838-slot-loader-manifest.schema.json From 7c5718cef23b6b3b3241b0e5c7f3f48d645f1116 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 14:46:11 -0700 Subject: [PATCH 11/27] RFC: clarify the opt-in requirement and include editorial refinements Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 340 +++++++++--------- 1 file changed, 170 insertions(+), 170 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index bfa137c623..18d36d1d4a 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -1,7 +1,8 @@ # 0838 Trident-Managed Extensions - Date: 2026-10-15 -- RFC PR: [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) +- RFC PR: + [microsoft/trident#838](https://github.com/microsoft/trident/pull/838) - Status: Proposal. ## Summary @@ -22,9 +23,9 @@ Discoverable Disk Images (DDIs): mountable filesystem images containing those files and compatibility metadata. A **Composable OS Image (COSI)** is a tar of OS partition images and metadata. -RFC 0789 proposes COSI 1.3 support for bundled extensions: optional metadata arrays -`extensions.sysexts` and `extensions.confexts` reference compressed DDIs by ID, -location, size and hash. OS and extensions can then ship as one artifact; +RFC 0789 proposes COSI 1.3 support for bundled extensions: optional metadata +arrays `extensions.sysexts` and `extensions.confexts` reference compressed DDIs +by ID, location, size and hash. OS and extensions can then ship as one artifact; Trident can also fetch extensions listed separately in Host Configuration. #### ACL storage and update behavior @@ -42,39 +43,33 @@ those changes. ### Problem Statement -Users build custom images on ACL, often adding stack files or preloaded -sysexts on mutable storage. ACL's A/B update replaces the core OS, not those -additions. Their stack therefore needs an out-of-band updater even though it -already ships in their image. +Users build custom images on ACL, often adding stack files or preloaded sysexts +on mutable storage. ACL's A/B update replaces the core OS, not those additions. +Their stack therefore needs an out-of-band updater even though it already ships +in their image. ### Goals - Update and roll back ACL's OS and declared sysexts together, using one COSI instead of an out-of-band stack updater. +- Support ACL's partition layout, where all writeable partitions are NOT + A/B-slotted. - Activate the matching extension set before consumers run on every boot, including rollback. - Preserve user state while confext defaults follow the OS slot. ## Scope -**ACL is the initial design and implementation target** because its sealed -A/B `/usr` and shared mutable storage require explicit slot selection. -Support for other platforms is future work. The public API avoids ACL-specific -settings so it can be reused later. - -The mode selects how Trident consumes COSI-bundled extensions. This RFC -develops `managed` (alias `acl`) for ACL; `simple` and `native` describe direct -placement on writable A/B filesystems. - -Explicit Host Configuration `os.sysexts` and `os.confexts` declarations remain -honored in every mode. They retain existing placement behavior except in -`managed`/`acl`, which owns the combined COSI and Host Configuration set. +**ACL is the initial design and implementation target** because its sealed A/B +`/usr` and shared mutable storage require managed slot selection. Support for +other platforms is future work. The public API avoids ACL-specific settings so +it can be reused later. ### Requirements -- Select bundled-extension handling through one public mode. -- A published store must contain valid manifests for every bootable slot; - images without a store need no initialization. +- Let users explicitly enable Trident to update bundled extensions alongside the OS. +- A published store must contain valid manifests for every bootable slot; images + without a store need no initialization. - Replace the inactive slot's managed extension set on reuse; do not retain older sets for that slot. - Preserve each platform's filesystem mutability and user-state persistence. @@ -83,9 +78,10 @@ honored in every mode. They retain existing placement behavior except in ### Out of Scope -- Image Customizer's managed-content authoring/bundling API and build design - (a required separate proposal; see [Dependencies](#dependencies)). -- Relocating the Trident-owned store; see [Future Possibilities](#future-possibilities). +- Image Customizer's managed-content authoring/bundling API and build design (a + required separate proposal; see [Dependencies](#dependencies)). +- Relocating the Trident-owned store; see + [Future Possibilities](#future-possibilities). - Updating arbitrary files on shared mutable storage. - Runtime Update for managed extension/policy changes in the initial release. - New image-less updates, initrd-scoped sysext content or ESP payload storage. @@ -99,8 +95,8 @@ honored in every mode. They retain existing placement behavior except in slot content before consumers run, without partial activation or late repair. - Repeated slot reuse removes obsolete extension files without changing the active slot or accumulating older sets. -- The public API, safe adoption and tests below are delivered; - user `/etc` edits and SELinux Enforcing are preserved. +- The public API, safe adoption and tests below are delivered; user `/etc` edits + and SELinux Enforcing are preserved. ## Dependencies @@ -126,39 +122,41 @@ Rollback uses the same selection process. #### Where the files live -The [store](#managed-content-store) is fixed at `/var/lib/trident/slots`. Its `a/` and `b/` directories -hold each slot's sysexts and confexts on shared ROOT. These are separate -directories, not new partitions. If A is running, an update prepares B's set -while retaining A's files. **Reusing B replaces its entire managed extension -set**, just as staging a new OS replaces B's partition contents. Keep one set -per slot, not a history of sets within each slot. +The [store](#managed-content-store) is fixed at `/var/lib/trident/slots`. Its +`a/` and `b/` directories hold each slot's sysexts and confexts on shared ROOT. +These are separate directories, not new partitions. If A is running, an update +prepares B's set while retaining A's files. **Reusing B replaces its entire +managed extension set**, just as staging a new OS replaces B's partition +contents. Keep one set per slot, not a history of sets within each slot. -The [staged update](#trident-staging) takes effect on reboot into the target OS slot. +The [staged update](#trident-staging) takes effect on reboot into the target OS +slot. #### Booting after an update -The **[Slot Loader Manifest](#slot-loader-manifest)** tells the loader exactly which managed extensions -belong to the booted slot. -It is created when the store is initialized, during +The **[Slot Loader Manifest](#slot-loader-manifest)** tells the loader exactly +which managed extensions belong to the booted slot. It is created when the store +is initialized, during [image construction](#base-image-construction-and-ic-integration) or first servicing, and replaced during update staging. No additional COSI format change is required. After reading `acl.slot`, the loader opens -`/sysroot/var/lib/trident/slots//slot-loader.json` in the initrd. -It uses this file, not Trident's SQLite database or a directory scan. +`/sysroot/var/lib/trident/slots//slot-loader.json` in the initrd. It uses +this file, not Trident's SQLite database or a directory scan. The loader [returns success without activation](#loader-startup-behavior) when -the store is absent or a valid manifest selects no images. -A loader failure on an uncommitted update boot triggers +the store is absent or a valid manifest selects no images. A loader failure on +an uncommitted update boot triggers [diagnostics and reboot](#update-boot-failure-recovery) into the committed OS. #### Writable `/etc` -User edits in the writable **upper** take precedence over [extension defaults](#confext-handling), -which take precedence over factory defaults. Changing OS slots changes the -extension defaults but keeps that shared upper, so user edits survive. -The native sysext merge still runs after `/etc` assembly. +User edits in the writable **upper** take precedence over +[extension defaults](#confext-handling), which take precedence over factory +defaults. Changing OS slots changes the extension defaults but keeps that shared +upper, so user edits survive. The native sysext merge still runs after `/etc` +assembly. ### Component deep dives @@ -171,18 +169,18 @@ implementation uses only `/var/lib/trident/slots`, with separate `a/` and `b/` directories on shared ROOT. The path must resolve to ACL's shared ROOT filesystem. ROOT must be mounted -before the loader runs and writable during staging. Target-slot selection -comes from the host's A/B configuration, not source COSI partition numbers. +before the loader runs and writable during staging. Target-slot selection comes +from the host's A/B configuration, not source COSI partition numbers. #### Base Image Construction and IC Integration -ACL images may omit the managed content store entirely. **If a store is -present, it must contain valid manifests for every bootable slot**, even if -they select no images. `{"version": 1}` is a valid empty manifest. Boot must -not depend on Trident servicing or datastore initialization. +ACL images may omit the managed content store entirely. **If a store is present, +it must contain valid manifests for every bootable slot**, even if they select +no images. `{"version": 1}` is a valid empty manifest. Boot must not depend on +Trident servicing or datastore initialization. -IC must support adding, replacing and removing managed payloads while -publishing their files and manifests together. Its API remains a +IC must support adding, replacing and removing managed payloads while publishing +their files and manifests together. Its API remains a [separate proposal](#dependencies). Integrating `trident offline-initialize` into ACL builds remains an independent @@ -199,13 +197,13 @@ loader must not invoke initialization to repair a missing manifest. This sequence applies to `managed`/`acl`. Direct-placement modes follow the [mode policies](#host-configuration) instead. -Use the normal A/B lifecycle checks before modifying the target. If an -update has booted B but B is not committed, A is still the fallback and must -not be overwritten. +Use the normal A/B lifecycle checks before modifying the target. If an update +has booted B but B is not committed, A is still the fallback and must not be +overwritten. -1. Collect the COSI and Host Configuration extensions. Validate their identities, - paths and whether the host can activate them. - Each boot path's loader must understand its slot's manifest version. +1. Collect the COSI and Host Configuration extensions. Validate their + identities, paths and whether the host can activate them. Each boot path's + loader must understand its slot's manifest version. 2. Select the inactive slot. Remove any old Slot Loader Manifest and synchronize its directory before clearing the Trident-owned extension directory, including interrupted-staging leftovers. @@ -219,9 +217,9 @@ not be overwritten. Boot reads only `slot-loader.json`; a leftover temporary file is never used. -Finalization retains Trident's existing one-shot boot handoff. Before arming -the candidate, it verifies that persistent boot defaults still select the -committed OS and that the candidate supports the recovery path below. +Finalization retains Trident's existing one-shot boot handoff. Before arming the +candidate, it verifies that persistent boot defaults still select the committed +OS and that the candidate supports the recovery path below. #### Trident Slot Loader @@ -234,10 +232,10 @@ committed OS and that the candidate supports the recovery path below. | Valid manifest selects no images | Exit successfully (`0`); no activation. | | Other lookup, parsing or selected-image errors | Return nonzero. | -Check store presence before slot detection. Successful early exits do not -create or replace runtime links or mounts. `{"version": 1}` is a valid empty -manifest; unreadable or malformed files are not. -Nonzero results follow the [update-boot failure policy](#update-boot-failure-recovery). +Check store presence before slot detection. Successful early exits do not create +or replace runtime links or mounts. `{"version": 1}` is a valid empty manifest; +unreadable or malformed files are not. Nonzero results follow the +[update-boot failure policy](#update-boot-failure-recovery). ##### Loader configuration @@ -254,21 +252,21 @@ Both paths name the same store: `/sysroot/...` is the initrd's view; These settings are private. `LOADER_MODE` controls slot detection, not service timing. -This RFC proposes a private kernel command-line input, -`trident.update_boot=1`, for an uncommitted A/B update boot. Absence means a -normal boot. It is boot configuration, not a manifest field or Host -Configuration option, so a missing or malformed manifest cannot hide it. +This RFC proposes a private kernel command-line input, `trident.update_boot=1`, +for an uncommitted A/B update boot. Absence means a normal boot. It is boot +configuration, not a manifest field or Host Configuration option, so a missing +or malformed manifest cannot hide it. -Trident stages a removable, prebuilt `trident-update.addon.efi` on the -candidate UKI's `.extra.d/` only, following the image's normal signing policy. -Factory and fallback entries do not carry it. Commit durably removes that copy -before any persistent boot-order/default promotion; subsequent boots -therefore use the normal failure policy. +Trident stages a removable, prebuilt `trident-update.addon.efi` on the candidate +UKI's `.extra.d/` only, following the image's normal signing policy. Factory and +fallback entries do not carry it. Commit durably removes that copy before any +persistent boot-order/default promotion; subsequent boots therefore use the +normal failure policy. ##### Slot Loader Manifest -The [JSON Schema](./0838-slotted-sysexts/0838-slot-loader-manifest.schema.json) defines format v1 -as UTF-8 JSON: +The [JSON Schema](./0838-slotted-sysexts/0838-slot-loader-manifest.schema.json) +defines format v1 as UTF-8 JSON: ```json { @@ -298,25 +296,26 @@ first using the loader uses that array order for `/etc` lowerdirs, where the first wins. The loader rejects unsupported versions, missing required fields, invalid known -fields, duplicate known JSON keys and duplicate names within either kind, even if -ignored fields differ. It requires readable, regular image files confined to -the selected slot's directory and applies the -compatibility checks below before activation. Symlinks, missing image files and invalid -metadata fail activation; none trigger a fallback to directory scanning. +fields, duplicate known JSON keys and duplicate names within either kind, even +if ignored fields differ. It requires readable, regular image files confined to +the selected slot's directory and applies the compatibility checks below before +activation. Symlinks, missing image files and invalid metadata fail activation; +none trigger a fallback to directory scanning. ##### Selecting extensions at boot 1. **Existing:** ACL mounts ROOT in the initrd. -2. **New:** `trident-slot-loader.service` runs after that mount. The loader reads - `acl.slot=a|b`, loads the slot's manifest, validates its images and prepares - `/run/extensions` links and confext lowerdirs. Missing, invalid or duplicate - slot values are errors. +2. **New:** `trident-slot-loader.service` runs after that mount. The loader + reads `acl.slot=a|b`, loads the slot's manifest, validates its images and + prepares `/run/extensions` links and confext lowerdirs. Missing, invalid or + duplicate slot values are errors. 3. **Integration change:** the existing `initrd-setup-root.service` gains a required dependency on the loader and consumes its lowerdir list when creating the existing writable `/etc` overlay. -4. **Existing:** `/run` survives switch-root, and `systemd-sysext.service` merges - the sysexts in the real root. Afterwards, `ensure-sysext.service` reloads unit - discovery and restarts targets; it does not perform the merge itself. +4. **Existing:** `/run` survives switch-root, and `systemd-sysext.service` + merges the sysexts in the real root. Afterwards, `ensure-sysext.service` + reloads unit discovery and restarts targets; it does not perform the merge + itself. On rollback, the booted slot selects its old set before consumers run. No later `trident commit` repair is needed. Sysext content is not available before the @@ -325,36 +324,38 @@ native merge in step 4. Loader failure must stop root setup; activation failure must prevent a healthy commit. Use required service dependencies, not `Wants=` alone or conditions that quietly skip the loader. Preserve another component's activation entries or -reject the conflict; the exact shared-directory publication protocol remains open. +reject the conflict; the exact shared-directory publication protocol remains +open. ##### Update-boot failure recovery The loader unit uses `Restart=no` and gains `OnFailure=trident-slot-loader-recovery.service`. This new initrd-only recovery unit uses `ConditionKernelCommandLine=trident.update_boot=1` and -`ConditionPathExists=/etc/initrd-release`, with `DefaultDependencies=no`. -It has no dependency on `/etc` assembly or Trident's datastore and runs with -a bounded timeout. Nonzero exits, crashes and timeouts take this path, not -just errors returned by the loader's main function. +`ConditionPathExists=/etc/initrd-release`, with `DefaultDependencies=no`. It has +no dependency on `/etc` assembly or Trident's datastore and runs with a bounded +timeout. Nonzero exits, crashes and timeouts take this path, not just errors +returned by the loader's main function. The handler captures diagnostics, then requests a reboot through systemd. -`SuccessAction=reboot` and `FailureAction=reboot` ensure that capture failure -or timeout does not leave the candidate stuck. It never re-arms the one-shot -entry or guesses the other slot: the consumed one-shot handoff returns the next boot +`SuccessAction=reboot` and `FailureAction=reboot` ensure that capture failure or +timeout does not leave the candidate stuck. It never re-arms the one-shot entry +or guesses the other slot: the consumed one-shot handoff returns the next boot to the committed default. Normal boots keep the existing fatal-error behavior without automatic reboot, preventing a fallback boot from entering a loop. Before reboot, the handler overwrites a bounded, human-readable record at `/sysroot/var/lib/trident/diagnostics/slot-loader-last-failure.log` and synchronizes it. This is shared ROOT, outside the slot directory that staging -clears; the old OS reads it at `/var/lib/trident/diagnostics/slot-loader-last-failure.log`. -Record the boot ID, attempted slot when known, failure phase, service result -and error reason, not manifest or image contents. Failure to persist the record -is reported to the journal/console but does not prevent reboot. If the loader -crashed before reporting a reason, capture its unit result and journal tail. - -Use leveled stderr diagnostics (slot, manifest path, store, activation method and -image errors), captured by the journal/console. Return nonzero on failure. +clears; the old OS reads it at +`/var/lib/trident/diagnostics/slot-loader-last-failure.log`. Record the boot ID, +attempted slot when known, failure phase, service result and error reason, not +manifest or image contents. Failure to persist the record is reported to the +journal/console but does not prevent reboot. If the loader crashed before +reporting a reason, capture its unit result and journal tail. + +Use leveled stderr diagnostics (slot, manifest path, store, activation method +and image errors), captured by the journal/console. Return nonzero on failure. Install the loader's dracut module before normal UKI build and signing. ##### Confext handling @@ -366,12 +367,13 @@ manifest order, only after all selected mounts are ready. The loader reuses confext images, not systemd 255's native `systemd-confext` merge, which would make `/etc` read-only. It does not create the final overlay; -ACL Bootengine consumes the list. With no confexts, there are no extra lowerdirs. +ACL Bootengine consumes the list. With no confexts, there are no extra +lowerdirs. #### ACL Bootengine -ACL already combines factory defaults with a persistent writable directory -using overlayfs. This RFC requires two integration changes: +ACL already combines factory defaults with a persistent writable directory using +overlayfs. This RFC requires two integration changes: - The existing `initrd-setup-root.service` gains `Requires=` and `After=` dependencies on `trident-slot-loader.service`. @@ -391,8 +393,8 @@ Bootengine does not modify a live mount or copy defaults into the writable directory. Each confext supplies a directory to the same overlay, not another overlay mount. -`overlaycleanup()` currently compares user files only with factory defaults. -How cleanup handles changed or removed extension defaults and deletion markers +`overlaycleanup()` currently compares user files only with factory defaults. How +cleanup handles changed or removed extension defaults and deletion markers (whiteouts) remains an open policy decision. ### Safety checks @@ -426,8 +428,8 @@ os: The proposed names distinguish filesystem-level slotting (`native`) from Trident-managed logical slots on shared storage (`managed`). `acl` is an alias -for `managed`, not a separate mode. Aliases are normalized before comparison -and serialized as the canonical value, so changing only the spelling does not +for `managed`, not a separate mode. Aliases are normalized before comparison and +serialized as the canonical value, so changing only the spelling does not trigger servicing. Boolean values are not mode values. | Value | COSI-bundled extensions | @@ -439,10 +441,10 @@ trigger servicing. Boolean values are not mode values. \* Future developments, not part of the initial implementation. -The mode applies to both sysexts and confexts. In `managed`/`acl`, explicit -Host Configuration URL/OCI images join the owned set and use managed -destinations; in other modes their existing explicit placement behavior is -retained. Unrelated factory/operator extensions are not adopted. +The mode applies to both sysexts and confexts. In `managed`/`acl`, explicit Host +Configuration URL/OCI images join the owned set and use managed destinations; in +other modes their existing explicit placement behavior is retained. Unrelated +factory/operator extensions are not adopted. Ignoring bundled payloads does not waive normal COSI metadata/OS-image validation. Intentionally ignored members are not reported as unused OS images. @@ -455,19 +457,18 @@ An empty `managed`/`acl` set publishes an empty target manifest. | Sysext | `/var/lib/extensions`, then `/etc/extensions` | | Confext | `/var/lib/confexts`, then `/usr/lib/confexts`, then `/usr/local/lib/confexts` | -The primary candidates retain Trident's existing default directories. -Ephemeral `/run` locations and initrd-scoped `/.extra/sysext` are not candidates. -An eligible directory exists or can be created on the target slot's writable -A/B filesystem without breaking image protection. Evaluate both kinds before +The primary candidates retain Trident's existing default directories. Ephemeral +`/run` locations and initrd-scoped `/.extra/sysext` are not candidates. An +eligible directory exists or can be created on the target slot's writable A/B +filesystem without breaking image protection. Evaluate both kinds before writing; if a consumed kind has no eligible destination, fail the update. `native` never falls back to `managed` implicitly. On ACL, shared ROOT and sealed `/usr` provide no eligible native destination. `simple` and `native` only place files in the target filesystem if it is A/B-slotted at a partition level. The image's existing boot setup activates -them: standard OS discovery for -`native`, or image-provided configuration for `simple`. Trident Slot Loader -does not activate these files. +them: standard OS discovery for `native`, or image-provided configuration for +`simple`. Trident Slot Loader does not activate these files. ### Identity and layout @@ -489,8 +490,8 @@ their deployed files without owning the surrounding native directory. ### Validation -A proof of concept combining the core managed-extension, A/B update and -writable `/etc` mechanisms was successfully executed on ACL. +A proof of concept combining the core managed-extension, A/B update and writable +`/etc` mechanisms was successfully executed on ACL. ### Required tests @@ -516,17 +517,17 @@ For the initial ACL implementation, cover: only the target changes, space stays bounded by two slot sets, and retained rollback data remains intact. Non-owned activation entries are preserved or rejected. -- Interrupt initial store creation, staging and manifest publication: - retained slots remain bootable, temporary manifests are never consumed, - and no partial set becomes active. +- Interrupt initial store creation, staging and manifest publication: retained + slots remain bootable, temporary manifests are never consumed, and no partial + set becomes active. #### Loader and ACL integration - Manifest versions/types and required fields, duplicate known keys/names (including different ignored fields), unknown optional fields, unsafe names and symlinks; omitted arrays and version-only manifests are valid. -- No store or a valid empty set exits successfully without links or mounts. - A present store with missing/malformed/unreadable metadata or selected images +- No store or a valid empty set exits successfully without links or mounts. A + present store with missing/malformed/unreadable metadata or selected images fails rather than activating a partial set. - ROOT/loader/Bootengine ordering, factory-only fallback for missing/empty lowerdir lists, and blocked root setup after loader failure. @@ -535,8 +536,8 @@ For the initial ACL implementation, cover: #### Update failure and rollback -- Loader errors, crashes and timeouts return a pending boot to the committed - OS without a loop; normal no-ops do not reboot. Include bootloader fallback, +- Loader errors, crashes and timeouts return a pending boot to the committed OS + without a loop; normal no-ops do not reboot. Include bootloader fallback, failed commit and power loss in both slot directions. - Marker cleanup precedes persistent boot promotion; cleanup failure preserves the previous default. @@ -545,8 +546,8 @@ For the initial ACL implementation, cover: #### Image construction and compatibility -- Base/customized images boot and reboot before datastore initialization, - with absent, empty or populated stores. Construction edits keep payloads and +- Base/customized images boot and reboot before datastore initialization, with + absent, empty or populated stores. Construction edits keep payloads and manifests consistent without adopting unrelated images. - Old/new loader, manifest, boot-context and Host Configuration reader combinations preserve supported rollback paths; unsafe adoption is refused. @@ -570,23 +571,22 @@ sysext-only 10 ms limit. Do not report coordination time as the whole impact. Compare matched ACL boots with equivalent hardware, kernel, services and payloads. Measure added boot-critical-path delay, not just process CPU time; -report cold/warm median and p95, preparation time and image counts. The no-op -5 ms goal is a starting budget, and none of these targets is yet demonstrated. +report cold/warm median and p95, preparation time and image counts. The no-op 5 +ms goal is a starting budget, and none of these targets is yet demonstrated. -Record staging, activation and rollback outcomes with slot, selected image names, -activation method and failure reason in servicing logs and boot diagnostics. -Correlate a recovered update failure with its persistent record's boot ID. -Success requires matching visible content and no healthy commit after activation -failure. +Record staging, activation and rollback outcomes with slot, selected image +names, activation method and failure reason in servicing logs and boot +diagnostics. Correlate a recovered update failure with its persistent record's +boot ID. Success requires matching visible content and no healthy commit after +activation failure. ## Servicing ### Introduction and opt-in -The default is no new behavior: omitted means `disabled`, existing OS -servicing and explicit URL/OCI declarations are unchanged, and bundled images -are not adopted. Images without a store boot normally without Trident -initialization. +The default is no new behavior: omitted means `disabled`, existing OS servicing +and explicit URL/OCI declarations are unchanged, and bundled images are not +adopted. Images without a store boot normally without Trident initialization. Customers opt in with `os.managedExtensions: managed` or its `acl` alias: @@ -610,8 +610,8 @@ Keep the loader and its boot contracts small, stable and long-lived. | Per-slot state | Write each slot's manifest in a format its loader supports. Preserve the retained slot's files and format; never convert it to a new-only version. | | Boot and store contracts | Keep paths, update markers and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | -Rollback needs compatible Trident state readers as well as a compatible -loader: older strict `Os` readers may reject new fields or values. +Rollback needs compatible Trident state readers as well as a compatible loader: +older strict `Os` readers may reject new fields or values. ### Preserving rollback capability @@ -637,14 +637,14 @@ directories. Migrate legacy aliases safely: deleting them can break rollback, while leaving them can shadow `/run`. Refuse unsafe transitions and leave unrelated entries untouched. -Reject unsupported Runtime Update requests, without promotion or partial application. -Explicit Host Configuration inputs keep 0789's servicing constraints; -`simple`/`native` remain deferred. +Reject unsupported Runtime Update requests, without promotion or partial +application. Explicit Host Configuration inputs keep 0789's servicing +constraints; `simple`/`native` remain deferred. ## Implementation Plan -Initial delivery is `managed`/`acl` on ACL, with `disabled` as the default -for bundled extensions. `simple`, `native`, configurable stores and native +Initial delivery is `managed`/`acl` on ACL, with `disabled` as the default for +bundled extensions. `simple`, `native`, configurable stores and native mutable-confext migration remain future work. ACL integration of `trident offline-initialize` is an independent goal, not a prerequisite for unmanaged-image boot. @@ -664,9 +664,9 @@ design/implementation effort (S/M/L), not elapsed-time commitments. Production enablement requires passing the initial test areas, demonstrated rollback compatibility and published measurements against the scenario budgets. -Unmet performance goals need explicit review; the compatibility-enforcement -TBD must be resolved before that gate. No new COSI fields, systemd source -changes or cluster-control-plane integration are required. +Unmet performance goals need explicit review; the compatibility-enforcement TBD +must be resolved before that gate. No new COSI fields, systemd source changes or +cluster-control-plane integration are required. ## Counter-Arguments @@ -686,7 +686,8 @@ changes or cluster-control-plane integration are required. - **New dedicated A/B extension partitions:** valid, but require reserving disk capacity for both slots upfront. Shared ROOT lets the two sets use available filesystem space as needed. -- **ESP payload storage:** unnecessary when the existing ROOT can hold the images. +- **ESP payload storage:** unnecessary when the existing ROOT can hold the + images. - **Systemd backports or changing ACL's merge order:** neither is required by this design. @@ -696,8 +697,8 @@ changes or cluster-control-plane integration are required. 1. Shared activation-path publication and legacy alias migration. 2. `/etc` reconciliation and integrity/attestation policy. -3. Whether managed sysext-only Runtime Update belongs in the first release; - this proposal starts with A/B servicing. +3. Whether managed sysext-only Runtime Update belongs in the first release; this + proposal starts with A/B servicing. 4. How rollback compatibility is admitted and enforced across loader, manifest, boot-context and servicing-state reader versions. @@ -705,8 +706,8 @@ changes or cluster-control-plane integration are required. ### Configurable store location -A configurable `os.managedContentStore` path could support other storage -layouts later. It would require early-mount dependencies, compatible loader +A configurable `os.managedContentStore` path could support other storage layouts +later. It would require early-mount dependencies, compatible loader configuration and rollback-safe migration of both slot sets. The initial ACL implementation supports only `/var/lib/trident/slots` on ROOT. @@ -729,19 +730,18 @@ lowerdirs live. ### Btrfs-backed managed store -On Btrfs-backed storage, the store could use a pair of subvolumes, one per -slot, to simplify inactive-slot replacement without changing paths or the -public API. +On Btrfs-backed storage, the store could use a pair of subvolumes, one per slot, +to simplify inactive-slot replacement without changing paths or the public API. ### Other platforms -Non-ACL backends can reuse the policy once validated against their storage -and activation models. +Non-ACL backends can reuse the policy once validated against their storage and +activation models. -A future `managed` backend could prepare links in writable A/B extension -search directories during staging. Switching OS slots then selects those -links without a loader. The owned images may live on shared or slotted -storage, but the links must still resolve after reboot. +A future `managed` backend could prepare links in writable A/B extension search +directories during staging. Switching OS slots then selects those links without +a loader. The owned images may live on shared or slotted storage, but the links +must still resolve after reboot. Choose separately for sysexts and confexts. Native confext integration can use slotted links or boot-selected `/run/confexts` entries if it preserves the From 51484d46da075c5b020cead40f1c6313fa83f6de Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 14:51:09 -0700 Subject: [PATCH 12/27] Updates --- docs/Development/RFCs/0838-slotted-sysexts.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 18d36d1d4a..deeb3b0479 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -67,13 +67,14 @@ it can be reused later. ### Requirements -- Let users explicitly enable Trident to update bundled extensions alongside the OS. +- Support ACL's shared storage without new partitions or systemd source changes. +- Let users explicitly enable Trident to update bundled extensions alongside the + OS. - A published store must contain valid manifests for every bootable slot; images without a store need no initialization. - Replace the inactive slot's managed extension set on reuse; do not retain older sets for that slot. - Preserve each platform's filesystem mutability and user-state persistence. -- Support ACL's shared storage without new partitions or systemd source changes. - Keep the policy reusable on platforms with writable A/B storage. ### Out of Scope From 55bc5faf430041418376f3f742e60b71298e2187 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 15:07:40 -0700 Subject: [PATCH 13/27] update notes --- docs/Development/RFCs/0838-slotted-sysexts.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index deeb3b0479..7a3d3c7754 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -496,7 +496,7 @@ A proof of concept combining the core managed-extension, A/B update and writable ### Required tests -*WIP: TODO: finalize* +*WIP: TODO: re-review* For the initial ACL implementation, cover: @@ -600,7 +600,7 @@ Customers opt in with `os.managedExtensions: managed` or its `acl` alias: ### Compatibility policy -*WIP: TODO: finalize* +*WIP: TODO: re-review* Keep the loader and its boot contracts small, stable and long-lived. @@ -616,7 +616,7 @@ older strict `Os` readers may reject new fields or values. ### Preserving rollback capability -*WIP: TODO: finalize* +*WIP: TODO: re-review* **TBD:** the exact compatibility matrix and admission/upgrade checks that prove rollback remains usable. Prefer an exceptionally stable loader, maintained @@ -625,7 +625,7 @@ retained rollback path is incompatible. ### Policy changes -*WIP: TODO: finalize* +*WIP: TODO: re-review* Mode and managed-set changes initially use A/B servicing; alias spelling alone does not trigger one. Preserve retained slots until the transition completes. @@ -694,7 +694,7 @@ cluster-control-plane integration are required. ## Open Questions -*WIP: TODO: finalize* +*WIP: TODO: re-review* 1. Shared activation-path publication and legacy alias migration. 2. `/etc` reconciliation and integrity/attestation policy. From 58a0bf0e70724ae7948c9225ec51a987aa3032cb Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 15:15:04 -0700 Subject: [PATCH 14/27] RFC: move detailed test coverage into a supporting plan Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 69 ++--------- .../RFCs/0838-slotted-sysexts/test-plan.md | 113 ++++++++++++++++++ 2 files changed, 123 insertions(+), 59 deletions(-) create mode 100644 docs/Development/RFCs/0838-slotted-sysexts/test-plan.md diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 7a3d3c7754..8f71aae916 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -498,65 +498,16 @@ A proof of concept combining the core managed-extension, A/B update and writable *WIP: TODO: re-review* -For the initial ACL implementation, cover: - -#### API and intake - -- `managed`/`acl` equivalence and canonical serialization; `disabled` ignores - bundled payloads without unused-image errors. Unsupported modes and Boolean - inputs fail, while explicit URL/OCI behavior remains unchanged by default. -- Both input sources, destination overrides, IDs distinct from filenames, - renames and cross-source collisions; unrelated extensions remain untouched. -- Compressed COSI and raw Host Configuration hashes, malformed images and - extension compatibility failures are checked before publication. - -#### Store lifetime and publication - -- Fixed ROOT placement and mount ordering; refuse writes through aliases, - traversal or unowned paths, or cleanup of a still-needed fallback slot. -- Repeated A/B reuse, additions/removals, empty sets, retries and mode changes: - only the target changes, space stays bounded by two slot sets, and retained - rollback data remains intact. Non-owned activation entries are preserved or - rejected. -- Interrupt initial store creation, staging and manifest publication: retained - slots remain bootable, temporary manifests are never consumed, and no partial - set becomes active. - -#### Loader and ACL integration - -- Manifest versions/types and required fields, duplicate known keys/names - (including different ignored fields), unknown optional fields, unsafe names - and symlinks; omitted arrays and version-only manifests are valid. -- No store or a valid empty set exits successfully without links or mounts. A - present store with missing/malformed/unreadable metadata or selected images - fails rather than activating a partial set. -- ROOT/loader/Bootengine ordering, factory-only fallback for missing/empty - lowerdir lists, and blocked root setup after loader failure. -- Multiple `/etc` layers, precedence, copy-up, whiteouts and removal; user edits - survive update and rollback. Assert visible content, not just file existence. - -#### Update failure and rollback - -- Loader errors, crashes and timeouts return a pending boot to the committed OS - without a loop; normal no-ops do not reboot. Include bootloader fallback, - failed commit and power loss in both slot directions. -- Marker cleanup precedes persistent boot promotion; cleanup failure preserves - the previous default. -- Failure records survive outside the reused slot. Read-only ROOT, capture - errors and timeouts do not prevent the rollback reboot. - -#### Image construction and compatibility - -- Base/customized images boot and reboot before datastore initialization, with - absent, empty or populated stores. Construction edits keep payloads and - manifests consistent without adopting unrelated images. -- Old/new loader, manifest, boot-context and Host Configuration reader - combinations preserve supported rollback paths; unsafe adoption is refused. - -#### Performance - -- Matched cold/warm benchmarks for the scenarios and budgets below, including - image-count scaling and end-to-end critical-path measurements. +Tests will be added for these areas: + +- Configuration and image sources. +- Store replacement and interrupted updates. +- Manifest parsing and boot integration. +- Update failure and rollback. +- Image construction and version compatibility. +- Boot-time impact. + +Details are in the supporting [test plan](./0838-slotted-sysexts/test-plan.md). ### Metrics diff --git a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md new file mode 100644 index 0000000000..2152a08bce --- /dev/null +++ b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md @@ -0,0 +1,113 @@ +# RFC 0838: Managed Extensions Test Plan + +*WIP: TODO: re-review* + +Supporting plan for [RFC 0838](../0838-slotted-sysexts.md#testing-and-metrics). +These are planned checks for ACL's `managed`/`acl` implementation and the +`disabled` default, not completed results. `simple` and `native` are deferred. + +## Test setup + +Use configuration tests for parsing, staging tests for file placement and +interrupted writes, and an ACL VM for actual boot, commit and rollback. +Run boot cases in both directions, A to B and B to A, with SELinux Enforcing. +Cover both bundled COSI images and Host Configuration (HC) URL/OCI inputs. + +Give each slot different `/usr` and `/etc` marker contents and create a user +override in writable `/etc`. Before staging, record the running slot's files +and hashes. Check what applications see after boot, not just what exists in +the store. + +## 1. Configuration and image sources + +| Exercise | Expected result | +| --- | --- | +| Omit the mode or use `disabled`, with bundled images and explicit HC URL/OCI declarations. | Bundled payloads are ignored without unused-image errors; OS servicing and explicit declarations behave as before. | +| Compare `managed` with `acl`; try Boolean, unknown and deferred mode values. | The alias produces the same placement and stored policy. Unsupported inputs are rejected, not treated as another mode. | +| Mix COSI images with HC-declared OCI images, including requested destinations. | Trident fetches and verifies both sources during staging, places them in the target managed store and lists them in its manifest. Boot needs no registry access. | +| Use an ID different from its filename, rename an image, or supply conflicting IDs/names across sources. | Identity and filename are handled separately; conflicts fail before publication rather than silently selecting one source. | +| Corrupt compressed COSI members or HC raw-image hashes; supply malformed or incompatible images. | Each source uses its own hash contract. Invalid inputs never become a published target set. | + +## 2. Store replacement and interrupted updates + +Start with A committed and populated sets for both slots. Stage B with added, +removed and renamed images; also stage an empty set. A's files must be unchanged +and B must contain only its new set. After a successful update and commit, +repeat with the slots reversed. Fixed-size inputs must not grow storage beyond +the two slot sets. + +Attempt cleanup through aliased or escaping paths, unowned directories, and +while A is still needed as the fallback for uncommitted B. The operation must +be refused without changing the protected files. Unrelated runtime activation +entries must be preserved or reported as conflicts. + +Interrupt first-use store creation, target clearing, image writes and manifest +publication, including immediately before and after the final rename. Retry +each case. Retained slots must stay bootable; temporary manifests must never +be read, and incomplete sets must never activate. + +Switch a previously managed target to `disabled`. Its reused slot gets an +empty manifest and no stale owned images; the retained slot keeps its data. +Native discovery directories and unrelated images are not wiped. + +## 3. Manifest parsing and boot integration + +Use the [manifest schema](./0838-slot-loader-manifest.schema.json) for structural +fixtures, plus loader tests for duplicate JSON keys, name uniqueness and file +access rules that JSON Schema cannot fully enforce. + +| Exercise | Expected result | +| --- | --- | +| Supply only `version`, omit either array, or add unknown root/item fields. | Omitted arrays mean empty. Unknown fields are ignored; known fields are still validated. | +| Remove required fields, use invalid types/versions, duplicate known keys or names, or use unsafe names/symlinks. | The loader rejects the input. Duplicate names still fail when their ignored fields differ. | +| Boot with no store or a valid empty manifest. | Successful exit without links, mounts or an automatic reboot. | +| Keep the store but remove, corrupt or make its manifest unreadable; remove a listed image or invalidate its metadata. | Activation fails, not a fallback to scanning files or an empty set. | +| Trace ROOT mounting, loader startup, Bootengine and the native sysext merge. | The loader runs after ROOT and before `/etc` assembly. Its links survive switch-root; native merge exposes the selected slot's `/usr` content. Loader failure blocks root setup and healthy commit. | + +Confirm that the fixed store resolves to shared ROOT; failed reads are errors. +For `/etc`, boot with multiple confexts and overlapping defaults. User changes +win over extension defaults, which win over factory defaults. Updates and +rollback preserve the user upper and select the right shipped defaults. + +Missing or empty lowerdir output must preserve factory-only assembly. Exercise +copy-up, whiteouts and changed/removed defaults against the agreed cleanup +policy; finalize those expectations when the RFC's reconciliation TBD is +resolved. + +## 4. Update failure and rollback + +On a marked, uncommitted update boot, inject a loader error, crash and timeout. +Each must reboot into the committed OS without re-arming the failed entry. +A failure on the fallback's normal boot must not create a reboot loop. + +Also exercise bootloader fallback, failed commit and power loss. Confirm the +visible OS and extension markers match after recovery. Fail marker removal +during commit: the previous persistent boot default must remain unchanged. + +Read the failure record from the recovered OS and correlate its boot ID and +reason. Repeat with read-only ROOT, failed diagnostic writes and capture +timeouts: lost diagnostics must be reported but must not block reboot. + +## 5. Image construction and version compatibility + +Boot and reboot base/customized images before Trident datastore initialization, +with absent, empty and populated stores. Image-building edits must keep payloads +and manifests consistent without adopting unrelated factory images. + +Test old/new loaders with their supported manifest versions and boot inputs, +then run the retained OS's Trident reader against the servicing state. Supported +combinations must boot and remain serviceable; incompatible adoption must be +rejected before activation. The exact release matrix depends on the RFC's +[rollback-compatibility decision](../0838-slotted-sysexts.md#preserving-rollback-capability). + +## 6. Boot-time impact + +Follow the RFC's [scenario budgets](../0838-slotted-sysexts.md#metrics), rather +than maintaining a second set of thresholds here. Compare matched boots for +absent-store and empty-manifest no-ops, sysext-only, and confext/mixed workloads. + +Report repetition count, image counts/sizes, cold/warm median and p95. +Include service startup and I/O, not just CPU time. For confexts, report +mounting/validation separately from coordination and include the full time +until `/etc` is ready. Existing native sysext merge time stays distinguishable +from newly added loader work. From 551f013baadb1d32dc283cff29afb7fcd0395bb0 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 15:27:10 -0700 Subject: [PATCH 15/27] RFC: validate the staged loader and manifest as a pair Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 50 ++++++++++++------- .../RFCs/0838-slotted-sysexts/test-plan.md | 17 +++++-- 2 files changed, 43 insertions(+), 24 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 8f71aae916..9ff90d98f2 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -203,8 +203,8 @@ has booted B but B is not committed, A is still the fallback and must not be overwritten. 1. Collect the COSI and Host Configuration extensions. Validate their - identities, paths and whether the host can activate them. Each boot path's - loader must understand its slot's manifest version. + identities, paths and whether the host can activate them. Check that the + loader in the target boot image supports the manifest format being written. 2. Select the inactive slot. Remove any old Slot Loader Manifest and synchronize its directory before clearing the Trident-owned extension directory, including interrupted-staging leftovers. @@ -287,6 +287,9 @@ Unknown fields are ignored in both the manifest and image objects. Compatible optional fields can be added within v1; changes that old loaders must understand require a new format version. +The manifest is written once for each staged set. Boot, commit and rollback +do not rewrite it; it changes only when that slot is reused. + An entry named `runtime` resolves to `/sysexts/runtime.raw` or `/confexts/runtime.raw`, according to its array, relative to the store. There are no caller-supplied image paths. @@ -496,8 +499,6 @@ A proof of concept combining the core managed-extension, A/B update and writable ### Required tests -*WIP: TODO: re-review* - Tests will be added for these areas: - Configuration and image sources. @@ -555,24 +556,34 @@ Customers opt in with `os.managedExtensions: managed` or its `acl` alias: Keep the loader and its boot contracts small, stable and long-lived. +The target image is normally expected to contain a loader from the same or a +newer release than the staging Trident. The actual check is format support: +Trident must write a manifest that the target loader can read, regardless of +the loader running on the servicing OS. + | Interface | Compatibility approach | | --- | --- | | Host Configuration | The enum allows future modes without changing the field shape. Initially support `disabled` and `managed`/`acl`; reject unsupported values. Canonicalize aliases and omit defaults from serialized state. | -| Loader and manifest | New loaders retain older released format readers. Old loaders ignore optional unknown fields within supported versions; incompatible semantics need a new version that they reject. | -| Per-slot state | Write each slot's manifest in a format its loader supports. Preserve the retained slot's files and format; never convert it to a new-only version. | +| Loader and manifest | Check the target pair during staging. Optional unknown fields are ignored within supported formats; unsupported versions are rejected. A release number alone does not prove support. | +| Per-slot state | Publish the manifest once and retain it with its loader through boot, commit and rollback. Do not regenerate or upgrade the retained slot's manifest. | | Boot and store contracts | Keep paths, update markers and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | -Rollback needs compatible Trident state readers as well as a compatible loader: -older strict `Os` readers may reject new fields or values. +Trident's shared servicing state is a separate compatibility concern: older +strict `Os` readers may reject new fields or values after rollback. ### Preserving rollback capability *WIP: TODO: re-review* -**TBD:** the exact compatibility matrix and admission/upgrade checks that prove -rollback remains usable. Prefer an exceptionally stable loader, maintained -backward compatibility and automated old/new boot tests. Block changes whose -retained rollback path is incompatible. +Rollback returns to the retained loader and its unchanged manifest. If that +pair booted successfully before the update, manifest-version changes in the +new slot do not affect it. There is no need for the old loader to read the +new slot's manifest, or for the new loader to read the old slot's. + +**TBD:** how staging discovers target-loader format support, and how preflight +checks shared boot configuration, store layout and servicing-state changes. +Keep those shared contracts stable; preserving a slot's pair does not protect +it from incompatible changes to shared state. ### Policy changes @@ -607,18 +618,18 @@ design/implementation effort (S/M/L), not elapsed-time commitments. | Phase | Work and completion gate | Size | Main risk | | --- | --- | :-: | --- | -| 1. Contracts and compatibility | Freeze the initial API, manifest and boot interfaces; define the rollback compatibility matrix and admission rules. | M/M | High: retained readers | +| 1. Contracts and compatibility | Freeze the initial API, manifest and boot interfaces; define target-loader format detection and shared-state rollback checks. | M/M | High: staging and shared contracts | | 2. Managed staging and lifecycle | Validate inputs; implement fixed-ROOT placement, atomic first-use publication, inactive-slot replacement and safe disable transitions. | M/L | High: ownership and interrupted writes | | 3. Loader and ACL integration | Implement selection, links and confext preparation; integrate Bootengine ordering/lowerdirs and package the dracut assets before UKI signing. | M/M | High: early-boot integration | | 4. Update-boot recovery | Add the candidate marker, one-shot fallback, persistent diagnostics and marker removal before commit promotion; prove fallback cannot loop. | M/M | High: commit and failure ordering | -| 5. Rollout and fault validation | Apply target/retained-reader preflight and execute the grouped functional, interruption, compatibility and rollback tests. | L/L | High: rollback viability | +| 5. Rollout and fault validation | Verify the staged loader/manifest pair, preserve the retained pair and check shared-state readers; execute the grouped fault and rollback tests. | L/L | High: rollback viability | | 6. Performance validation | Measure no-op and sysext-only budgets plus complete confext cost; publish cold/warm results and settle the confext total budget. | S/M | Medium: hardware and image-count variance | Production enablement requires passing the initial test areas, demonstrated rollback compatibility and published measurements against the scenario budgets. -Unmet performance goals need explicit review; the compatibility-enforcement TBD -must be resolved before that gate. No new COSI fields, systemd source changes or -cluster-control-plane integration are required. +Unmet performance goals need explicit review; target-loader detection and +shared-state compatibility checks must be defined before that gate. No new COSI +fields, systemd source changes or cluster-control-plane integration are required. ## Counter-Arguments @@ -651,8 +662,9 @@ cluster-control-plane integration are required. 2. `/etc` reconciliation and integrity/attestation policy. 3. Whether managed sysext-only Runtime Update belongs in the first release; this proposal starts with A/B servicing. -4. How rollback compatibility is admitted and enforced across loader, manifest, - boot-context and servicing-state reader versions. +4. How staging discovers which manifest formats the target loader supports. +5. How changes to shared boot configuration, store layout and servicing state + are checked for compatibility with the retained OS. ## Future Possibilities diff --git a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md index 2152a08bce..3c8ea05294 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md +++ b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md @@ -94,11 +94,18 @@ Boot and reboot base/customized images before Trident datastore initialization, with absent, empty and populated stores. Image-building edits must keep payloads and manifests consistent without adopting unrelated factory images. -Test old/new loaders with their supported manifest versions and boot inputs, -then run the retained OS's Trident reader against the servicing state. Supported -combinations must boot and remain serviceable; incompatible adoption must be -rejected before activation. The exact release matrix depends on the RFC's -[rollback-compatibility decision](../0838-slotted-sysexts.md#preserving-rollback-capability). +Stage manifests for same-release and newer target loaders, plus a target that +does not support the chosen format. Preflight must check the target loader, +not the one running on the servicing OS, and reject an unsupported pair before +activation. + +Record the retained loader and manifest bytes, update the other slot, then +roll back. Both retained files must be unchanged and boot together; no manifest +conversion or cross-slot reader compatibility is required. + +Separately test shared boot/store changes and the retained OS's Trident reader +against the servicing state. These still need +[rollback-safety checks](../0838-slotted-sysexts.md#preserving-rollback-capability). ## 6. Boot-time impact From 078b53d18bd76fe1e8b06427b441c12eefdb7f75 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 16:28:37 -0700 Subject: [PATCH 16/27] RFC: illustrate the final usr and etc filesystem layers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 2 + .../0838-filesystem-layer-stack.svg | 63 +++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 docs/Development/RFCs/0838-slotted-sysexts/0838-filesystem-layer-stack.svg diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 9ff90d98f2..b783b6ea28 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -159,6 +159,8 @@ defaults. Changing OS slots changes the extension defaults but keeps that shared upper, so user edits survive. The native sysext merge still runs after `/etc` assembly. +![Final /usr and /etc layers, backed by the booted slot's verity-protected /usr partition](./0838-slotted-sysexts/0838-filesystem-layer-stack.svg) + ### Component deep dives #### Managed content store diff --git a/docs/Development/RFCs/0838-slotted-sysexts/0838-filesystem-layer-stack.svg b/docs/Development/RFCs/0838-slotted-sysexts/0838-filesystem-layer-stack.svg new file mode 100644 index 0000000000..8b208ed740 --- /dev/null +++ b/docs/Development/RFCs/0838-slotted-sysexts/0838-filesystem-layer-stack.svg @@ -0,0 +1,63 @@ + + + Final read-only /usr and writable /etc layer stacks + Both stacks use the booted slot's dm-verity-protected /usr + partition as their base. Native systemd-sysext overlays selected read-only + sysext /usr trees on the base /usr filesystem after switch-root. Earlier, + ACL Bootengine assembles writable /etc in the initrd: shared ROOT /etc is the + writable upper, selected read-only confext /etc trees are the next lower + layers, and /usr/share/distro/etc from the protected partition supplies the + factory-default lower. Higher layers take precedence. Extension image files + and user changes live on shared ROOT, not in the protected /usr partition. + + + + + + + + + + + Final /usr: read-only merge + systemd-sysext, after switch-root + No writable upper + + /usr from selected sysext images + Read-only mounts; image files on shared ROOT + + /usr base filesystem + Read-only lower layer + + Final /etc: writable overlay + ACL Bootengine, in the initrd (before /usr merge) + + User /etc changes + Writable upper on shared ROOT; survives rollback + + Selected confext /etc defaults + Read-only lowers; image files on shared ROOT + + /usr/share/distro/etc + Read-only factory-default lower + + + + + Booted slot: USR-A or USR-B + dm-verity-protected /usr partition (read-only) + From c3cbfb884d97ea25e945a4835432a58cd4fceb61 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 16:34:52 -0700 Subject: [PATCH 17/27] RFC: simplify update-boot recovery without command-line changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 75 ++++++++++--------- .../RFCs/0838-slotted-sysexts/test-plan.md | 8 +- 2 files changed, 43 insertions(+), 40 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index b783b6ea28..951b7a2bd0 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -255,16 +255,16 @@ Both paths name the same store: `/sysroot/...` is the initrd's view; These settings are private. `LOADER_MODE` controls slot detection, not service timing. -This RFC proposes a private kernel command-line input, `trident.update_boot=1`, -for an uncommitted A/B update boot. Absence means a normal boot. It is boot -configuration, not a manifest field or Host Configuration option, so a missing -or malformed manifest cannot hide it. +The loader also needs to know whether this boot is an uncommitted update +attempt. The proposed approach is for finalization to expose Trident's pending +update context in a small initrd-readable record on shared ROOT, without +opening its SQLite datastore in the initrd. -Trident stages a removable, prebuilt `trident-update.addon.efi` on the candidate -UKI's `.extra.d/` only, following the image's normal signing policy. Factory and -fallback entries do not carry it. Commit durably removes that copy before any -persistent boot-order/default promotion; subsequent boots therefore use the -normal failure policy. +The record must identify the candidate boot, not merely say that an update +is pending: the fallback OS sees the same shared storage. It is separate from +the immutable manifest and must remain usable when manifest or slot validation +fails. No new kernel command-line option or UKI addon is required; the existing +`acl.slot` still selects the slot. ##### Slot Loader Manifest @@ -335,30 +335,29 @@ open. ##### Update-boot failure recovery -The loader unit uses `Restart=no` and gains -`OnFailure=trident-slot-loader-recovery.service`. This new initrd-only recovery -unit uses `ConditionKernelCommandLine=trident.update_boot=1` and -`ConditionPathExists=/etc/initrd-release`, with `DefaultDependencies=no`. It has -no dependency on `/etc` assembly or Trident's datastore and runs with a bounded -timeout. Nonzero exits, crashes and timeouts take this path, not just errors -returned by the loader's main function. - -The handler captures diagnostics, then requests a reboot through systemd. -`SuccessAction=reboot` and `FailureAction=reboot` ensure that capture failure or -timeout does not leave the candidate stuck. It never re-arms the one-shot entry -or guesses the other slot: the consumed one-shot handoff returns the next boot -to the committed default. Normal boots keep the existing fatal-error behavior -without automatic reboot, preventing a fallback boot from entering a loop. - -Before reboot, the handler overwrites a bounded, human-readable record at -`/sysroot/var/lib/trident/diagnostics/slot-loader-last-failure.log` and -synchronizes it. This is shared ROOT, outside the slot directory that staging -clears; the old OS reads it at -`/var/lib/trident/diagnostics/slot-loader-last-failure.log`. Record the boot ID, -attempted slot when known, failure phase, service result and error reason, not -manifest or image contents. Failure to persist the record is reported to the -journal/console but does not prevent reboot. If the loader crashed before -reporting a reason, capture its unit result and journal tail. +An initrd-only failure handler checks the pending-update context when the +loader fails, crashes or times out. On an uncommitted candidate boot, it records +the failure and requests a reboot. Trident's existing one-shot boot attempt has +already been consumed, so the next boot returns to the committed default. The +handler never re-arms the failed entry or restarts the loader. + +Normal-boot errors remain failures without automatic reboot, so a fallback +failure cannot start a reboot loop. The handler runs independently of `/etc` +assembly and the loader process; diagnostic capture has a bounded timeout and +must not prevent the candidate's reboot if writing fails. + +The handler writes and synchronizes a bounded, human-readable failure record at +`/var/lib/trident/diagnostics/slot-loader-last-failure.log` on shared ROOT, +written through `/sysroot` in the initrd. It survives slot cleanup and is +readable from the old OS. It contains the boot ID, attempted slot when known, +failure phase, unit result and reason, with a short journal tail after a crash. +Capture failures are reported to the journal/console. + +**TBD:** the pending-context file format, how the handler identifies the +candidate without relying on successful loader parsing, and its lifecycle +across finalization, cancellation, commit and power loss. The handoff must +remain consistent with Trident's servicing state; after commit or fallback, +stale context must not cause an automatic reboot. Use leveled stderr diagnostics (slot, manifest path, store, activation method and image errors), captured by the journal/console. Return nonzero on failure. @@ -411,8 +410,8 @@ cleanup handles changed or removed extension defaults and deletion markers | Native placement | Bundled `simple` and `native` images go only to writable A/B filesystems. Resolve the actual backing filesystem, including an overlay's writable upper. | Reject an unsafe explicit destination or an exhausted native candidate list before staging. | | Ownership and cleanup | Clear only the inactive managed slot tree. Direct placement changes only Trident-tracked files, not entire native directories. Reject aliases, overlapping paths, traversal and untrusted writers. | Preserve the running/fallback slot and unrelated data; refuse unsafe cleanup. | | Image validity | Validate delivery hashes and standard [`extension-release` compatibility](https://www.freedesktop.org/software/systemd/man/latest/os-release.html): distro, level/version, architecture and scope. `ID=_any` is allowed, not required. | Reject invalid selected images. Custom mounting does not bypass these checks. | -| Publication and activation | Keep temporary images undiscoverable and publish a complete managed manifest last. A valid empty manifest or absent store is a no-op; a missing manifest in an existing store is not. | Block root setup and healthy commit after activation failure; apply the update-boot recovery policy when marked. | -| Update recovery | Verify the committed default, one-shot support, recovery unit and marker addon before activation. Remove the marker before persistent boot promotion. | Refuse an unsafe handoff. Pending-boot failures record diagnostics and reboot once; normal-boot failures do not loop. | +| Publication and activation | Keep temporary images undiscoverable and publish a complete managed manifest last. A valid empty manifest or absent store is a no-op; a missing manifest in an existing store is not. | Block root setup and healthy commit after activation failure; apply recovery only for an identified uncommitted candidate boot. | +| Update recovery | Verify the committed default, one-shot support, failure handler and candidate-specific pending context before activation. Commit must complete the pending-state transition before persistent boot promotion. | Refuse an unsafe handoff. Candidate failures record diagnostics and reboot; stale context or a fallback boot must not trigger another reboot. | | Writable-store trust | Delivery checks do not add boot-time hashes or authenticate the manifest. Signing, integrity and measurement remain platform policies; sealed `/usr` attestation does not cover the store. | Do not claim inherited trust. Loss of shared storage can affect both slots; A/B cannot restore it, and boot must not require a network download. | | SELinux | Preserve Enforcing mode and correct payload/mount labels. | Surface access or labeling errors; never disable checks or exempt COSI inputs. | @@ -568,7 +567,7 @@ the loader running on the servicing OS. | Host Configuration | The enum allows future modes without changing the field shape. Initially support `disabled` and `managed`/`acl`; reject unsupported values. Canonicalize aliases and omit defaults from serialized state. | | Loader and manifest | Check the target pair during staging. Optional unknown fields are ignored within supported formats; unsupported versions are rejected. A release number alone does not prove support. | | Per-slot state | Publish the manifest once and retain it with its loader through boot, commit and rollback. Do not regenerate or upgrade the retained slot's manifest. | -| Boot and store contracts | Keep paths, update markers and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | +| Boot and store contracts | Keep paths, pending-update handoff and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | Trident's shared servicing state is a separate compatibility concern: older strict `Os` readers may reject new fields or values after rollback. @@ -623,7 +622,7 @@ design/implementation effort (S/M/L), not elapsed-time commitments. | 1. Contracts and compatibility | Freeze the initial API, manifest and boot interfaces; define target-loader format detection and shared-state rollback checks. | M/M | High: staging and shared contracts | | 2. Managed staging and lifecycle | Validate inputs; implement fixed-ROOT placement, atomic first-use publication, inactive-slot replacement and safe disable transitions. | M/L | High: ownership and interrupted writes | | 3. Loader and ACL integration | Implement selection, links and confext preparation; integrate Bootengine ordering/lowerdirs and package the dracut assets before UKI signing. | M/M | High: early-boot integration | -| 4. Update-boot recovery | Add the candidate marker, one-shot fallback, persistent diagnostics and marker removal before commit promotion; prove fallback cannot loop. | M/M | High: commit and failure ordering | +| 4. Update-boot recovery | Define the pending-context handoff; integrate one-shot fallback, persistent diagnostics and commit/cancellation cleanup; prove fallback cannot loop. | M/M | High: state and failure ordering | | 5. Rollout and fault validation | Verify the staged loader/manifest pair, preserve the retained pair and check shared-state readers; execute the grouped fault and rollback tests. | L/L | High: rollback viability | | 6. Performance validation | Measure no-op and sysext-only budgets plus complete confext cost; publish cold/warm results and settle the confext total budget. | S/M | Medium: hardware and image-count variance | @@ -667,6 +666,8 @@ fields, systemd source changes or cluster-control-plane integration are required 4. How staging discovers which manifest formats the target loader supports. 5. How changes to shared boot configuration, store layout and servicing state are checked for compatibility with the retained OS. +6. How pending-update context reaches the initrd and stays consistent through + commit, cancellation and interrupted updates. ## Future Possibilities diff --git a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md index 3c8ea05294..39af32a764 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md +++ b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md @@ -76,13 +76,15 @@ resolved. ## 4. Update failure and rollback -On a marked, uncommitted update boot, inject a loader error, crash and timeout. +On an uncommitted candidate boot, inject a loader error, crash and timeout. Each must reboot into the committed OS without re-arming the failed entry. A failure on the fallback's normal boot must not create a reboot loop. Also exercise bootloader fallback, failed commit and power loss. Confirm the -visible OS and extension markers match after recovery. Fail marker removal -during commit: the previous persistent boot default must remain unchanged. +visible OS and extension markers match after recovery. Fail the pending-state +transition during commit: the previous persistent boot default must remain +unchanged. Stale context after commit, cancellation or fallback must not cause +another automatic reboot. Finalize these cases with the pending-context contract. Read the failure record from the recovered OS and correlate its boot ID and reason. Repeat with read-only ROOT, failed diagnostic writes and capture From 2e0c10a108ca3d881258ad3e134781cd9cf88fb2 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 20 Oct 2026 17:26:35 -0700 Subject: [PATCH 18/27] docs: simplify slotted extension boot failure recovery Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0838-slotted-sysexts.md | 98 +++++++++---------- .../RFCs/0838-slotted-sysexts/test-plan.md | 25 ++--- 2 files changed, 61 insertions(+), 62 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 951b7a2bd0..2982646cdf 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -148,8 +148,8 @@ this file, not Trident's SQLite database or a directory scan. The loader [returns success without activation](#loader-startup-behavior) when the store is absent or a valid manifest selects no images. A loader failure on -an uncommitted update boot triggers -[diagnostics and reboot](#update-boot-failure-recovery) into the committed OS. +a newly deployed slot follows the +[update-boot failure policy](#update-boot-failure-recovery). #### Writable `/etc` @@ -207,22 +207,23 @@ overwritten. 1. Collect the COSI and Host Configuration extensions. Validate their identities, paths and whether the host can activate them. Check that the loader in the target boot image supports the manifest format being written. -2. Select the inactive slot. Remove any old Slot Loader Manifest and synchronize - its directory before clearing the Trident-owned extension directory, - including interrupted-staging leftovers. +2. Select the inactive slot. Remove any old Slot Loader Manifest and + `slot-loader.pending` marker, then synchronize its directory before clearing the + Trident-owned extension directory, including interrupted-staging leftovers. 3. Reserve space to stage the new extension files. Write temporary `.raw` files inside the cleared directory, outside systemd's search paths. Verify hashes, filesystem structure and extension compatibility. Reject invalid images with an invalid-input error; flush valid files and directory entries to disk. 4. Build the manifest from the incoming extensions, not the old list. Once every image is ready, write and synchronize `slot-loader.json.tmp`, atomically - rename it to `slot-loader.json`, then synchronize the slot directory. + rename it to `slot-loader.json`, then synchronize the slot directory. Write + and synchronize a `slot-loader.pending` marker beside the published manifest. -Boot reads only `slot-loader.json`; a leftover temporary file is never used. +Boot reads the manifest and marker; a leftover temporary file is never used. Finalization retains Trident's existing one-shot boot handoff. Before arming the candidate, it verifies that persistent boot defaults still select the committed -OS and that the candidate supports the recovery path below. +OS, the marker is durable and the candidate supports the recovery path below. #### Trident Slot Loader @@ -242,29 +243,24 @@ unreadable or malformed files are not. Nonzero results follow the ##### Loader configuration -ACL packaging supplies fixed loader settings: +ACL packaging supplies an initrd-readable loader env file: ```ini LOADER_MODE=acl TRIDENT_SYSEXT_STORE=/sysroot/var/lib/trident/slots TRIDENT_SYSEXT_LINK_STORE=/var/lib/trident/slots +ROLLBACK_ON_LOADER_FAILURE=true ``` Both paths name the same store: `/sysroot/...` is the initrd's view; `/var/lib/...` is the symlink destination after switching to the real root. -These settings are private. `LOADER_MODE` controls slot detection, not service -timing. - -The loader also needs to know whether this boot is an uncommitted update -attempt. The proposed approach is for finalization to expose Trident's pending -update context in a small initrd-readable record on shared ROOT, without -opening its SQLite datastore in the initrd. - -The record must identify the candidate boot, not merely say that an update -is pending: the fallback OS sees the same shared storage. It is separate from -the immutable manifest and must remain usable when manifest or slot validation -fails. No new kernel command-line option or UKI addon is required; the existing -`acl.slot` still selects the slot. +The paths and `LOADER_MODE` are fixed; `LOADER_MODE` controls slot detection, +not service timing. `ROLLBACK_ON_LOADER_FAILURE` defaults to `true` and can be +set to `false` in this env file to leave a failed boot stopped for inspection. +The failure handler reads this setting independently of the loader. The +`slot-loader.pending` marker beside each slot's manifest identifies a new slot +boot without opening Trident's SQLite datastore in the initrd. No new kernel +command-line option or UKI addon is required; `acl.slot` selects the slot. ##### Slot Loader Manifest @@ -335,29 +331,29 @@ open. ##### Update-boot failure recovery -An initrd-only failure handler checks the pending-update context when the -loader fails, crashes or times out. On an uncommitted candidate boot, it records -the failure and requests a reboot. Trident's existing one-shot boot attempt has -already been consumed, so the next boot returns to the committed default. The -handler never re-arms the failed entry or restarts the loader. - -Normal-boot errors remain failures without automatic reboot, so a fallback -failure cannot start a reboot loop. The handler runs independently of `/etc` -assembly and the loader process; diagnostic capture has a bounded timeout and -must not prevent the candidate's reboot if writing fails. - -The handler writes and synchronizes a bounded, human-readable failure record at -`/var/lib/trident/diagnostics/slot-loader-last-failure.log` on shared ROOT, -written through `/sysroot` in the initrd. It survives slot cleanup and is -readable from the old OS. It contains the boot ID, attempted slot when known, -failure phase, unit result and reason, with a short journal tail after a crash. -Capture failures are reported to the journal/console. - -**TBD:** the pending-context file format, how the handler identifies the -candidate without relying on successful loader parsing, and its lifecycle -across finalization, cancellation, commit and power loss. The handoff must -remain consistent with Trident's servicing state; after commit or fallback, -stale context must not cause an automatic reboot. +Staging writes `/slot-loader.pending` beside `slot-loader.json` before arming +the one-shot boot attempt. The loader removes the marker only after its own +successful execution, including a valid empty manifest, and synchronizes the +slot directory. This does not wait for the rest of the boot or for commit. +Loader failure leaves the marker in place. + +On loader failure, crash or timeout, an initrd failure handler reads `acl.slot` +and checks that slot's marker without relying on manifest parsing or the loader +process. It logs the slot, boot ID, unit result and reason to the journal/console. +There is no separate failure record; initrd journal messages may not survive a +reboot unless journal persistence is configured. + +If the marker exists and `ROLLBACK_ON_LOADER_FAILURE=true` (the default), the +handler requests a reboot. The one-shot attempt has already been consumed, so +the next boot selects the committed default. When the setting is `false`, the +failed boot stays stopped for inspection. A missing marker or invalid slot never +triggers an automatic reboot, preventing fallback failures from looping. The +handler does not re-arm the failed entry or restart the loader and must run +independently of `/etc` assembly. + +Cancellation or restaging clears stale markers. A failed candidate's marker +may remain on shared ROOT after fallback, but only the booted slot's marker +controls recovery. Use leveled stderr diagnostics (slot, manifest path, store, activation method and image errors), captured by the journal/console. Return nonzero on failure. @@ -411,7 +407,7 @@ cleanup handles changed or removed extension defaults and deletion markers | Ownership and cleanup | Clear only the inactive managed slot tree. Direct placement changes only Trident-tracked files, not entire native directories. Reject aliases, overlapping paths, traversal and untrusted writers. | Preserve the running/fallback slot and unrelated data; refuse unsafe cleanup. | | Image validity | Validate delivery hashes and standard [`extension-release` compatibility](https://www.freedesktop.org/software/systemd/man/latest/os-release.html): distro, level/version, architecture and scope. `ID=_any` is allowed, not required. | Reject invalid selected images. Custom mounting does not bypass these checks. | | Publication and activation | Keep temporary images undiscoverable and publish a complete managed manifest last. A valid empty manifest or absent store is a no-op; a missing manifest in an existing store is not. | Block root setup and healthy commit after activation failure; apply recovery only for an identified uncommitted candidate boot. | -| Update recovery | Verify the committed default, one-shot support, failure handler and candidate-specific pending context before activation. Commit must complete the pending-state transition before persistent boot promotion. | Refuse an unsafe handoff. Candidate failures record diagnostics and reboot; stale context or a fallback boot must not trigger another reboot. | +| Update recovery | Verify the committed default, one-shot support, failure handler and durable `slot-loader.pending` marker before activation. Commit must complete the pending-state transition before persistent boot promotion. | Refuse an unsafe handoff. Candidate failures log to the journal and reboot when enabled; a fallback boot must not trigger another reboot. | | Writable-store trust | Delivery checks do not add boot-time hashes or authenticate the manifest. Signing, integrity and measurement remain platform policies; sealed `/usr` attestation does not cover the store. | Do not claim inherited trust. Loss of shared storage can affect both slots; A/B cannot restore it, and boot must not require a network download. | | SELinux | Preserve Enforcing mode and correct payload/mount labels. | Surface access or labeling errors; never disable checks or exempt COSI inputs. | @@ -530,9 +526,9 @@ ms goal is a starting budget, and none of these targets is yet demonstrated. Record staging, activation and rollback outcomes with slot, selected image names, activation method and failure reason in servicing logs and boot -diagnostics. Correlate a recovered update failure with its persistent record's -boot ID. Success requires matching visible content and no healthy commit after -activation failure. +diagnostics. Correlate a recovered update failure by boot ID when journal +messages survive. Success requires matching visible content and no healthy +commit after activation failure. ## Servicing @@ -567,7 +563,7 @@ the loader running on the servicing OS. | Host Configuration | The enum allows future modes without changing the field shape. Initially support `disabled` and `managed`/`acl`; reject unsupported values. Canonicalize aliases and omit defaults from serialized state. | | Loader and manifest | Check the target pair during staging. Optional unknown fields are ignored within supported formats; unsupported versions are rejected. A release number alone does not prove support. | | Per-slot state | Publish the manifest once and retain it with its loader through boot, commit and rollback. Do not regenerate or upgrade the retained slot's manifest. | -| Boot and store contracts | Keep paths, pending-update handoff and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | +| Boot and store contracts | Keep paths, marker lifecycle, one-shot handoff and failure semantics stable. Rollback selects matching extensions before consumers run, without late `trident commit` repair. | Trident's shared servicing state is a separate compatibility concern: older strict `Os` readers may reject new fields or values after rollback. @@ -622,7 +618,7 @@ design/implementation effort (S/M/L), not elapsed-time commitments. | 1. Contracts and compatibility | Freeze the initial API, manifest and boot interfaces; define target-loader format detection and shared-state rollback checks. | M/M | High: staging and shared contracts | | 2. Managed staging and lifecycle | Validate inputs; implement fixed-ROOT placement, atomic first-use publication, inactive-slot replacement and safe disable transitions. | M/L | High: ownership and interrupted writes | | 3. Loader and ACL integration | Implement selection, links and confext preparation; integrate Bootengine ordering/lowerdirs and package the dracut assets before UKI signing. | M/M | High: early-boot integration | -| 4. Update-boot recovery | Define the pending-context handoff; integrate one-shot fallback, persistent diagnostics and commit/cancellation cleanup; prove fallback cannot loop. | M/M | High: state and failure ordering | +| 4. Update-boot recovery | Integrate the `slot-loader.pending` marker, one-shot fallback, journal diagnostics and configurable reboot; prove fallback cannot loop. | M/M | High: state and failure ordering | | 5. Rollout and fault validation | Verify the staged loader/manifest pair, preserve the retained pair and check shared-state readers; execute the grouped fault and rollback tests. | L/L | High: rollback viability | | 6. Performance validation | Measure no-op and sysext-only budgets plus complete confext cost; publish cold/warm results and settle the confext total budget. | S/M | Medium: hardware and image-count variance | diff --git a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md index 39af32a764..fb496a009b 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md +++ b/docs/Development/RFCs/0838-slotted-sysexts/test-plan.md @@ -42,9 +42,9 @@ be refused without changing the protected files. Unrelated runtime activation entries must be preserved or reported as conflicts. Interrupt first-use store creation, target clearing, image writes and manifest -publication, including immediately before and after the final rename. Retry -each case. Retained slots must stay bootable; temporary manifests must never -be read, and incomplete sets must never activate. +publication, including immediately before and after the final rename and marker +write. Retry each case. Retained slots must stay bootable; temporary manifests +must never be read, and incomplete sets must never activate. Switch a previously managed target to `disabled`. Its reused slot gets an empty manifest and no stale owned images; the retained slot keeps its data. @@ -76,19 +76,22 @@ resolved. ## 4. Update failure and rollback -On an uncommitted candidate boot, inject a loader error, crash and timeout. -Each must reboot into the committed OS without re-arming the failed entry. -A failure on the fallback's normal boot must not create a reboot loop. +On a newly deployed slot, inject a loader error, crash and timeout. Each must +retain the marker, log to the journal and, with rollback enabled, reboot into +the committed OS without re-arming the failed entry. Disable rollback in the +loader env file and confirm the failed boot stays stopped. On loader success, +including an empty manifest, confirm the marker is removed without waiting for +commit. A failure on the fallback's normal boot must not create a reboot loop. Also exercise bootloader fallback, failed commit and power loss. Confirm the visible OS and extension markers match after recovery. Fail the pending-state transition during commit: the previous persistent boot default must remain -unchanged. Stale context after commit, cancellation or fallback must not cause -another automatic reboot. Finalize these cases with the pending-context contract. +unchanged. Restaging and cancellation must clear stale markers; a failed +candidate's remaining marker must not cause the fallback to reboot. -Read the failure record from the recovered OS and correlate its boot ID and -reason. Repeat with read-only ROOT, failed diagnostic writes and capture -timeouts: lost diagnostics must be reported but must not block reboot. +Check the failed boot's journal/console for slot, boot ID and reason. Repeat +with read-only ROOT and journal persistence disabled: logging must not block +reboot, and initrd journal messages need not survive it. ## 5. Image construction and version compatibility From a0478dca18be8e906fd35346b9d2758a32aa1fc5 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 21 Oct 2026 10:07:44 -0700 Subject: [PATCH 19/27] Fix typo --- docs/Development/RFCs/0838-slotted-sysexts.md | 64 ++++++++++--------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 2982646cdf..41d2e07d2c 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -43,7 +43,7 @@ those changes. ### Problem Statement -Users build custom images on ACL, often adding stack files or preloaded sysexts +Users build custom images on ACL, often adding loose files or preloaded sysexts on mutable storage. ACL's A/B update replaces the core OS, not those additions. Their stack therefore needs an out-of-band updater even though it already ships in their image. @@ -147,8 +147,8 @@ After reading `acl.slot`, the loader opens this file, not Trident's SQLite database or a directory scan. The loader [returns success without activation](#loader-startup-behavior) when -the store is absent or a valid manifest selects no images. A loader failure on -a newly deployed slot follows the +the store is absent or a valid manifest selects no images. A loader failure on a +newly deployed slot follows the [update-boot failure policy](#update-boot-failure-recovery). #### Writable `/etc` @@ -208,8 +208,9 @@ overwritten. identities, paths and whether the host can activate them. Check that the loader in the target boot image supports the manifest format being written. 2. Select the inactive slot. Remove any old Slot Loader Manifest and - `slot-loader.pending` marker, then synchronize its directory before clearing the - Trident-owned extension directory, including interrupted-staging leftovers. + `slot-loader.pending` marker, then synchronize its directory before clearing + the Trident-owned extension directory, including interrupted-staging + leftovers. 3. Reserve space to stage the new extension files. Write temporary `.raw` files inside the cleared directory, outside systemd's search paths. Verify hashes, filesystem structure and extension compatibility. Reject invalid images with @@ -253,11 +254,11 @@ ROLLBACK_ON_LOADER_FAILURE=true ``` Both paths name the same store: `/sysroot/...` is the initrd's view; -`/var/lib/...` is the symlink destination after switching to the real root. -The paths and `LOADER_MODE` are fixed; `LOADER_MODE` controls slot detection, -not service timing. `ROLLBACK_ON_LOADER_FAILURE` defaults to `true` and can be -set to `false` in this env file to leave a failed boot stopped for inspection. -The failure handler reads this setting independently of the loader. The +`/var/lib/...` is the symlink destination after switching to the real root. The +paths and `LOADER_MODE` are fixed; `LOADER_MODE` controls slot detection, not +service timing. `ROLLBACK_ON_LOADER_FAILURE` defaults to `true` and can be set +to `false` in this env file to leave a failed boot stopped for inspection. The +failure handler reads this setting independently of the loader. The `slot-loader.pending` marker beside each slot's manifest identifies a new slot boot without opening Trident's SQLite datastore in the initrd. No new kernel command-line option or UKI addon is required; `acl.slot` selects the slot. @@ -285,8 +286,8 @@ Unknown fields are ignored in both the manifest and image objects. Compatible optional fields can be added within v1; changes that old loaders must understand require a new format version. -The manifest is written once for each staged set. Boot, commit and rollback -do not rewrite it; it changes only when that slot is reused. +The manifest is written once for each staged set. Boot, commit and rollback do +not rewrite it; it changes only when that slot is reused. An entry named `runtime` resolves to `/sysexts/runtime.raw` or `/confexts/runtime.raw`, according to its array, relative to the store. @@ -331,17 +332,17 @@ open. ##### Update-boot failure recovery -Staging writes `/slot-loader.pending` beside `slot-loader.json` before arming -the one-shot boot attempt. The loader removes the marker only after its own -successful execution, including a valid empty manifest, and synchronizes the +Staging writes `/slot-loader.pending` beside `slot-loader.json` before +arming the one-shot boot attempt. The loader removes the marker only after its +own successful execution, including a valid empty manifest, and synchronizes the slot directory. This does not wait for the rest of the boot or for commit. Loader failure leaves the marker in place. On loader failure, crash or timeout, an initrd failure handler reads `acl.slot` and checks that slot's marker without relying on manifest parsing or the loader -process. It logs the slot, boot ID, unit result and reason to the journal/console. -There is no separate failure record; initrd journal messages may not survive a -reboot unless journal persistence is configured. +process. It logs the slot, boot ID, unit result and reason to the +journal/console. There is no separate failure record; initrd journal messages +may not survive a reboot unless journal persistence is configured. If the marker exists and `ROLLBACK_ON_LOADER_FAILURE=true` (the default), the handler requests a reboot. The one-shot attempt has already been consumed, so @@ -351,9 +352,9 @@ triggers an automatic reboot, preventing fallback failures from looping. The handler does not re-arm the failed entry or restart the loader and must run independently of `/etc` assembly. -Cancellation or restaging clears stale markers. A failed candidate's marker -may remain on shared ROOT after fallback, but only the booted slot's marker -controls recovery. +Cancellation or restaging clears stale markers. A failed candidate's marker may +remain on shared ROOT after fallback, but only the booted slot's marker controls +recovery. Use leveled stderr diagnostics (slot, manifest path, store, activation method and image errors), captured by the journal/console. Return nonzero on failure. @@ -555,8 +556,8 @@ Keep the loader and its boot contracts small, stable and long-lived. The target image is normally expected to contain a loader from the same or a newer release than the staging Trident. The actual check is format support: -Trident must write a manifest that the target loader can read, regardless of -the loader running on the servicing OS. +Trident must write a manifest that the target loader can read, regardless of the +loader running on the servicing OS. | Interface | Compatibility approach | | --- | --- | @@ -572,15 +573,15 @@ strict `Os` readers may reject new fields or values after rollback. *WIP: TODO: re-review* -Rollback returns to the retained loader and its unchanged manifest. If that -pair booted successfully before the update, manifest-version changes in the -new slot do not affect it. There is no need for the old loader to read the -new slot's manifest, or for the new loader to read the old slot's. +Rollback returns to the retained loader and its unchanged manifest. If that pair +booted successfully before the update, manifest-version changes in the new slot +do not affect it. There is no need for the old loader to read the new slot's +manifest, or for the new loader to read the old slot's. **TBD:** how staging discovers target-loader format support, and how preflight -checks shared boot configuration, store layout and servicing-state changes. -Keep those shared contracts stable; preserving a slot's pair does not protect -it from incompatible changes to shared state. +checks shared boot configuration, store layout and servicing-state changes. Keep +those shared contracts stable; preserving a slot's pair does not protect it from +incompatible changes to shared state. ### Policy changes @@ -626,7 +627,8 @@ Production enablement requires passing the initial test areas, demonstrated rollback compatibility and published measurements against the scenario budgets. Unmet performance goals need explicit review; target-loader detection and shared-state compatibility checks must be defined before that gate. No new COSI -fields, systemd source changes or cluster-control-plane integration are required. +fields, systemd source changes or cluster-control-plane integration are +required. ## Counter-Arguments From 2185123cb4709007a55c474e07b708aef680e650 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 21 Oct 2026 10:17:24 -0700 Subject: [PATCH 20/27] Clarifications --- docs/Development/RFCs/0838-slotted-sysexts.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index 41d2e07d2c..b9369e6464 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -70,12 +70,13 @@ it can be reused later. - Support ACL's shared storage without new partitions or systemd source changes. - Let users explicitly enable Trident to update bundled extensions alongside the OS. -- A published store must contain valid manifests for every bootable slot; images - without a store need no initialization. -- Replace the inactive slot's managed extension set on reuse; do not retain - older sets for that slot. +- Base ACL images must continue to boot as-is. +- Clean and reuse the inactive slot for staging an update's extensions, ensuring + the active slot's extensions remain usable and available for any eventual + rollback. - Preserve each platform's filesystem mutability and user-state persistence. -- Keep the policy reusable on platforms with writable A/B storage. +- (Future) Make this sysext management feature reusable on platforms with + writable A/B storage, not only ACL. ### Out of Scope From 76519678cbbac4b7d811e2c06af15917829984f0 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 21 Oct 2026 10:24:54 -0700 Subject: [PATCH 21/27] Fixes --- docs/Development/RFCs/0838-slotted-sysexts.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts.md b/docs/Development/RFCs/0838-slotted-sysexts.md index b9369e6464..c1cb85d1ed 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts.md +++ b/docs/Development/RFCs/0838-slotted-sysexts.md @@ -85,7 +85,9 @@ it can be reused later. - Relocating the Trident-owned store; see [Future Possibilities](#future-possibilities). - Updating arbitrary files on shared mutable storage. -- Runtime Update for managed extension/policy changes in the initial release. +- Runtime Update for managed extension changes in the initial release. Allowing + for runtime updates of managed extensions should be planned in a standalone + document. - New image-less updates, initrd-scoped sysext content or ESP payload storage. - Bundled disk streaming; 0789's exclusion remains. @@ -694,6 +696,9 @@ labels, compatibility and ordering; version alone is insufficient. Managed sysext-only Runtime Update can follow; ACL cannot replace its `/etc` lowerdirs live. +An implementation of this could involve an extension-only COSI, or +OCI-distributed extension files. + ### Btrfs-backed managed store On Btrfs-backed storage, the store could use a pair of subvolumes, one per slot, From 2a4a31b4ce3808f1652860d017d2a60fd74e5f4b Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 21 Oct 2026 10:35:14 -0700 Subject: [PATCH 22/27] docs: clarify slot-specific runtime setup in the overview diagram Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../0838-managed-extensions-overview.svg | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/Development/RFCs/0838-slotted-sysexts/0838-managed-extensions-overview.svg b/docs/Development/RFCs/0838-slotted-sysexts/0838-managed-extensions-overview.svg index f185481e82..281f88bf83 100644 --- a/docs/Development/RFCs/0838-slotted-sysexts/0838-managed-extensions-overview.svg +++ b/docs/Development/RFCs/0838-slotted-sysexts/0838-managed-extensions-overview.svg @@ -3,9 +3,11 @@ COSI delivery, per-slot storage and two boot-time activation paths Trident replaces the inactive slot's extension files in the shared store, preserving the active slot. At each boot the initrd loader reads acl.slot - and that slot's slot-loader.json. It mounts the selected - confext as a lower layer in the writable etc overlay, and creates sysext links in run. - After switch-root, native systemd-sysext merges the selected system extension into usr. + and that slot's slot-loader.json. It creates symlinks in /run/extensions to that + slot's sysexts and mounts its confexts under /run as additional etc lowerdirs. + Bootengine uses those lowerdirs in the writable etc overlay. The /run filesystem + survives switch-root, keeping the slot-specific links available for native + systemd-sysext to merge the selected system extensions into usr.