You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Move the agent to /opt/unbounded/agent for Azure Container Linux - #323
Azure Container Linux mounts /usr read-only and is provisioned only by Ignition. This moves AKS Flex Node's host-side files to /opt/unbounded/agent so it can run there, and documents how to bootstrap such a host through Ignition.
Host root
The agent's host-side files go under /opt/unbounded/agent (the library's hostroot.Path): bin/aks-flex-node, the blue/green layout and recovery script under lib/aks-flex-node, and the agent library's helpers. The config, state, logs and units don't move. /opt/unbounded/agent/bin isn't on the default PATH.
/opt/unbounded itself belongs to the host, which may stage offline artifacts or OCI layouts there. The agent creates it with mode 0755 when it's missing and otherwise never changes or removes it, or anything else in it.
A host installed by an earlier release keeps its layout under /usr/local at first. start, the daemon, agent-upgrade, recover-agent-upgrade and the nspawn pre-start hook first link /opt/unbounded/agent to it, inside whatever /opt/unbounded already is, so the units and links that release wrote stay valid and the host can go back to it.
Once neither the current nor the last-good binary is from an earlier release, the daemon moves the files into a real /opt/unbounded/agent with the agent library's hostroot.ReconcileMove, under the activation lock. The move spans two daemon starts:
The first copies the files, rewrites the agent and recovery units, the recovery script, the nspawn hooks and the LocalDNS unit, and queues the daemon's restart. Before writing the nspawn hooks it installs this release's nspawn lifecycle helper from the running binary: the copied helper is the earlier release's binary, whose pre-start would point the hooks back at the /usr/local helper the move removes, so the machine would not start again after its next restart. It queues the restart through the AgentUpgrade executor's Restart, which first runs systemctl reset-failed aks-flex-node-agent.service (Don't count an upgrade's restart toward the agent's start limit #386): systemd counts the planned restart against the unit's start limit, from v255 a daemon-reload no longer clears it, and a refused restart would leave the agent stopped. Nothing under /usr/local is removed yet.
The restarted daemon, running from the copy, rewrites the units again and removes the files under /usr/local and the emptied /usr/local/lib/aks-flex-node.
A move that is interrupted, or whose restart fails, leaves both copies in place, and the next start finishes it.
The daemon keeps the reconcile gate shut after queuing its restart, when the move swapped the copy in and then failed, since its AgentUpgrade executor would still name /usr/local, and while a move hasn't finished, so its next start retries the move's last step. If it isn't restarted within two minutes it exits, and Restart= starts it again. That spacing keeps it clear of the unit's start limit, so it never trips recovery into a rollback. The move runs once the daemon has written its own service assets, after any pending AgentUpgrade is judged, and before the gate opens.
Reset and uninstall.sh sweep both roots. uninstall.sh removes /opt/unbounded/agent itself, as the link to /usr/local or an emptied directory, and an interrupted move's /opt/unbounded/agent.staging and .moving marker. Like the agent library, it keeps a link an operator made at /opt/unbounded/agent to somewhere else. It never removes /opt/unbounded.
install.sh and bootstrap.sh choose the directory from the host and never run the binary to decide: /opt/unbounded/agent/bin on a host already installed there; /usr/local/bin where that's writable, as before, so an earlier release finds itself there; and /opt/unbounded/agent/bin otherwise, as on Azure Container Linux. /opt/unbounded alone doesn't count as an installation. --install-dir and AKS_FLEX_NODE_INSTALL_DIR are deprecated: a value that names another directory is ignored, with a warning. After a reset, bootstrap.sh finds the layout reset kept under the root it installs into, the host root or /usr/local, as install.sh does.
start and the daemon copy the running binary to /opt/unbounded/agent/bin when no usable one is there. The daemon and start remove the copy in /usr/local/bin once the library's hostroot.LegacyReleased reports nothing runs from /usr/local: a finished installation under /opt/unbounded/agent, or one an operator linked elsewhere, not while a move is under way. The plain binary doesn't mark an earlier installation; only the blue/green layout does. The daemon keeps that copy while it runs from it, as on a host an earlier release installed directly in /usr/local/bin, and the next start, from /opt/unbounded/agent, removes it.
An AgentUpgrade to an earlier release isn't refused as such. On a host installed under a real /opt/unbounded/agent its daemon can't find its files and the agent rolls back to last-good, as for any upgrade whose daemon can't start. Stage and direct activation refuse while a move hasn't finished, since the files still under /usr/local are a whole installation that release would run from. Otherwise they first remove the copy in /usr/local/bin, and refuse to switch while something there still leads to an executable, which that release would adopt as its own binary. A direct activation checks again, under the activation lock, that the host root still resolves where it did when it built its paths, and refuses if a move changed it in between. Rollback is quick, because systemd 255 runs the recovery unit on the first failure; Refuse an AgentUpgrade to a release that predates the host root #377 tracks refusing such a candidate before switching. On Azure Container Linux, SELinux denies the recovery script's systemctl reset-failed, so nothing can clear the agent unit's start limit there (Azure Container Linux denies systemctl reset-failed, so nothing clears the agent unit's start limit #385).
A move is never made onto a filesystem that can't run programs: it's skipped where /opt is mounted noexec, and the copy is run before any unit names it and undone if it can't. On a fresh host, preflight fails where the agent's install directory would be on a noexec filesystem (the library's host.Preflight). start checks the binary it copies under /opt/unbounded/agent runs, and fails before the machine is registered if it can't.
When the host root can't be migrated, the nspawn pre-start hook keeps the machine's config and lets it start, and recover-agent-upgrade still restores last-good. The daemon refuses to start, which is where the problem is reported.
Clients on the host ran the kubelet's exec credential, which names the binary inside the machine. They now run the host's.
Reset now also removes LocalDNS state and the nspawn lifecycle helper.
operations.md describes the current layout. docs/usage/migration.md describes how a host an earlier release installed moves, and upgrades to and from earlier releases.
Ignition
Bootstrapping a host through Ignition is documented, as cloud-init is, rather than rendered by a command. scripts/aks-flex-node-bootstrap.bu is a Butane config (flatcar 1.1.0, which renders Ignition 3.4.0) built from three files:
the release's bootstrap.sh, unchanged;
base-config.json, which is {} when bootstrap data is fetched from AKS;
bootstrap.env, which holds bootstrap.sh's AKS_FLEX_NODE_* settings as a systemd environment file.
It enables aks-flex-node-bootstrap.service, which:
runs the script on first boot;
fails an attempt that hasn't finished after 30 minutes, and retries with a backoff up to 5 minutes until the agent is installed, and doesn't run after that;
removes /etc/aks-flex-node/first-boot, whose files carry the token and signed agent URL, once bootstrap succeeds.
A service principal credential is a commented-out entry in the template. "Hosts provisioned by Ignition" in getting-started shows the files, how to render them with butane or its container image, and how to quote values in the environment file. It also says the pinned agent version applies only when a host is provisioned, so a reimaged host needs an AgentUpgrade to the pool's version, and that the config should pin a version with its digest rather than a signed URL that expires. And it names the way to rejoin after a reset: provision the host again from its Ignition config. docs/usage/azure-container-linux.md covers upgrades, the minimum version, reprovisioning and rejoining on these hosts; how they take OS updates is #379.
Reset and uninstall.sh disable and remove the unit by name. A host not provisioned through Ignition has no such unit, and nothing is done about it.
Tests
bootstrap_test.sh now runs in make test-install and PR checks, in a private mount namespace, isolated from the host's /usr/local and /opt. The script tests also check that the installers and uninstall.sh leave an existing /opt/unbounded, its mode and its files alone, under a writable and a read-only /usr/local. A Go test checks each script's HOST_ROOT and LEGACY_ROOT equal hostroot.Path and hostroot.LegacyPath. Another ties the Butane template to the agent: the unit name reset removes, the agent-unit condition, the unit's type, start timeout and restart settings, the files it runs and removes, and their modes. uninstall_test.sh covers a host without the first-boot unit. bootstrap_test.sh covers a rerun after reset with the layout kept under the host root and under /usr/local, the refusal while the agent unit is installed, and the refusal of any other link at the binary path.
The daemon's hold-and-exit decision, the bounded wait and its start-limit spacing, installing the lifecycle helper before the nspawn hooks, the seed's gating on hostroot.LegacyReleased, the downgrade guard, and the check that a copied binary runs have unit tests.
The Azure e2e upgrades a node installed by v0.2.0 to the build under test and checks it is linked, with /opt/unbounded0755 root. It then upgrades again and checks it is moved into /opt/unbounded/agent. Before that second upgrade it mounts /optnoexec and checks the daemon keeps the host linked with nothing copied. After the move it checks the nspawn lifecycle helper is the running agent's binary, restarts the machine through it as a host reboot would, and checks the node rejoins and runs a workload with the hooks still on the moved helper. It then checks that an AgentUpgrade back to the earlier release fails before switching while /usr/local/bin/aks-flex-node is executable, and otherwise rolls back with nothing installed under /usr/local. It also checks, after a join under /opt/unbounded/agent, that the copy in /usr/local/bin is removed.
hack/acl is a local harness that boots Azure Container Linux from the documented Butane template, rendered with the Butane container image, and joins it to kind. It quotes a probe value in bootstrap.env as the docs say and checks that it arrives unchanged. It also checks that /opt/unbounded is created 0755 and kept after reset.
Boot arguments: they go into the image's first-boot addon, which ignition-quench deletes after the first boot. ukiboot.py is copied from unbounded.
Upgrades:acl.py upgrade runs, on the VM, an AgentUpgrade to a second build (U2), one to a candidate whose daemon can't start, which rolls back to last-good (U3), a reboot (U7), and a check for SELinux denials that name the agent (U10). Running it in CI is Run the Azure Container Linux harness in CI #380.
From the review: #377 (refuse a candidate older than the host root), #378 (refresh the nspawn lifecycle helper on every daemon start), #379 (the ACL OS update model), #380 (the ACL harness in CI), #381 (an operator's link at /opt/unbounded/agent in the install scripts), #382 (uninstall.sh's fallback), #383 (first-boot failures that can't succeed on a retry), #384 (E2E hardening). Found by the harness: #385 (reset-failed denied on Azure Container Linux). Later: #398 (direct activation can remove the old binary the agent service still runs from), and #399 (an Azure Container Linux VM in the Azure E2E).
The agent library places its own binaries and helper scripts under a fixed
/usr/local prefix. Azure Container Linux mounts /usr read-only, so nothing
can be installed there and a Flex Node cannot run on it at all.
The version that makes the prefix configurable is not released yet, so this
pins the commit under review upstream rather than a tag. That is temporary
and has to become a release version before this merges.
The same build also reports host package capability truthfully instead of
assuming a package manager exists, which matters here because the image has
none.
Hosts that mount /usr read-only cannot take the agent's own binaries and
helper scripts in the default location. A config field lets them name a
writable one.
The prefix is declared, never inferred. Keying off distribution identity
would misclassify a hardened host whose /usr happens to be read-only, and
would silently relocate files on any host whose os-release changed. A wrong
guess is expensive to recover from, because generated systemd units reference
these paths absolutely.
Validation is delegated to the agent library rather than duplicated. The
prefix is interpolated into generated units and into a shell script, neither
of which quotes it, so the accepted syntax is deliberately narrow; keeping
that rule in one place stops the two projects from disagreeing about what is
accepted.
defaultAgentUpgradePaths hardcoded /usr/local/bin and
/usr/local/lib/aks-flex-node, both under a read-only /usr on Azure Container
Linux, and it is called with no override, so the location could not be
changed.
The prefix is read from this project's own config rather than the agent
library's applied config. That distinction was found the hard way: the
library exposes a lookup for exactly this, but bootstrap here never writes
that file, so on a real host it is absent, the lookup silently returns the
default, and the failure surfaces later as
initialize agent binary layout: no executable agent binary found
which names neither the prefix nor the path it looked in.
An unreadable or absent config yields the empty prefix, which selects the
default. That is what a host installed before this existed actually has on
disk, so nothing changes for it.
Callers that already hold a config pass it directly; the lookup exists for
entry points started by systemd, which cannot inherit the prefix from the
environment that ran bootstrap.
recoveryScriptPath was a second hardcoded location under /usr/local/lib. On
Azure Container Linux this fails bootstrap after the node has already joined:
install-service: write /usr/local/lib/aks-flex-node/aks-flex-node-recovery.sh:
mkdir /usr/local/lib/aks-flex-node: read-only file system
which is a worse failure than refusing up front, because the cluster has a
node in it by then.
The constant stays, because it is also the substitution placeholder inside
the embedded recovery unit. Only the install location becomes prefix
relative; the literal in the unit template has to keep matching the asset.
systemdSystemDir is deliberately not prefix relative. Units have to live where
systemd looks for them, and /etc is writable even when /usr is not.
The prefix work it depends on was consolidated upstream and rebased onto
main after the bootstrap ownership change merged, so the previously pinned
commit no longer exists on any branch.
Still an unreleased commit rather than a tag, and still has to become a
release version before this merges.
The recovery script ships with the default-prefix paths as literals, and
installation rewrites them for the host's layout. The rewrite looked for the
paths resolved from the installed config instead of the literals. On a host
with a custom prefix those are the prefixed paths, which the script does not
contain, so nothing was replaced and the script kept reading last-good from
/usr/local. A failed upgrade could then not be rolled back.
The replacement now keys on the default layout, which is what the embedded
script contains, and a test pins that the asset and the default layout agree.
UninstallService read the host prefix from /etc/aks-flex-node/config.json when
it ran. The AgentReset MachineOperation and the RP delete path both run
ResetNode first, which removes that directory, and only then uninstall the
service. By then the prefix read as the default, so on a host with a custom
prefix the recovery script was left behind.
The prefix is now a parameter. The daemon passes the one from its loaded
config, and the reset command reads the installed config before anything is
removed. The recovery script is removed under that prefix and under the
default, so a host that changed prefix is also cleaned up.
Removal checks for the file first. On a read-only /usr, unlinking a file that
does not exist returns EROFS rather than ENOENT, which would fail the
default-prefix sweep on a file that was never there.
Reset called the library's interface and route cleanups directly and skipped
its LocalDNS cleanup, so every reset of a node with LocalDNS enabled left the
unit, the nft table, the dummy interface and the helper script on the host. It
also never removed the nspawn lifecycle helper.
Reset now runs the library's CleanupNetwork, which includes the LocalDNS
cleanup, and removes the nspawn lifecycle helper. Both helpers live under the
host prefix, so reset takes the prefix and removes them under it and under the
default. The managed agent binaries are still kept.
The e2e RP delete check now also asserts the LocalDNS state, the nspawn helper
and the recovery script are gone. The MSI node enables LocalDNS, so this runs
against a real LocalDNS setup.
Repave cleans up the network with the library's CleanupNetwork, which removes
the LocalDNS helper. Without the prefix it only looks under /usr/local, so on a
host with a custom prefix the old helper stayed until StartNode wrote a new
one. That was harmless because StartNode runs right after, but it now uses the
same prefix as the rest of the node.
…t one
install.sh and uninstall.sh always used /usr/local. The agent resolves its
binaries from agent.hostPrefix, so on a host with a custom prefix the installer
put the binary where the agent does not look. On Azure Container Linux it could
not install at all, because /usr is read-only.
Both scripts now take the prefix from agent.hostPrefix in the node config when
it exists, or from AKS_FLEX_NODE_HOST_PREFIX, which covers the usual case where
the config is written after installation. install.sh fails if the two disagree.
install.sh also treated Azure Container Linux as Azure Linux 3, since it
reports ID=azurelinux with a 3.x VERSION_ID. It is now identified by
VARIANT_ID=azurecontainerlinux, or by ID_LIKE=flatcar for an image that drops
VARIANT_ID. Installing there with the default prefix fails and says which
variable and config field to set.
uninstall.sh reads the prefix before running reset, which removes the config.
It now also removes the managed blue/green layout, which reset keeps and which
uninstall previously left behind, and the recovery unit when it falls back to
removing units directly. It can be sourced now, and uninstall_test.sh covers
the prefix handling and binary removal.
bootstrap.sh always installed the agent in /usr/local/bin, or wherever
--install-dir pointed. The agent resolves its binaries from agent.hostPrefix,
so any other combination left it looking in the wrong place, and on Azure
Container Linux /usr is read-only.
--host-prefix, or AKS_FLEX_NODE_HOST_PREFIX, now sets agent.hostPrefix, and
the binary goes in <prefix>/bin. Without the flag the prefix comes from the
base config and config overrides, and defaults to /usr/local as before. The
prefix is resolved before rendering because fetching bootstrap data runs the
installed binary, so bootstrap data cannot move it; a response that tries is
refused. --install-dir is deprecated and only accepted when it names the same
directory.
The test runs every case as root, so a regression that fell back to the
default prefix would have replaced a real /usr/local/bin/aks-flex-node. It now
runs in a private mount namespace with /usr/local overlaid, fails if any
prefixed case writes there, and works from inside its temp dir so a relative
path cannot land in the checkout.
scripts/bootstrap_test.sh existed but nothing ran it, so neither make test nor
PR checks caught a bootstrap.sh regression. make test-install, which both use,
now runs it after the installer and uninstaller tests.
Azure Container Linux is provisioned only by Ignition and has a read-only /usr,
so bootstrap.sh cannot be downloaded and run by hand at first boot.
aks-flex-node ignition renders an Ignition config that writes the embedded
bootstrap.sh, with the base config in place of its placeholder, and any
service principal credential, and enables aks-flex-node-bootstrap.service to
run the script with the arguments given after --. The agent goes under
/opt/aks-flex-node unless --host-prefix or the base config names another
prefix.
The unit retries with a growing delay until the agent unit exists, and is
skipped after that, since bootstrap.sh refuses to run over an existing
deployment. It removes the script, which carries the base config, once
bootstrap succeeds. The arguments are checked against the script's options,
and a test keeps that list in step with the script, so mistakes surface on the
workstation instead of at first boot.
Reset disables, stops, and removes the unit before stopping the agent, since
in the daemon's reset paths stopping the agent ends the reset. uninstall.sh
does the same when there is no binary to run reset with.
hack/acl/acl.py boots Azure Container Linux under QEMU from the config that
aks-flex-node ignition renders and joins it to a kind cluster, with the AKS
Flex controller deployed in the cluster in place of Azure. It checks that the
first-boot unit installs the agent under the prefix and removes the script
that carries the token, that a workload runs, that a reboot skips the unit,
and that reset, through a MachineOperation or the CLI, removes everything it
should. Each removed path is first checked to exist.
It passes on build 3.0.20260818 with both reset paths, and fails as it should
against an agent whose reset does not stop the first-boot unit.
The networking, VM launch, and UKI patching are adapted from the unbounded
harness; ukiboot.py is copied from it unchanged.
configuration.md did not list agent.hostPrefix. It now says what the prefix
moves and what it does not, that hosts with a read-only /usr need it, and to
reset before changing it.
The getting-started list of what bootstrap.sh does put the agent download after
the bootstrap data fetch, but the binary is installed first, and now under the
host prefix. The bootstrap design says why the Ignition wrapper keys on the
agent unit rather than a completion marker, which reset would leave behind.
Philip Lombardi (plombardi89)
changed the title
Support Azure Container Linux hosts with a configurable install prefix
Support Azure Container Linux: host install prefix and Ignition bootstrap
Sep 24, 2026
The agent library build now includes the fixes from its review: reset takes
the host prefix once, the applied config's checksum is checked before its
prefix is used, and the nspawn machines order after the nftables flush.
hack/acl/ukiboot.py is copied unchanged from the same build. It now requires
exactly one UKI, measures the command line in encoded bytes, and cleans up
after a failed qemu-nbd start.
newRootCommand attaches the package-level token.Command, so building two
roots at once writes its parent from both goroutines. The ignition
registration test ran in parallel with the existing root command test, and
the race detector in PR checks caught it. It now runs serially.
The host prefix is replaced by the host root from the agent library,
/opt/unbounded, on every host. The binary layout, the recovery script and unit,
and the library's helpers are all built from the root resolved through symlinks.
A host installed by an earlier release keeps its layout under /usr/local. start,
the daemon, agent-upgrade, recover-agent-upgrade, and the nspawn hooks first
link /opt/unbounded to it, so the paths the older release wrote into links and
units still match the ones built here. Reset and uninstall.sh sweep both roots
and do not depend on the link.
A hidden host-root command prints the root. install.sh and bootstrap.sh ask the
downloaded binary for it, from a copy under /var/lib/aks-flex-node rather than
a possibly noexec /tmp. A release without the command goes to /usr/local/bin as
before, and where that is read-only the scripts say the release is too old. The
agent library refuses an AgentUpgrade to such a release on a host installed
under /opt/unbounded.
Clients that run on the host used the kubelet's exec credential, which names the
binary inside the machine, /usr/local/bin/aks-flex-node. They now run the one
under the host root.
agent.hostPrefix, bootstrap.sh --host-prefix, AKS_FLEX_NODE_HOST_PREFIX,
ignition --host-prefix, and the Azure Container Linux check in install.sh are
removed. The Azure e2e gains an upgrade from v0.2.0, the agent-upgrade e2e
checks that a release before the host root is refused, and the agent library
pin moves to Azure/unbounded#815 at d01744cd.
Philip Lombardi (plombardi89)
changed the title
Support Azure Container Linux: host install prefix and Ignition bootstrap
Support Azure Container Linux: install under /opt/unbounded and Ignition bootstrap
Sep 25, 2026
A rejoin took the agent-upgrade path whenever a managed layout existed, but reset removes the service and the config directory and keeps the layout, so the config copy that follows failed. Under /usr/local the layout's directory was not readable by the SSH user, so the check never matched and every rejoin used install.sh. Under /opt/unbounded it matches. The upgrade path is now taken only while the service is installed.
…L pages
operations.md's "Where the agent is installed" now describes only where
the agent is installed now. How a host an earlier release installed
under /usr/local moves to /opt/unbounded/agent, what reset and uninstall
do meanwhile, and how upgrades to and from earlier releases behave go to
a new page, migration.md, which operations.md links.
The "Azure Container Linux" section under "Managed agent upgrade" moves
to a page of its own, azure-container-linux.md, linked from there: most
of it is about operating such a host, not about upgrades. Both pages are
in the usage index and the URL contract.
main's #401 moves Go to 1.26.10 in go.mod and the controller image, where this branch had 1.26.9. Both take main's 1.26.10, which meets unbounded v0.15.0's 1.26.9.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The cross-restart filesystem migration, systemd recovery interactions, and platform-specific limitations require final human validation across supported host layouts.
…s unfinished
The move's second pass, run by the restarted daemon, rewrites the units
before it removes the files under /usr/local and the library's .moving
marker. If the rewrite failed, as resolving the machine's nspawn config
does on a GPU host whose NVIDIA state is incomplete, the daemon only
logged it and opened its gate, and nothing retried until its next start.
Meanwhile LegacyReleased reported false, so guardLegacySeed let an
AgentUpgrade or a direct activation through. A release before the host
root would then run from the files left under /usr/local rather than
fail to start and roll back.
holdForHostRoot now also holds while hostRootMoveUnfinished reports a
real /opt/unbounded/agent that still holds the marker, so the daemon
takes no work, exits after two minutes, and its next start retries the
second pass. guardLegacySeed, which both Stage and direct activation
run, refuses first while the move is unfinished, and says to restart
the agent's unit. A host linked to /usr/local has no marker and is
unaffected. The marker's name is the library's, unexported, as in
uninstall.sh.
migration.md says an agent whose move hasn't finished takes no work and retries at its next start, and that AgentUpgrades and direct activations are refused until the move finishes.
A direct activation checks for an unfinished move in guardLegacySeed,
before it holds the activation lock, and then builds its layout from the
host root. A daemon whose first pass renamed its copy into place between
the two leaves the root unchanged by the time Preflight compares it
under the lock, but the move hasn't finished, and the files under
/usr/local that a release before the host root would run from are still
there.
Preflight now also refuses, under the lock, while hostRootMoveUnfinished
reports a move that hasn't finished, or when it can't tell, and says to
restart the agent's unit to finish it. guardLegacySeed's check stays,
for the refusal before anything is staged, and for Stage, whose daemon
takes no AgentUpgrade during its own move.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Cross-release migration and recovery require final human validation, including resolution of the local-reset concurrency issue.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Coordinate local reset with migration before resolving cleanup paths
pkg/daemon/hostroot_move.go:88
Local reset does not hold this lock: pkg/cmd/reset/reset.go:27–32 constructs both UninstallService and ResetNode before executing either, and UninstallService stops the daemon only after removing its files. If migration swaps the host-root link for the copied directory after those tasks capture their paths but before the daemon stops, reset still uses the legacy paths. It can then succeed while leaving the copied nspawn lifecycle helper and recovery script under /opt/unbounded/agent. Coordinate local reset with the activation lock before constructing the cleanup tasks and keep migration excluded until the daemon is stopped, or stop the daemon before resolving any cleanup paths. Add coverage for reset overlapping the move.
#323 lands first, as only it takes the agent past unbounded v0.8.0;
this branch then rebases onto it. List what that takes: skipping its
host-side steps in the container, --installed-agent PATH, one helper
for the credential plugin, the host's /opt/unbounded instead of its
/usr/local, and docs presenting host mode as interim. The lifecycle
helper's item gains the hooks' design, and the unbounded items #323's
version and #962's status. The Azure VM item covers joining with a
managed identity and a service principal, and host mode on ACL
retires once this branch lands, rather than #323 closing.
#383 asks, for #323's bootstrap unit, whether a unit stuck retrying
keeps ACL from reaching multi-user.target, which its WantedBy= orders
after it. It doesn't: with a stand-in unit with this branch's bootstrap
unit's settings that always fails, the harness's VM reached
multi-user.target and running while the unit kept retrying. The
distinct exit code #383 asks of bootstrap.sh, for
RestartPreventExitStatus=, applies to this unit too once #323 lands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Actions
Failure
Azure Container Linux mounts
/usrread-only and is provisioned only by Ignition. This moves AKS Flex Node's host-side files to/opt/unbounded/agentso it can run there, and documents how to bootstrap such a host through Ignition.Host root
The agent's host-side files go under
/opt/unbounded/agent(the library'shostroot.Path):bin/aks-flex-node, the blue/green layout and recovery script underlib/aks-flex-node, and the agent library's helpers. The config, state, logs and units don't move./opt/unbounded/agent/binisn't on the defaultPATH./opt/unboundeditself belongs to the host, which may stage offline artifacts or OCI layouts there. The agent creates it with mode0755when it's missing and otherwise never changes or removes it, or anything else in it.A host installed by an earlier release keeps its layout under
/usr/localat first.start, the daemon,agent-upgrade,recover-agent-upgradeand the nspawn pre-start hook first link/opt/unbounded/agentto it, inside whatever/opt/unboundedalready is, so the units and links that release wrote stay valid and the host can go back to it.Once neither the current nor the last-good binary is from an earlier release, the daemon moves the files into a real
/opt/unbounded/agentwith the agent library'shostroot.ReconcileMove, under the activation lock. The move spans two daemon starts:/usr/localhelper the move removes, so the machine would not start again after its next restart. It queues the restart through the AgentUpgrade executor'sRestart, which first runssystemctl reset-failed aks-flex-node-agent.service(Don't count an upgrade's restart toward the agent's start limit #386): systemd counts the planned restart against the unit's start limit, from v255 adaemon-reloadno longer clears it, and a refused restart would leave the agent stopped. Nothing under/usr/localis removed yet./usr/localand the emptied/usr/local/lib/aks-flex-node.A move that is interrupted, or whose restart fails, leaves both copies in place, and the next start finishes it.
The daemon keeps the reconcile gate shut after queuing its restart, when the move swapped the copy in and then failed, since its AgentUpgrade executor would still name
/usr/local, and while a move hasn't finished, so its next start retries the move's last step. If it isn't restarted within two minutes it exits, andRestart=starts it again. That spacing keeps it clear of the unit's start limit, so it never trips recovery into a rollback. The move runs once the daemon has written its own service assets, after any pending AgentUpgrade is judged, and before the gate opens.Reset and
uninstall.shsweep both roots.uninstall.shremoves/opt/unbounded/agentitself, as the link to/usr/localor an emptied directory, and an interrupted move's/opt/unbounded/agent.stagingand.movingmarker. Like the agent library, it keeps a link an operator made at/opt/unbounded/agentto somewhere else. It never removes/opt/unbounded.install.shandbootstrap.shchoose the directory from the host and never run the binary to decide:/opt/unbounded/agent/binon a host already installed there;/usr/local/binwhere that's writable, as before, so an earlier release finds itself there; and/opt/unbounded/agent/binotherwise, as on Azure Container Linux./opt/unboundedalone doesn't count as an installation.--install-dirandAKS_FLEX_NODE_INSTALL_DIRare deprecated: a value that names another directory is ignored, with a warning. After a reset,bootstrap.shfinds the layout reset kept under the root it installs into, the host root or/usr/local, asinstall.shdoes.startand the daemon copy the running binary to/opt/unbounded/agent/binwhen no usable one is there. The daemon andstartremove the copy in/usr/local/binonce the library'shostroot.LegacyReleasedreports nothing runs from/usr/local: a finished installation under/opt/unbounded/agent, or one an operator linked elsewhere, not while a move is under way. The plain binary doesn't mark an earlier installation; only the blue/green layout does. The daemon keeps that copy while it runs from it, as on a host an earlier release installed directly in/usr/local/bin, and the next start, from/opt/unbounded/agent, removes it.An AgentUpgrade to an earlier release isn't refused as such. On a host installed under a real
/opt/unbounded/agentits daemon can't find its files and the agent rolls back to last-good, as for any upgrade whose daemon can't start. Stage and direct activation refuse while a move hasn't finished, since the files still under/usr/localare a whole installation that release would run from. Otherwise they first remove the copy in/usr/local/bin, and refuse to switch while something there still leads to an executable, which that release would adopt as its own binary. A direct activation checks again, under the activation lock, that the host root still resolves where it did when it built its paths, and refuses if a move changed it in between. Rollback is quick, because systemd 255 runs the recovery unit on the first failure; Refuse an AgentUpgrade to a release that predates the host root #377 tracks refusing such a candidate before switching. On Azure Container Linux, SELinux denies the recovery script'ssystemctl reset-failed, so nothing can clear the agent unit's start limit there (Azure Container Linux denies systemctl reset-failed, so nothing clears the agent unit's start limit #385).A move is never made onto a filesystem that can't run programs: it's skipped where
/optis mountednoexec, and the copy is run before any unit names it and undone if it can't. On a fresh host,preflightfails where the agent's install directory would be on anoexecfilesystem (the library'shost.Preflight).startchecks the binary it copies under/opt/unbounded/agentruns, and fails before the machine is registered if it can't.When the host root can't be migrated, the nspawn pre-start hook keeps the machine's config and lets it start, and
recover-agent-upgradestill restores last-good. The daemon refuses to start, which is where the problem is reported.Clients on the host ran the kubelet's exec credential, which names the binary inside the machine. They now run the host's.
Reset now also removes LocalDNS state and the nspawn lifecycle helper.
operations.mddescribes the current layout.docs/usage/migration.mddescribes how a host an earlier release installed moves, and upgrades to and from earlier releases.Ignition
Bootstrapping a host through Ignition is documented, as cloud-init is, rather than rendered by a command.
scripts/aks-flex-node-bootstrap.buis a Butane config (flatcar1.1.0, which renders Ignition 3.4.0) built from three files:bootstrap.sh, unchanged;base-config.json, which is{}when bootstrap data is fetched from AKS;bootstrap.env, which holdsbootstrap.sh'sAKS_FLEX_NODE_*settings as a systemd environment file.It enables
aks-flex-node-bootstrap.service, which:/etc/aks-flex-node/first-boot, whose files carry the token and signed agent URL, once bootstrap succeeds.A service principal credential is a commented-out entry in the template. "Hosts provisioned by Ignition" in getting-started shows the files, how to render them with
butaneor its container image, and how to quote values in the environment file. It also says the pinned agent version applies only when a host is provisioned, so a reimaged host needs an AgentUpgrade to the pool's version, and that the config should pin a version with its digest rather than a signed URL that expires. And it names the way to rejoin after a reset: provision the host again from its Ignition config.docs/usage/azure-container-linux.mdcovers upgrades, the minimum version, reprovisioning and rejoining on these hosts; how they take OS updates is #379.Reset and
uninstall.shdisable and remove the unit by name. A host not provisioned through Ignition has no such unit, and nothing is done about it.Tests
bootstrap_test.shnow runs inmake test-installand PR checks, in a private mount namespace, isolated from the host's/usr/localand/opt. The script tests also check that the installers anduninstall.shleave an existing/opt/unbounded, its mode and its files alone, under a writable and a read-only/usr/local. A Go test checks each script'sHOST_ROOTandLEGACY_ROOTequalhostroot.Pathandhostroot.LegacyPath. Another ties the Butane template to the agent: the unit name reset removes, the agent-unit condition, the unit's type, start timeout and restart settings, the files it runs and removes, and their modes.uninstall_test.shcovers a host without the first-boot unit.bootstrap_test.shcovers a rerun after reset with the layout kept under the host root and under/usr/local, the refusal while the agent unit is installed, and the refusal of any other link at the binary path.hostroot.LegacyReleased, the downgrade guard, and the check that a copied binary runs have unit tests./opt/unbounded0755root. It then upgrades again and checks it is moved into/opt/unbounded/agent. Before that second upgrade it mounts/optnoexecand checks the daemon keeps the host linked with nothing copied. After the move it checks the nspawn lifecycle helper is the running agent's binary, restarts the machine through it as a host reboot would, and checks the node rejoins and runs a workload with the hooks still on the moved helper. It then checks that an AgentUpgrade back to the earlier release fails before switching while/usr/local/bin/aks-flex-nodeis executable, and otherwise rolls back with nothing installed under/usr/local. It also checks, after a join under/opt/unbounded/agent, that the copy in/usr/local/binis removed.hack/aclis a local harness that boots Azure Container Linux from the documented Butane template, rendered with the Butane container image, and joins it to kind. It quotes a probe value inbootstrap.envas the docs say and checks that it arrives unchanged. It also checks that/opt/unboundedis created0755and kept after reset.--imageit boots the Azure build of the image, which it exports from the shared compute gallery through a temporary managed disk, as unbounded's agent e2e does (e2e: boot the Azure Container Linux host from the shared gallery image unbounded#885).ignition-quenchdeletes after the first boot.ukiboot.pyis copied from unbounded.acl.py upgraderuns, on the VM, an AgentUpgrade to a second build (U2), one to a candidate whose daemon can't start, which rolls back to last-good (U3), a reboot (U7), and a check for SELinux denials that name the agent (U10). Running it in CI is Run the Azure Container Linux harness in CI #380.up,upgrade --no-rebootandtestpass on2a376a6, on Azure Container Linux 3.0.20261020 (gallery version 3.20261021.1). The only denials are Azure Container Linux denies systemctl reset-failed, so nothing clears the agent unit's start limit #385's.Follow-ups
From the review: #377 (refuse a candidate older than the host root), #378 (refresh the nspawn lifecycle helper on every daemon start), #379 (the ACL OS update model), #380 (the ACL harness in CI), #381 (an operator's link at
/opt/unbounded/agentin the install scripts), #382 (uninstall.sh's fallback), #383 (first-boot failures that can't succeed on a retry), #384 (E2E hardening). Found by the harness: #385 (reset-faileddenied on Azure Container Linux). Later: #398 (direct activation can remove the old binary the agent service still runs from), and #399 (an Azure Container Linux VM in the Azure E2E).