k8s 1.36 readiness: clean on removals; pilot user namespaces, watch VolumeGroupSnapshot

[devantler]

🤖 Generated by Claude Code at the maintainer's request (interactive session).

k8s 1.36 ("Haru", GA ~22 Apr 2026) — platform readiness & follow-ups

A review of Kubernetes 1.36 against this platform. Headline: nothing in 1.36 forces a change — the platform is clean against every removal and deprecation. The version bump itself is now possible (Talos 1.13.3 raised the ceiling) but not urgent, and a few GA features are worth adopting opportunistically. This issue tracks the follow-ups.

✅ Clean bill of health — 1.36 removals/deprecations that do not affect us

1.36 change	Why we're unaffected
IPVS kube-proxy deprecated	Cilium runs kubeProxyReplacement: true (k8s/bases/infrastructure/controllers/cilium/helm-release.yaml) — no kube-proxy, no IPVS at all
gitRepo volume removed	Not used anywhere (GitOps via Flux)
Service externalIPs deprecated (removal ~1.43)	Not used — ingress is the Cilium Gateway → Hetzner LB
Legacy AppArmor annotations removed	talos/cluster/apparmor.yaml is a Talos kernel LSM arg, not the removed container.apparmor.security.beta.kubernetes.io/* pod annotation
SELinux volume labeling GA (breaking in 1.37)	Talos uses AppArmor, not SELinux
Endpoints API deprecation	No raw kind: Endpoints; EndpointSlice via Cilium

🔼 The version bump (k8s 1.35.5 → 1.36) — optional, not urgent

• Unlocked: Talos 1.13.3 (already pinned, ksail.prod.yaml) supports k8s 1.31–1.36. The lockstep comment in ksail.prod.yaml makes "bump kubernetesVersion when Talos's range moves" the maintainer's manual step (protected file).
• No forcing function: the Talos #13350 scheduler-config bug (1.36 plugin fields leaking into a 1.35 kube-scheduler config) was a 1.13.2 regression fixed in 1.13.3, which we already pin — so 1.35.5-on-1.13.3 is healthy.
• Recommendation: let 1.36.2/1.36.3 bake before bumping; nothing here is pressing, and we just landed 1.35.5.
• Stale doc to fix (protected ksail.prod.yaml): the comment still reads "Talos v1.12.4 supports Kubernetes 1.30-1.35" even though talos.version is now v1.13.3 (supports 1.31–1.36) — left behind by the Renovate Talos bump (fix(deps): update dependency siderolabs/talos to v1.13.3 #1801). Update when the next prod change touches the file.

🎯 Opportunities (GA features worth adopting)

• User Namespaces → GA — the standout. hostUsers: false maps container UID 0 to an unprivileged host UID (defence in depth, complements restricted PSS + Kubescape/Tetragon + SPIRE mTLS). Beta/on-by-default since 1.30, so it works on 1.35.5 today.
  • Pilot on whoami (stateless canary) — feat(whoami): pilot user namespaces (hostUsers: false) in prod #1808 (draft).
  • Validate in prod: pod starts, id inside the container shows the remapped UID, no events.
  • Extend to homepage (apex, always-on; has read-only configMap mounts — verify those mount cleanly under userns). — live-verified 2026-07-04: hostUsers: false, pods Running, canary Succeeded.
  • Then stateful workloads (Longhorn/CNPG) — these need idmap-volume support; evaluate per-app, last.
  • Generalize once proven — move off per-app overlay patches to a Kyverno +(hostUsers): false anchor rule in k8s/bases/infrastructure/cluster-policies/best-practices/add-security-context.yaml (the repo's established pattern for cross-cutting pod-spec defaults), scoped via namespace/label allow-list to exclude stateful workloads (Longhorn/OpenBao/CNPG/SPIRE). Do not add the blanket rule before per-app validation — it would hit volume-bearing workloads where idmap support is load-bearing.
• PSI Metrics → GA — CPU/memory/IO pressure-stall metrics from the kubelet (needs cgroup v2, which Talos already uses). Useful given our OOM-cascade/rightsizing history, but Coroot's eBPF node-agent already surfaces much of this. Gated on the 1.36 bump. Low priority; wire into Coroot scraping if/when we bump.
• VolumeGroupSnapshot → GA — crash-consistent multi-PVC snapshots; would pair with the Velero CSI/data-mover work (feat(velero): CSI snapshot + data-mover backups for Longhorn PVCs #1790) for CNPG. Triple-gated and not actionable now: Longhorn 1.11.2 does not implement the CSI GROUP_SNAPSHOT capability, our snapshot-controller is 5.0.4 (group snapshots need ~v8.x), and Velero support is young. CNPG already has WAL/barman backups. Watch — re-evaluate when Longhorn ships group-snapshot support.

🚫 Considered and skipped

• Mutating Admission Policy / Declarative Validation → GA — native CEL admission is nice, but Kyverno already covers our mutations plus cosign image verification and generate rules that MutatingAdmissionPolicy can't do. No migration value.

---

Sources

• k8s v1.36 release · sneak peek
• User Namespaces GA · externalIPs deprecation · VolumeGroupSnapshot GA
• Talos #13350 · Longhorn 1.11 CSI snapshots

[devantler]

🤖 Generated by the Daily AI Assistant

User-namespaces chain status (2026-07-02): whoami pilot validated live — hostUsers=false, Running, 0 restarts, no warning events (the id-in-container check needs an exec/debug the autonomous agent isn't permitted; one-liner: kubectl --context admin@prod debug <whoami-pod> -n whoami --image=busybox:1.37 --target=whoami -- cat /proc/self/uid_map — a non-0-origin first field confirms the remap). The homepage extension (#2336, merged 06-30) is applied to the source Deployment but NOT live on homepage-primary — the Flagger canary has been wedged in Failed since 06-27 and never restarts, so the promotion never ran. Tracked with full root-cause evidence and a one-click unwedge in #2387; the 'Extend to homepage' checkbox (and everything after it: stateful workloads, the Kyverno generalization) is blocked on that unwedge + a green canary run.

[devantler]

🤖 Generated by the Daily AI Assistant

Userns pilot status update (live-verified 2026-07-04 ~20:30Z):

• homepage is now a LIVE userns workload: deploy/homepage-primary has hostUsers: false, pods 2/2 Running, canary/homepage Succeeded (the fix(homepage): Flagger canary wedged in Failed since 06-27 — new revisions (#2322, #2336) never restart analysis #2387 canary wedge cleared with today's deploy rollovers — fix(homepage): Flagger canary wedged in Failed since 06-27 — new revisions (#2322, #2336) never restart analysis #2387 closed with evidence). Together with the whoami pilot, the userns pilot AC now has two healthy production workloads.
• The feat(homepage): add Enable Banking bookmark link #2322 bookmark reference is stale: there is no bookmark namespace or deployment on prod anymore — that pilot target no longer exists and can be dropped from the checklist.
• Prevention follow-up carried here (from fix(homepage): Flagger canary wedged in Failed since 06-27 — new revisions (#2322, #2336) never restart analysis #2387): consider a Coroot/Alertmanager alert on flagger_canary_status stuck in Failed, so a wedged canary is noticed within hours instead of days.

Suggested next child: decide the wider hostUsers: false rollout order (candidates: the remaining stateless tenant apps first, infra controllers last) and encode it as a Kyverno audit→enforce ratchet like the C-0017 pattern, rather than per-app one-offs.