PoC: offload DP/DRA management to KMM

[abyrne55]

This PR replaces the operator's direct DP/DRA DaemonSet management with a single KMM (Kernel Module Management) Module CR. Two controllers collapse into one, and the operator no longer needs to build DaemonSets, manage RBAC, handle pod readiness, or construct OpenShift SCCs at runtime.

	Before	After
Controllers	2 (DevicePluginReconciler, DRAReconciler)	1 (KMMReconciler)
K8s objects managed at runtime	DaemonSets, ServiceAccounts, ClusterRoles, ClusterRoleBindings, DeviceClasses, AdmissionPolicies, SCCs	1 Module CR
Embedded YAML + loaders	9 files	0

Existing ClusterPolicy specs work unchanged — KMM is used transparently under the hood.

Beyond simplifying what we already have, KMM also brings capabilities we'd otherwise need to build: OOT kernel module loading (see new CRD field below), kernel-version-aware rollouts, ordered upgrades, pre-flight validation, and build/sign integration. Ordered upgrades and pre-flight validation apply immediately for OOT driver modules; build/sign integration is available when needed.

How it works

A new KMMReconciler sub-controller replaces DevicePluginReconciler and DRAReconciler in the reconcile loop. It uses controllerutil.CreateOrPatch to manage the Module CR idempotently, with SetControllerReference so the Module is garbage-collected when the ClusterPolicy is deleted.

DRA RBAC (ServiceAccount, ClusterRole, ClusterRoleBinding), OpenShift SCCs, and ValidatingAdmissionPolicy resources have been moved out of operator-managed code and into deployment tooling (Helm templates and kustomize overlays).

KMM availability is detected at startup via API group discovery (kmm.sigs.x-k8s.io), same pattern as DRAEnable. If the CRD isn't installed, the operator starts normally and reports an error in ClusterPolicy status.

New CRD field

spec:
  kernelModule:            # optional; omit for in-tree driver
    moduleName: "xe"       # kernel module to load
    image: "registry.example.com/xe-driver:1.0"
    version: "1.0.0"

When kernelModule is set, KMM loads the specified OOT driver module on each node. When omitted, the in-tree kernel driver is used.

Known limitations

• DP mode untested. The device-plugin path through KMM (mod.Spec.DevicePlugin) is implemented and unit-tested but hasn't been validated on a cluster.
• OOT driver mode untested. Requires building a driver container image we don't have yet.

Test results

Tested on OCP 4.22.2 (SNO) with two Intel Arc Pro B70 GPUs. KMM and NFD pre-installed.

• DRA mode with in-tree driver: Module CR created, KMM spun up DRA DaemonSet, DeviceClasses created, GPU devices visible in ResourceSlices, test pod with DRA ResourceClaim scheduled and saw /dev/dri devices
• XPU Manager runs alongside KMM-managed DRA
• ClusterPolicy deletion cleans up Module CR (DaemonSet garbage-collected by KMM)
• make test passes (102/102 specs including 8 KMM controller tests)

This PR was written in part with the assistance of generative AI.

[tkatila]

Hi @abyrne55 ! Thanks for the PoC!

• K8s object creation (serviceaccount, SCCs, etc.) is moved from operator runtime to operator deployment time, what is the reasoning for this move?
  • While it can have security benefits (less access rights in the operator), it limits what the operator can do.
  • It also forces to use the "is openshift" variable in the chart.
  • https://operatorframework.io/operator-capabilities/ would suggest that operator should be capable of creating objects, but I'm not sure if there is general consensus on it.
• Was there a specific KMM version required for this to work? I just applied KMM via the kubectl oneliner from the repo. Seems to be "latest" image. I'm getting this error with DRA:

E0722 06:33:59.300935       1 clusterpolicy_controller.go:234] Return sub-controller errorfailed to reconcile KMM Module gpu-policy-gpu for ClusterPolicy gpu-policy: admission webhook "vmodule.kb.io" denied the request: failed to validate DRA volumes: spec.dra.volumes[0]: hostPath "/etc/cdi" is not allowed; only /dev, /sys, /var, /opt and /run paths are permitted

• And this with DP (in KMM):

E0722 06:31:42.060239       1 controller.go:347] "Reconciler error" err="could handle device plugin: DaemonSet.apps \"g
pu-policy-gpu-device-plugin-l9t68\" is invalid: spec.template.spec.containers[0].volumeMounts[3].mountPath: Invalid value: \"/var/lib/kubelet/device-plugins\": must be unique" logger="kmm" controller="DevicePluginReconciler" controllerGroup="kmm.sigs.x-k8s.io" controllerKind="Module" Module="intel-gpu-base-operator/gpu-policy-gpu" namespace="intel-gpu-base-operator" name="gpu-policy-gpu" reconcileID="57b85f12-1653-490f-b814-05f0da238e78"

• I see KMM has received a helm chart (Adding helm deployment kubernetes-sigs/kernel-module-management#1309). Is there a timeline for the first release with it?

I'm still thinking if I'd like to have DP/DRA deployment also in the base-operator side. For anyone not wanting to deploy KMM to their cluster. Both methods could use the same building blocks so there wouldn't be too much overhead. More testing though.