On Nodes without an AMD GPU, the k8s-node-labeller DaemonSet Pod init container runs forever.

[dlipovetsky]

The operator creates the k8s-node-labeller DaemonSet.

The Pods of this DaemonSet are scheduled on all nodes. On those nodes where there is no AMDGPU, the init container runs forever:

  initContainers:
  - command:
    - sh
    - -c
    - "echo \"# added by gpu operator \nblacklist amdgpu\" > /host-etc/modprobe.d/blacklist-amdgpu.conf;
      while [ ! -d /host-sys/class/kfd ] || [ ! -d /host-sys/module/amdgpu/drivers/
      ]; do echo \"amdgpu driver is not loaded \"; sleep 2 ;done"

This appends to the log every 2 seconds:

driver-init amdgpu driver is not loaded
driver-init amdgpu driver is not loaded
driver-init amdgpu driver is not loaded
...

Are there some strategies to avoid this? Would the user that deploys the GPU operator need to provide an allowlist or blocklist of Nodes, or can the operator add some selectors that would work in any cluster, or something else? I'll add some ideas as I have them.

Thanks!