NCCL Benchmark Fails: nvidia-smi Not Found in Driver Chroot Environment

[dnugmanov]

Bug Description

The NCCL benchmark cronjob consistently fails with nvidia-smi: command not found when attempting to check for running GPU processes. The issue occurs during the isRunningProcessOnGPU() function call, which tries to execute chroot /run/nvidia/driver nvidia-smi.

Error Details

Error Message:

{"error":"failed to execute nvidia-smi: exit status 127","level":"fatal","msg":"Failed to check running processes on GPU","slurmNode":"worker-0","time":"2025-06-04T14:16:56Z"}

Exit Code: 127 (command not found)

Environment

• Slurm Operator Version: Based on image cr.eu-north1.nebius.cloud/soperator/nccl_benchmark:1.19.0-jammy-slurm24.05.5
• Container Runtime: CRI-O

Investigation Results

root@worker-0:/# ls -la /run/nvidia/driver/
total 0
drwxr-xr-x 3 root root  60 May 27 12:39 .
drwxr-xr-x 6 root root 120 May 27 12:38 ..
drwxr-xr-x 2 root root  40 May 27 12:39 etc

root@worker-0:/# chroot /run/nvidia/driver nvidia-smi
chroot: failed to run command 'nvidia-smi': No such file or directory

root@worker-0:/# which nvidia-smi
/usr/bin/nvidia-smi

Could you please advise on this error?
Is this a misconfiguration on my side, or does the operator have strict placement requirements for the nvidia-smi binary in benchmark pods that I'm not meeting?

[Uburro]

show plz

root@worker-0:~# chroot /run/nvidia/driver  bash
root@worker-0:/# which nvidia-smi
/usr/bin/nvidia-smi

[dnugmanov]

@Uburro

root@worker-0:/# chroot /run/nvidia/driver  bash
chroot: failed to run command 'bash': No such file or directory
root@worker-0:/# which nvidia-smi
/usr/bin/nvidia-smi
root@worker-0:/# ls -la /run/nvidia/driver
total 0
drwxr-xr-x 3 root root  60 May 27 12:39 .
drwxr-xr-x 6 root root 120 May 27 12:38 ..
drwxr-xr-x 2 root root  40 May 27 12:39 etc
root@worker-0:/# tree /run/nvidia/driver
/run/nvidia/driver
`-- etc

1 directory, 0 files
root@worker-0:/#

[Uburro]

@Uburro

root@worker-0:/# chroot /run/nvidia/driver  bash
chroot: failed to run command 'bash': No such file or directory
root@worker-0:/# which nvidia-smi
/usr/bin/nvidia-smi
root@worker-0:/# ls -la /run/nvidia/driver
total 0
drwxr-xr-x 3 root root  60 May 27 12:39 .
drwxr-xr-x 6 root root 120 May 27 12:38 ..
drwxr-xr-x 2 root root  40 May 27 12:39 etc
root@worker-0:/# tree /run/nvidia/driver
/run/nvidia/driver
`-- etc

1 directory, 0 files
root@worker-0:/#

yes, you should put /usr/bin/nvidia-smi inside /run/nvidia/driver ( full path /run/nvidia/driver/usr/bin/nvidia-smi )

[dnugmanov]

I have copied binary but cronjob still throws error

root@worker-0:/run/nvidia/driver/usr/bin# ls -la /run/nvidia/driver/usr/bin/nvidia-smi
-rwxr-xr-x 1 root root 1137440 Jun  4 16:54 /run/nvidia/driver/usr/bin/nvidia-smi
root@worker-0:/run/nvidia/driver/usr/bin# cd /
root@worker-0:/# ls -la /run/nvidia/driver/usr/bin/nvidia-smi
-rwxr-xr-x 1 root root 1137440 Jun  4 16:54 /run/nvidia/driver/usr/bin/nvidia-smi
root@worker-0:/# tree /run/nvidia/driver
/run/nvidia/driver
|-- etc
`-- usr
    `-- bin
        `-- nvidia-smi

3 directories, 1 file
root@worker-0:/# chroot /run/nvidia/driver nvidia-smi
chroot: failed to run command 'nvidia-smi': No such file or directory
root@worker-0:/# chroot /run/nvidia/driver /usr/bin/nvidia-smi
chroot: failed to run command '/usr/bin/nvidia-smi': No such file or directory

btw: How is it supposed to work in your environments? I don't use anything custom including images.

[Uburro]

btw: How is it supposed to work in your environments? I don't use anything custom including images.

VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
kubernetes 1.30
containerd v1.7.27
and gpu operator.

root@worker-0:~# ls -la /run/nvidia/driver/run/nvidia/
total 8
drwxr-xr-x 6 root root  140 May 22 15:53 .
drwxr-xr-x 1 root root 4096 May 22 15:56 ..
drwxr-xr-x 3 root root   60 May 22 15:52 driver
drwxr-xr-x 3 root root   60 May 22 15:56 mps
-rw-r--r-- 1 root root    5 May 22 15:53 nvidia-driver.pid
drwxr-xr-x 2 root root   60 May 22 15:56 toolkit
drwxr-xr-x 2 root root  140 May 22 15:57 validations
root@worker-0:~# chroot /run/nvidia/driver  echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/mpi/gcc/openmpi-4.1.7a1/bin

[dnugmanov]

Thanks, @Uburro

Am i right that root cause here is that host directories are different? As worker pod mounts /run/nvidia from host.

- mountPath: /run/nvidia
  mountPropagation: HostToContainer
  name: nvidia

Do you install drivers via gpu-operator?

[Uburro]

Thanks, @Uburro

Am i right that root cause here is that host directories are different? As worker pod mounts /run/nvidia from host.

- mountPath: /run/nvidia
  mountPropagation: HostToContainer
  name: nvidia

Do you install drivers via gpu-operator?

Yes, you are right. We use this value https://github.com/nebius/nebius-k8s-applications/blob/6f374ce84fc748cb954311a1390e42357b61c3b1/nvidia-gpu-operator/chart/values.yaml#L33

[dnugmanov]

We don't use gpu-operator, we bake all tooling + drivers to host OS image using https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html#ubuntu

WDYT how could we consider both cases and support both from soperator side? To support our option we no need to run chroot.

[Uburro]

We have plans to completely abandon the NCCL benchmark cron jobs in favor of ActiveCheck . For now, it’s still an early alpha and the spec is unstable. You can disable the NCCL benchmark and write your own checks. Without chroot I mean