Getting nvidia-device-plugin container CrashLoopBackOff | version v0.14.0 | container runtime : containerd

[DineshwarSingh]

Getting nvidia-device-plugin container CrashLoopBackOff error. Using K8-device-plugin version v0.14.0 and container runtime as containerd. Same is working fine with container runtime as dockerd.

Pod ErrorLog:

I0524 08:28:03.907585       1 main.go:256] Retreiving plugins.
W0524 08:28:03.908010       1 factory.go:31] No valid resources detected, creating a null CDI handler
I0524 08:28:03.908084       1 factory.go:107] Detected non-NVML platform: could not load NVML library: libnvidia-ml.so.1: cannot open shared object file: No such file or directory
I0524 08:28:03.908113       1 factory.go:107] Detected non-Tegra platform: /sys/devices/soc0/family file not found
E0524 08:28:03.908121       1 factory.go:115] Incompatible platform detected
E0524 08:28:03.908130       1 factory.go:116] If this is a GPU node, did you configure the NVIDIA Container Toolkit?
E0524 08:28:03.908136       1 factory.go:117] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
E0524 08:28:03.908142       1 factory.go:118] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
E0524 08:28:03.908149       1 factory.go:119] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes
E0524 08:28:03.915664       1 main.go:123] error starting plugins: error creating plugin manager: unable to create plugin manager: platform detection failed

nvidia-smi output:

sh-4.2$ nvidia-smi
Wed May 24 08:57:00 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                        On | 00000000:00:1E.0 Off |                    0 |
| N/A   25C    P8                9W /  70W|      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

[elezar]

Hi @DineshwarSingh, could you comment on how the device plugin is configured / installed. Note that the device plugin also requires that the NVIDIA Container Toolkit be installed on the system and be configured as a runtime class in Containerd. Have you installed the toolkit and configured Containerd to use it as a runtime?

[DineshwarSingh]

Hi @elezar,
Thanks for your response !
we are using amazon linux 2 and NVIDIA Container Toolkit is installed. Please see below details:
sh-4.2$ nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.13.1

Regards,
Dinesh

[elezar]

@DineshwarSingh how is the Device Plugin deployed?

What are the contents of your Containerd config.toml file?

[DineshwarSingh]

@elezar device plugin is deployed using helm version v0.14.0.
/etc/containerd/config.toml content is as below:
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"

[elezar]

Thanks for the information. I assume that containerd has been restared since the config has been changed?

Could you also provide the contents of your /etc/nvidia-container-runtime/config.toml file?

[DineshwarSingh]

Hi @elezar,
please find below contents of your /etc/nvidia-container-runtime/config.toml file:

disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
#accept-nvidia-visible-devices-as-volume-mounts = false

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
#user = "root:video"
ldconfig = "@/sbin/ldconfig"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"

#Specify the runtimes to consider. This list is processed in order and the PATH
#searched for matching executables unless the entry is an absolute path.
runtimes = [
"docker-runc",
"runc",
]

mode = "auto"

[nvidia-container-runtime.modes.csv]

mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

=========================================================================
Thanks,
Dinesh

[DineshwarSingh]

Hi @elezar,
Do you have any suggestion to fix this issue? Any possible help is highly appreciated !

Thanks,
Dinesh

[zachfi]

I am also seeing this in my environment. My config and output look the same as above. Happy to provide more details.

[zachfi]

The following packages are installed.

[root@k4 ~]# pacman -Q | grep nvidia
libnvidia-container 1.13.1-1
libnvidia-container-tools 1.13.1-1
nvidia 530.41.03-15
nvidia-container-runtime 3.13.1-1
nvidia-container-toolkit 1.13.1-1
nvidia-utils 530.41.03-1
opencl-nvidia 530.41.03-1

Contianerd has been verified with the following.

[root@k4 ~]# ctr run --rm --gpus 0 -t docker.io/nvidia/cuda:12.1.1-runtime-ubuntu20.04 n nvidia-smi
Fri Jun  2 17:01:46 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 970          Off| 00000000:01:00.0 Off |                  N/A |
| 11%   43C    P0               35W / 170W|      0MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

The plugin was installed with the following manifest.

# https://github.com/NVIDIA/k8s-device-plugin#enabling-gpu-support-in-kubernetes
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nvidia-device-plugin
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nvidia-device-plugin
      annotations: {}
    spec:
      priorityClassName: system-node-critical
      securityContext: {}
      containers:
        - image: nvcr.io/nvidia/k8s-device-plugin:v0.14.0
          imagePullPolicy: IfNotPresent
          name: nvidia-device-plugin-ctr
          env:
            - name: NVIDIA_MIG_MONITOR_DEVICES
              value: all
          securityContext:
            capabilities:
              add:
                - SYS_ADMIN
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
      tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoSchedule
          key: nvidia.com/gpu
          operator: Exists
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia
---
apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark
  namespace: default
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:nbody
      args: ['nbody', '-gpu', '-benchmark']
      resources:
        limits:
          nvidia.com/gpu: 1
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: all
        - name: NVIDIA_DRIVER_CAPABILITIES
          value: all

Using the following information to get started. https://docs.k3s.io/advanced#nvidia-container-runtime-support

Let me know what other details are helpful.

[elezar]

Note that using the CTR comman: ctr run --rm --gpus 0 -t docker.io/nvidia/cuda:12.1.1-runtime-ubuntu20.04 n nvidia-smi is not sufficient in this case since this uses a different mechanism for injecting the devices (by adding the nvidia-container-runtime-hook directly) from the Device plugin.

Note that in an earlier comment you mention /etc/containerd/config.toml referencing the NVIDIA Container runtime, but the k3s documentation mentions:

Confirm that the nvidia container runtime has been found by k3s: grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

Could you confirm that the nvidia runtime is defined there too? If it is (but is not the default runtime), specifying the nvidia runtime class name for the device plugin pod too may address this.

[zachfi]

Good eyes. It looks like it also is detected here.

# grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

Here is the full config, but how do I know if it is the non-default?

version = 2
[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"
[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["https://registry.default.svc.cluster.local:5000"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.default.svc.cluster.local:5000".tls]
  ca_file = "/etc/ssl/ca.pem"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
  runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

[zachfi]

Also note that the GPU benchmark pod also fails to schedule with the runtimeClassName set due to "Insufficient nvidia.com/gpu".

[elezar]

@zachfi since there is no plugins."io.containerd.grpc.v1.cri".containerd.default_runtime_name = "nvidia" entry, the nvidia runtime is not the default runtime. As such you would also need to launch the Device Plugin specifying runtimeClassname: nvidia. This ensures that the containers of the device plugin are started using the nvidia-container-runtime, injecting the required devices. (Please also confirm that NVIDIA_VISIBLE_DEVICES = all in this container too).

The issue that the benchmark container is seeing is because tha device plugin is not reporting any nvidia.com/gpu resources to the Kubelet and as such the pod cannot be scheduled.

[zachfi]

NVIDIA_VISIBLE_DEVICES was set in the benchmark pod, but not in the daemonset pods. I've added this and it looks to fail in the same way.

❯ k -n kube-system logs nvidia-device-plugin-qxs5n
I0608 13:59:52.537232       1 main.go:154] Starting FS watcher.
I0608 13:59:52.537319       1 main.go:161] Starting OS watcher.
I0608 13:59:52.537643       1 main.go:176] Starting Plugins.
I0608 13:59:52.537659       1 main.go:234] Loading configuration.
I0608 13:59:52.537778       1 main.go:242] Updating config with default resource matching patterns.
I0608 13:59:52.537990       1 main.go:253] 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "none",
    "failOnInitError": true,
    "nvidiaDriverRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "plugin": {
      "passDeviceSpecs": false,
      "deviceListStrategy": [
        "envvar"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/bin/nvidia-ctk",
      "containerDriverRoot": "/driver-root"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {}
  }
}
I0608 13:59:52.538003       1 main.go:256] Retreiving plugins.
W0608 13:59:52.538273       1 factory.go:31] No valid resources detected, creating a null CDI handler
I0608 13:59:52.538319       1 factory.go:107] Detected non-NVML platform: could not load NVML library: libnvidia-ml.so.1: cannot open shared object file: No such file or directory
I0608 13:59:52.538344       1 factory.go:107] Detected non-Tegra platform: /sys/devices/soc0/family file not found
E0608 13:59:52.538352       1 factory.go:115] Incompatible platform detected
E0608 13:59:52.538361       1 factory.go:116] If this is a GPU node, did you configure the NVIDIA Container Toolkit?
E0608 13:59:52.538368       1 factory.go:117] You can check the prerequisites at: https://github.com/NVIDIA/k8s-device-plugin#prerequisites
E0608 13:59:52.538373       1 factory.go:118] You can learn how to set the runtime at: https://github.com/NVIDIA/k8s-device-plugin#quick-start
E0608 13:59:52.538379       1 factory.go:119] If this is not a GPU node, you should set up a toleration or nodeSelector to only deploy this plugin on GPU nodes
E0608 13:59:52.538500       1 main.go:123] error starting plugins: error creating plugin manager: unable to create plugin manager: platform detection failed

[elezar]

Are you ALSO specifying the nvidia runtime class for the device plugin containers?