@Scout I am seeing that my dagster pod doesn't have

nvidia-smi

for using my nvidia GPUs that I've allocated to it in k8s. Is there any way that I can fix this to add

nvidia-smi

to the pod so I can view the GPU utilization within the pod?

To ensure that your Dagster pod can utilize NVIDIA GPUs and have access to

nvidia-smi

, you need to make sure that the Kubernetes nodes have the NVIDIA drivers and the NVIDIA Container Toolkit installed. This setup allows the Kubernetes cluster to schedule GPU workloads and provides the necessary tooling within the pods. However, the specific steps to add

nvidia-smi

to a pod in Kubernetes are not covered in the documentation provided. Typically, this involves using a container image that includes the NVIDIA drivers and tools, or installing them as part of the pod's startup process. You would also need to ensure that your Kubernetes cluster is configured to expose GPUs to pods using resource limits. If you have already allocated NVIDIA GPUs to your pod using Kubernetes resource limits and still do not have access to

nvidia-smi

, you may need to: 1. Use a container image that includes the NVIDIA CUDA toolkit, which contains

nvidia-smi

. 2. Ensure that the NVIDIA device plugin for Kubernetes is properly installed and running on your cluster. This plugin is responsible for advertising NVIDIA GPU resources to the Kubernetes scheduler. For more detailed guidance on how to set up and troubleshoot NVIDIA GPU utilization in Kubernetes, you may need to refer to the Kubernetes documentation, NVIDIA's documentation, or seek assistance from the community or support channels that specialize in Kubernetes and NVIDIA GPU integration.