Answers: <p>Ok so i finally made it work.</p>
  
  <p>It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-11-08T14:56:34.463Z
Answers: 2019-11-08T14:56:34.463Z
Answers: lifeguru42
Body: <p>I have setup a kubernetes node with a Nvidia Tesla k80 and followed <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="nofollow noreferrer">this tutorial</a> to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.</p>
  
  <p>I have managed to install the nvidia daemonsets and I can now see the following pods: </p>
  
  <pre><code>nvidia-driver-installer-gmvgt
  nvidia-gpu-device-plugin-lmj84
  </code></pre>
  
  <p>The problem is that even while using the recommended image <code>nvidia/cuda:10.0-runtime-ubuntu18.04</code> I still can't find the Nvidia drivers inside my pod:</p>
  
  <pre><code>root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
  bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src
  </code></pre>
  
  <p>But the tutorial mentions:</p>
  
  <blockquote>
    <p>CUDA libraries and debug utilities are made available inside the container at <code>/usr/local/nvidia/lib64</code> and <code>/usr/local/nvidia/bin</code>, respectively.</p>
  </blockquote>
  
  <p>I have also tried to test if CUDA was working through <code>torch.cuda.is_available()</code> but I get False as a return value.</p>
  
  <p>Many thanks in advance for your help.</p>
  
Comments: CC BY-SA 4.0
Comments: 2020-05-09T15:58:17.750Z
Comments: https://stackoverflow.com/q/58766003/2777965
Comments: 030
Tags: <kubernetes><google-cloud-platform><google-compute-engine>
Title: GKE - GPU nvidia - cuda drivers dont work
UserDisplayName: lifeguru42
ThreadText: # GKE - GPU nvidia - cuda drivers dont work
  
  (Asked by: username_0 on 2019-11-08)
  
  I have setup a kubernetes node with a Nvidia Tesla k80 and followed [this tutorial](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus) to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.
  
  I have managed to install the nvidia daemonsets and I can now see the following pods:
  
  ```
  nvidia-driver-installer-gmvgt
  nvidia-gpu-device-plugin-lmj84
  
  ```
  
  The problem is that even while using the recommended image `nvidia/cuda:10.0-runtime-ubuntu18.04` I still can't find the Nvidia drivers inside my pod:
  
  ```
  root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
  bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src
  
  ```
  
  But the tutorial mentions:
  
  > CUDA libraries and debug utilities are made available inside the container at `/usr/local/nvidia/lib64` and `/usr/local/nvidia/bin`, respectively.
  
  I have also tried to test if CUDA was working through `torch.cuda.is_available()` but I get False as a return value.
  
  Many thanks in advance for your help.
  
  **Tags:** kubernetes, google-cloud-platform, google-compute-engine
  
  > **Comments:**
  > **username_1**: https://stackoverflow.com/q/58766003/2777965  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-11-08. Score: 1 | Accepted answer
  
  Ok so i finally made it work.
  
  It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..
  
  
  ---
  
  