title: docker stack deploy with GPU , but can't find nvidia devices
body: <h1>docker stack deploy with GPU , but can't find nvidia devices</h1>
  
  <h2>description:</h2>
  
  <p>When I use <code>docker-compose up</code> start the program, the code works well! But when I use <code>docker stack deploy -c docker-compose.yml test</code> to start the program, it can't find visible nvidia devices. My docker-compose.yml and error logs are shown as below.</p>
  
  <p>I'am very confused about that why I have same configures, using different lanching way of <code>docker-compose up</code> and <code>docker stack deploy -c docker-compose.yml test</code>, the first working well, but the second not. Is it currently not perfect for docker swarm supporting for GPUs, or there are other ways I have not found?</p>
  
  <h2>environmental configuration</h2>
  
  <pre><code>docker version: 18.06.0-ce
  NVIDIA Docker: 1.0.1
  Ubuntu: 16:04
  </code></pre>
  
  <h3>/etc/docker/daemon.json</h3>
  
  <p>Of course, I modify the file /etc/docker/daemon.json, changed the runtime type. And restart it.</p>
  
  <pre><code>{
    "default-runtime": "nvidia",
    "runtimes": {
      "nvidia": {
        "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
      }
    }
  }
  </code></pre>
  
  <pre><code>sudo systemctl daemon-reload
  sudo systemctl start docker
  </code></pre>
  
  <h3>docker-compose.yml configure file</h3>
  
  <pre><code>version: "3"
  
  volumes:
    nvidia_driver_430.14:
      external: true
  
  services:
    tts-server:
      build:
        context: ./
        dockerfile: ./docker/tts_server/Dockerfile
      deploy:
        replicas: 1
      image: tts-system/tts-server-gpu
      environment:
        NVIDIA_VISIBLE_DEVICES: 0
      devices:
        - /dev/nvidia0
        - /dev/nvidiactl
        - /dev/nvidia-uvm
      volumes:
        - ./models:/tts_system/models:ro
        - ./config:/tts_system/config:ro
        - nvidia_driver_430.14:/usr/local/nvidia:ro
      networks:
        - overlay
      ports:
        - "9091:9090"
  </code></pre>
  
  <h3>error logs of program</h3>
  
  <pre><code>2019-07-02 07:50:24.805114: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2499885000 Hz
  2019-07-02 07:50:24.808418: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4112870 executing computations on platform Host. Devices:
  2019-07-02 07:50:24.808457: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): &lt;undefined&gt;, &lt;undefined&gt;
  2019-07-02 07:50:24.811640: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
  2019-07-02 07:50:24.811684: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:155] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
  E0702 07:50:24.811846     1 decoder.cc:80] Filed to create session: Invalid argument: 'visible_device_list' listed an invalid GPU id '0' but visible device count is -1
  </code></pre>
  
  <p>This problem has been bothering me for a long time, thanks very much.</p>