Body: <p>I am trying to use the GPU enabled version of the guppy_basecaller on an HPC cluster.  I am requesting a node that has 2 GPUs and am requesting 1 of the two GPUs.  So, I am trying to set the <code>-x cuda:--</code> flag.  What I was thinking this needed was the device number, ie if the allocation gave me a node and had the <code>CUDA_VISIBLE_DEVICES=</code> flag set, I would use <code>-x cuda:0</code> or <code>-x cuda:1</code> depending on which GPU I was assigned.  When I set it to 0 or auto, it seems to work when I have a whole.  When I set it to 1 when I am given a job with that device id flag set, the program errors out.  Am I thinking about the flag wrong?  The documents I have found on this are a bit sparse.</p>
  
  <pre><code>CUDA_VISIBLE_DEVICES=1
  working on run id  8138
  working on subset  0
  
  git er done ...
  ONT Guppy basecalling software version 3.0.3+7e7b7d0
  config file:        /groups/ju_lab/softwares/ont-guppy/data/dna_r9.4.1_450bps_hac.cfg
  model file:         /groups/ju_lab/softwares/ont-guppy/data/template_r9.4.1_450bps_hac.jsn
  input path:         ./
  save path:          ./
  chunk size:         1000
  chunks per runner:  1000
  records per file:   4000
  num basecallers:    4
  gpu device:         cuda:1
  kernel path:
  runners per device: 4
  
  [guppy/error] main: Could not create CUDA device 1: CUDA error at /builds/ofan/ont_core_cpp/ont_core/basecall_nn/nn_caller_cuda.
  cpp:651: CUDA_ERROR_INVALID_DEVICE
  [guppy/warning] main: An error occurred in the basecaller. Aborting.
  <span class="math-container">```</span>
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-11T21:36:06.863Z
Comments.Text: I should have mentioned, I am trying to request 1/2 of a node because as far as I can tell guppy can't use both GPUs.  And probably should have mentioned this is on a PBS/Torque cluster using Centos 7.  And this is my command to call guppy: `guppy_basecaller -i ./ --save_path ./ --flowcell FLO-MIN106 --kit SQK-LSK109 --gpu_runners_per_device 4 -r --min_qscore 7.0 -x cuda:${CUDA_VISIBLE_DEVICES} --disable_pings`
Comments.UserDisplayName: Bob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-11T22:39:27.113Z
Comments.Text: New information.  Requesting a full node with 2 gpus, both `-x cuda:0` and `-x cuda:1` work.  But, setting the device to 1 does not result in the second gpu actually being utilized as measure by watching nvidia-smi gpu activity. So I am not sure what that flag means.  What I really need to do is specify which GPU device to use so that I don't kill other users jobs.  In SLURM with cgroups, this would not be a problem, but on this PBS/Torque cluster, I can get access to both even though I am requesting one.
Comments.UserDisplayName: Bob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-11T22:59:58.623Z
Comments.Text: OK, solved I think.  I believe `-x cuda:--` is looking for the item in the list given by `CUDA_VISIBLE_DEVICES` rather than the actual device id.
Comments.UserDisplayName: Bob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-13T12:04:42.457Z
Comments.Text: Welcome to the site. You can [edit] the question to include the relevant code into the question. Also to mark as solved you can (and are encouraged) to post your solution as an answer (and then mark it as the accepted one). This way others, like me we, can learn from you :D
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T12:30:00.080Z
Comments.Text: Did using `-x auto` not find the device you wanted?
Comments.UserDisplayName: Michael Hall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-21T14:45:30.157Z
Comments.Text: please where did you get the software of guppy_basecaller fro Centos 7? thanks
Comments.UserDisplayName: archengzi
Tags: <minion>
Title: using guppy_basecaller on node with 2 GPUs
ThreadText: # using guppy_basecaller on node with 2 GPUs
  
  (Asked by: username_0 on 2019-05-11)
  
  I am trying to use the GPU enabled version of the guppy\_basecaller on an HPC cluster. I am requesting a node that has 2 GPUs and am requesting 1 of the two GPUs. So, I am trying to set the `-x cuda:--` flag. What I was thinking this needed was the device number, ie if the allocation gave me a node and had the `CUDA_VISIBLE_DEVICES=` flag set, I would use `-x cuda:0` or `-x cuda:1` depending on which GPU I was assigned. When I set it to 0 or auto, it seems to work when I have a whole. When I set it to 1 when I am given a job with that device id flag set, the program errors out. Am I thinking about the flag wrong? The documents I have found on this are a bit sparse.
  
  ```
  CUDA_VISIBLE_DEVICES=1
  working on run id  8138
  working on subset  0
  
  git er done ...
  ONT Guppy basecalling software version 3.0.3+7e7b7d0
  config file:        /groups/ju_lab/softwares/ont-guppy/data/dna_r9.4.1_450bps_hac.cfg
  model file:         /groups/ju_lab/softwares/ont-guppy/data/template_r9.4.1_450bps_hac.jsn
  input path:         ./
  save path:          ./
  chunk size:         1000
  chunks per runner:  1000
  records per file:   4000
  num basecallers:    4
  gpu device:         cuda:1
  kernel path:
  runners per device: 4
  
  [guppy/error] main: Could not create CUDA device 1: CUDA error at /builds/ofan/ont_core_cpp/ont_core/basecall_nn/nn_caller_cuda.
  cpp:651: CUDA_ERROR_INVALID_DEVICE
  [guppy/warning] main: An error occurred in the basecaller. Aborting.
  ```
  
  ```
  
  **Tags:** minion
  
  > **Comments:**
  > **username_0**: I should have mentioned, I am trying to request 1/2 of a node because as far as I can tell guppy can't use both GPUs.  And probably should have mentioned this is on a PBS/Torque cluster using Centos 7.  And this is my command to call guppy: `guppy_basecaller -i ./ --save_path ./ --flowcell FLO-MIN106 --kit SQK-LSK109 --gpu_runners_per_device 4 -r --min_qscore 7.0 -x cuda:${CUDA_VISIBLE_DEVICES} --disable_pings`  
  > **username_0**: New information.  Requesting a full node with 2 gpus, both `-x cuda:0` and `-x cuda:1` work.  But, setting the device to 1 does not result in the second gpu actually being utilized as measure by watching nvidia-smi gpu activity. So I am not sure what that flag means.  What I really need to do is specify which GPU device to use so that I don't kill other users jobs.  In SLURM with cgroups, this would not be a problem, but on this PBS/Torque cluster, I can get access to both even though I am requesting one.  
  > **username_0**: OK, solved I think.  I believe `-x cuda:--` is looking for the item in the list given by `CUDA_VISIBLE_DEVICES` rather than the actual device id.  
  > **username_1**: Welcome to the site. You can [edit] the question to include the relevant code into the question. Also to mark as solved you can (and are encouraged) to post your solution as an answer (and then mark it as the accepted one). This way others, like me we, can learn from you :D  
  > **username_2**: Did using `-x auto` not find the device you wanted?  
  > **username_3**: please where did you get the software of guppy_basecaller fro Centos 7? thanks  
  
  ---
  
  ## 0 answers
  
  