title: how to know how many GPUs are used in pytorch?
body: <p>The bash file I used to launch the training looks like this:</p>
  <pre class="lang-sh prettyprint-override"><code>CUDA_VISIBLE_DEVICES=3,4 python -m torch.distributed.launch \
  --nproc_per_node=2  train.py \
  --batch_size 6 \
  --other_args
  </code></pre>
  <p>I found that the batch size of tensors in each GPU is acctually <code>batch_size / num_of_gpu</code> = <code>6/2</code> = 3.</p>
  <p>When I initialize my network, I need to know <em>the batch size in each GPU</em>.
  (Ps. in this phase, I can't use <code>input_tensor.shape</code> to get the size of batch-dimension, since there are no data fed in jet.)</p>
  <p>Somehow I could not find where does the pytorch store the parameter <code>--nproc_per_node</code>.
  So how could I know how many GPUs are used, without passing it manually as <code>--other_args</code>?</p>