There is a difference, between the global GPU index (the GPU which is available) and the local one PyTorch uses. If your GPU cluster is well configured it should set the environment Variable CUDA_VISIBLE_DEVICES to keep track of which GPU is available.
The PyTorch index must be given w.r.t. the available GPUs.
For Example, if GPUs 0 and 3 would be available, the environment variable CUDA_VISIBLE_DEVICES would be set to "0,3". If you now specify a device with "cuda:0" you take the first device out of that list (GPU0), with "cuda:1"you get the second device out of that list, which should be GPU 3 in a global context.
After this short explanation: If this environment variable is set, you should be able to always use "cuda:0" if you don’t overwrite CUDA_VISIBLE_DEVICES manually.