Solved by ebarsoum in post #3 
              
              
                Setting CUDA_VISIBLE_DEVICES=1 mean your script will only see one GPU which is GPU1. However, inside your script it will be cuda:0 and not cuda:1. Because it only see one GPU and its index start at 0. 
For example if you do: CUDA_VISIBLE_DEVICES=2,4,5, your script will see 3 GPUs with index 0, 1 and…