Solved by amirhf in post #2 
              
              
                I am assuming you are using a distributed launch. The warning message is self-explanatory. It seems like in each particular process other GPUs are still visible to the process. Ideally, on local_rank X, you want to only GPU X to be visible. There are some workarounds: 

Set CUDA_VISIBLE_DEVICES = in…