Solved by theevann in post #2 
              
              
                Ok, solved it. 
Since I was a bit lost due to lack of documentation for using pytorch + mpi + gpu, I will give here the steps I followed. 
The main thing I was missing is that I needed a openmpi which is "cuda-aware". 
Main steps to follow:

Install “cuda-aware” openMPI : Need to compile from source
…