Solved by pietern in post #2 
              
              
                You have to use DistributedDataParallel with as many processes as the value of world_size. If you specify rank=1 on a single process, it will hang around waiting for the process with rank=0 to start. 
Check out the launch utility for easy launching of multiple processes. You can use it like this: 
p…