Solved by rvarm1 in post #2 
              
              
                Hi, 
To concretely answer your question around communication, basically nodes need some way to discover each other, whether that is through a shared-filesystem approach, or through a main IP address that every node can talk to. 
Here are some helpful tutorials around PyTorch distributed and DDP: 
Py…