Hi,
To concretely answer your question around communication, basically nodes need some way to discover each other, whether that is through a shared-filesystem approach, or through a main IP address that every node can talk to.
Here are some helpful tutorials around PyTorch distributed and DDP:
PyTorch DDP tutorial: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html 10
PyTorch distributed overview: https://pytorch.org/tutorials/beginner/dist_overview.html 3