I am trying to train a network with “Distributed Data Parallel” on multiple nodes, each having a different public IP address by sshing simultaneously into these nodes using “pdsh” coordination tool as suggested in this tutorial 22.
Specifically, given a local machine with public IP address “Ip0” and 2 remote nodes with Public IP address “Ip1” and  “Ip2” respectively on which training is to be performed remotely from the local machine, how to go about making such a set-up?
Also, how to ensure before running the training script on each remote node that the 2 remote nodes have access to each other?
Thanks in advance.