Just to make sure on how you initialize process group, do you set
MASTER_ADDR to be address of your node 1 on both nodes?
(so node 2 needs to know address of node 1 before you launch it)
https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization 2