According to “Environment variable initialization” section in PyTorch Distributed tutorial 1, rank 0 node will be used to set up all connections, and both MASTER_PORT and MASTER_ADDR must be free port and address of rank 0 node. Therefore, you should always use the same values for all the nodes in the same process group.