Actually, after reviewing the output file for the RPC implementation, it looks like each node acted as the master, despite the if/else statement to check for rank 0 as per the RPC model parallelism tutorial.
My code is here: GitHub - ajayp1/Distributed-Torch-testing: test scripts for trying out PyTorch's distributed libraries 10
Separate but related issue: when I run the MNIST example from this tutorial 1, I’m not able to broadcast the MASTER_ADDR and PORT variables to the worker nodes. I managed to at least set the variables on the master node by putting the assignment under if __name__ == "__main__":, but it still doesn’t broadcast to other nodes. Am I supposed to use a certain launcher so that e.g. MPI can communicate between nodes?
My code for this example using just torch.distributed is in the same repo, files have MNIST in the title.