Hi @yueyilia, I tried your code locally (python 3.8.5 and nightly build of pytorch which is 1.10 now), it worked for me and is able to get past init_rpc. I think the issue may be with how your cluster of machines is configured where they cannot establish connections with each other, but I’m not exactly sure.



 yueyilia:

I plan to use 100 servers to build thousands of environments. I don’t know how to assign a unique rank without sockets. Is there any usage that I don’t know?


You can modify your script to use torchleastic via torch.distributed.run (Elastic Launch) — PyTorch master documentation. This essentially performs the work you are doing by having all nodes rendezvous then assign ranks to each process, with added fault tolerance and elasticity if you choose to enable it. You need to modify your script so that it acts as a single process, then run the command:
python -m torch.distributed.run
    --nnodes=$NUM_NODES
    --nproc_per_node=$NUM_TRAINERS
    --rdzv_id=$JOB_ID
    --rdzv_backend=c10d
    --rdzv_endpoint=$HOST_NODE_ADDR
    YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)

The rank is automatically assigned with torchelastic. Each process will be able to read it’s own rank as an environment variable (e.g. rank = os.environ["RANK"]). Here are the list of the environment variables that get be passed into your script: torch.distributed.run (Elastic Launch) — PyTorch master documentation 2