RuntimeError: Connection timed out in Single node Multi GPU training
I am trying to pre-train DistilBERT with single node multigpu as given here https://github.com/huggingface/transformers/tree/master/examples/distillation.

I have set the IP address of my gcp instance and port number. I am getting this error. Any solutions?

> File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 400, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Connection timed out
By any chance, did you open the port number you're using?<|||||>Yes, it is established. What is the master_addr and master_port when submitting a job with single node and multigpu config in GCP?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi, @kamalravi were you able to solve this issue?? I am facing the same issue right now where my `init_process_group` freezes and doesn't move forward for training. Could you take a look at this and suggest any relevant solutions if you have found them??

Thanks!

https://github.com/pytorch/pytorch/issues/53395#issuecomment-954103393