Below is copied from https://github.com/pytorch/pytorch/issues/38702 21, as we closed that issue and moved the discussion here.

The strange thing is that none of the 4 nodes’s ip is 10.38.10.112<21724>. I don’t know why it will try to connect the ip and the port.

Could you please check immediately before  init_process_group  in the code to confirm that  MASTER_ADDR ,  MASTER_PORT , and  NCCL_SOCKET_IFNAME  are configured properly? Sometimes these can be different from what you set in command line especially when you are using notebook. You can do so by calling os.getenv().
Sharing the code will also be helpful. If the code is confidential, we can start by sharing how you invoke  init_process_group .