Thanks for posting @Zhe_Zhe. Can you explain more on your set up environment so that we can help more? Is this only happening in NCCL, what nccl version are you at?
Also, for good measure, you can try setting NCCL_DEBUG=INFO (also see Distributed communication package - torch.distributed — PyTorch 1.10.1 documentation 1) and check if you see anything out of the ordinary (perhaps some warnings or otherwise messages that indicate a problem with the topology).
There’s another github issue that have similar problem, you can check there as well to see if it helps Model parallel with DDP get `Socket Timeout` error when using NCCL, while GLOO works fine · Issue #25767 · pytorch/pytorch · GitHub 2