Precision-7920-Tower:281611:281611 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
Precision-7920-Tower:281611:281611 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
Precision-7920-Tower:281611:281611 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda11.0


MS-7B23:8658:8658 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0>
MS-7B23:8658:8658 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
MS-7B23:8658:8658 [0] NCCL INFO NET/IB : No device found.
MS-7B23:8658:8658 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0>
MS-7B23:8658:8664 [0] NCCL INFO Setting affinity for GPU 0 to 3f
MS-7B23:8658:8664 [0] NCCL INFO Call to connect returned Connection refused, retrying
MS-7B23:8658:8664 [0] NCCL INFO Call to connect returned Connection refused, retrying

It looks like NCCL is having a problem establishing a connection. Can you verify that your interfaces are correct. The Common environment variables section provides some information Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation 48.