Solved by William_Ravenscroft in post #3 
              
              
                To follow up, I think I actually had 2 issues firstly I had to set 
export NCCL_SOCKET_IFNAME=<VALUE>
export NCCL_IB_DISABLE=1

Replacing  with your relevant interface  - use the ifconfig to find it. And I think my second issue was using a dataloader with multiple workers but I hadn’t allocated enou…