To follow up, I think I actually had 2 issues firstly I had to set
export NCCL_SOCKET_IFNAME=<VALUE>
export NCCL_IB_DISABLE=1

Replacing  with your relevant interface  - use the ifconfig to find it. And I think my second issue was using a dataloader with multiple workers but I hadn’t allocated enough processes to the job in my job submission.