The NCCL error you post doesn’t convey any information that can help unfortunately. Take a look at https://pytorch.org/docs/stable/distributed.html#other-nccl-environment-variables 301 for some environment variables you can set that may help you in debugging this issue.