Typically this indicates an error in the NCCL library itself (not at the PyTorch layer), and as a result we don’t have much visibility into the cause of this error, unfortunately. Is this error consistent, or does the training work if you re-run? Are you using 3 nodes with 8 gpus each?