bing:

RuntimeError: NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:514, invalid usage, NCCL version 2.6.3


Are you using system NCCL? As of today, the master is still on NCCL 2.4.8 IIUC. Let’s first get DDP run correctly and then deal with system NCCL. Can you try changing backend='nccl' to backend='gloo'?