fangwei123456:

Precision-5820-Tower-X-Series:2205682:2205699 [0] init.cc:573 NCCL WARN Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 17000

Precision-5820-Tower-X-Series:2205681:2205698 [0] init.cc:573 NCCL WARN Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 17000



I guess the issue is caused by pushing the model to GPU0, while .to(rank) would be expected.
Could you run this example 9 and check if it’s working on your system?