Hii @ZiyiZhu, I think the code would run, and it’s like using multiple NCCL allreduce in a HogWild manner, but I am not confident on the correctness on DDP in this use case. The allreduce is not an atomic operation, it needs to go through the ring (say we are using ring allreduce here) twice to collect and propagate the values. So, it is possible that, after allreduce, model parameters on different processes in the same group are no longer in sync. This breaks DDP’s assumption, and this gap could get larger over more iterations. You might need some extra code to re-sync DDP model parameters every n iterations.