If I’m spawning 2 process on 1 machine with 2 GPUs. AFAIK, in each process, there will be a model replica during DDP construction. And the all_reduce synchronization happened during loss.backward(). So I’m wondering which of the 2 process is actually doing this reduction work? I can see that during all_reduce, we need to somehow communicate between the 2 processes. But is there like a ‘major’ process doing more work? Like doing the average of the gradients?


These are the Gloo allreduce algorithms gloo/allreduce.h at c22a5cfba94edf8ea4f53a174d38aa0c629d070f · facebookincubator/gloo · GitHub 6


I am using GLOO currently, I just checked in Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation 2, the GLOO on GPU has limited functionality, just 2 supported functions: broadcast and all_reduce. Are these 2 functions enough for a basic DDP example on GPU to work fine?


Yes, those two functions are enough to implement a DDP algorithm. If you are doing distributed GPU training, it is recommended to use the NCCL backend.
More information on the distributed communication packages Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation.

Also, how could I check if my all_reduce is done on GPU, not on CPU?

For clarification, do you mean that the all_reduce algorithm is run on GPU?

And besides, could I use GLOO backend to launch 2 processes on CPU to do DDP?

Yes, initialize a process group before creating your DDP model.
A tutorial on DDP Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.9.0+cu102 documentation 3.