Hi, I just started to learn how to do Distributed training in pytorch. And learnt from the basic tutorials from here: Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.9.0+cu102 documentation 4 and I also read the DDP paper. But I still have some questions here.

If I’m spawning 2 process on 1 machine with 2 GPUs. AFAIK, in each process, there will be a model replica during DDP construction. And the all_reduce synchronization happened during loss.backward(). So I’m wondering which of the 2 process is actually doing this reduction work? I can see that during all_reduce, we need to somehow communicate between the 2 processes. But is there like a ‘major’ process doing more work? Like doing the average of the gradients?
I am using GLOO currently, I just checked in Distributed communication package - torch.distributed — PyTorch 1.9.0 documentation 1, the GLOO on GPU has limited functionality, just 2 supported functions: broadcast and all_reduce. Are these 2 functions enough for a basic DDP example on GPU to work fine?
Also, how could I check if my all_reduce is done on GPU, not on CPU? And besides, could I use GLOO backend to launch 2 processes on CPU to do DDP?
Much appreciated, I might interpret some materials wrong, please help me out.

Edit:
I know I can use nvidia-smi to check my GPU usage to see if I have both GPU working. But the thing is, I have 2 different machines, one have 3 GPU, the other have 2 GPUs. I have set up to use only 2 processes in my simple DDP script. On the first machine, when running the script, I see that GPU memory usage is around 1.5G and 800M for the 2 GPUs by checking nvidia-smi. But on the second machine, with the same script, I can only see 42Mb GPU memory usage increments on both GPU cards. So that’s why I’m wondering if they actually run on GPUs, or loaded the work to CPUs internally when GPUs is somehow not configured correctly when I build pytorch from source.