Hi,

Yes they are all summed
You are they are averaged accross GPU only (not within). So for 2 GPU, you will get the sum/2.

I don’t think DDP has an option to disable this averaging. You can multiply the final gradients by the number of devices if you need the sum.