right, DDP averages gradients in default, as it assumes batch size is the same for each rank. If the batch size is not the same for each rank, or the loss function does not expect averaged gradients, you can call ’ register_comm_hook()’ on the DDP model, define your own communication strategy (sum, or other ops)