How does gradient averaging work in DistributedDataParallel training?


Every DDP instance will have its own copy of the local model, and DDP will setup post autograd hooks on every parameter (i.e., DDP hooks).
In every forward pass, DDP feeds the input data to its own local model, and returns the local output to the application.
The application uses the local output to compute the local loss, and calls backward on the local loss, which kicks off the local autograd engine to compute gradient for all parameters. When one local gradient becomes ready, that will trigger the corresponding DDP hook. The DDP hook will run allreduce on the given gradients, and write the averaged grads back to the parameter.grad field.
When backward is done, parameter.grad should all be globally averaged gradients. Optimizer can then consume that grad to update parameters.


Is there some way to handle this ignore class imbalance during distributed training?

DDP simply averages (sum and then divide by the number of DDP world size) all local gradients. So, it should work as long as the ignored data do not contribute to the local gradients.