It is the backward of Broadcast that calculates the gradient: https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/_functions.py#L8-L30 17. Gradients are automatically accumulated into .grad attribute in the autograd engine.