@pietern you mentioned that gradient accumulation only occurs in the source model, but according to https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py 11 “In the forward
pass, the module is replicated on each device, and each replica handles a
portion of the input. During the backwards pass, gradients from each replica
are summed into the original module.” this implies that gradients are accumulated in the leaf nodes of each of the replicas.