Hi,
We use the autograd to automatically accumulate the gradients inside the original Parameter whichi is on the main GPU.
Yes the gradients are added element-wise. The same as if you were re-using a given Tensor in multiple expressions when computing your loss.