Hi Miturian!



 miturian:

how does the optimizer calculate the average gradient?


It depends entirely on how the “loss” on which you call loss.backward()
is calculated.
Autograd simply computes the gradient of on whatever scalar you called
.backward().
Most pytorch loss functions calculate the average across the minibatch
of the per-sample losses, but they often given you the option to
compute the 'sum' or apply sample weightings.
You could, for example, have a minibatch of ten samples, but only
compute the loss for the first sample, ignoring the other nine.  If you
did that, you would only get the gradient of that first sample.
And, just to be clear, the optimizer does not calculate the gradients,
averaged or otherwise.  By the time you call opt.step() the gradients
have already been calculated (when you called loss.backward()).
Best.
K. Frank