Does PyTorch average or sum gradients over a minibatch? autograd
  
  
    For example when you retrieve the gradients like so: 
loss = F.nll_loss(output, target)
loss.backward()
for key, value in model.named_parameters():
    mygrad = value.grad

is mygrad the sum of gradients over the minibatch or the average?
  


My question relates this one.
#Quote Mr SimonW
All autograd does is just to calculate the gradient, it has no notion of batching, and I don’t see how it can have different behavior with different batching mechanism.