For example when you retrieve the gradients like so:
loss = F.nll_loss(output, target)
loss.backward()
for key, value in model.named_parameters():
    mygrad = value.grad

is mygrad the sum of gradients over the minibatch or the average?