I apologize if this is adequately explained in the documention. If so, please just give me a link…
I have only found
  
    
    
    Does PyTorch average or sum gradients over a minibatch? autograd
  
  
    For example when you retrieve the gradients like so: 
loss = F.nll_loss(output, target)
loss.backward()
for key, value in model.named_parameters():
    mygrad = value.grad

is mygrad the sum of gradients over the minibatch or the average?
  

https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html
neither of which seem explain this.
However, question is:
The gradient in a given portion of the computational graph depends not only on the loss and the parameter values, but also on the input. If this is true, then even though the loss is summed over the minibatch outputs before backpropagation, how does the optimizer calculate the average gradient? Are all batch_size number of activations for each neuron saved during the forward step, or how?
I hope the question makes sense.