I’m slightly confused as to what param.grad is calculating mathematically?
Here, it says “Computes and returns the sum of gradients of outputs with respect to the inputs.”
https://pytorch.org/docs/stable/autograd.html
Can someone help me interpret this?
Suppose I’m looking at the weight of one neuron in a fully connected hidden layer denoted as w. Suppose my batch size is 64.
Is param.grad for the w the partial derivative dL/dw calculated for each of the 64 raw inputs, and then summed?