Dear KFrank
Thank you very much for your detailed answers. I think I am getting close to understanding it, but will make one more question, to which the answer is hopefully ‘yes, that’s it’. If so, I’ll update this thread with a description of my own misunderstanding, so perhaps others making the same mistake can get some extra value from this thread.
Anyway:
I agree that my ‘1/x’ example was poorly chosen. I was simply grasping for a function with a non-constant gradient.
Let’s assume a very simple network:
y=x*w
with the loss being avr(y)
Let’s assume w=2, x=[1,2] (we have a batchSize of 2, input is a scalar)
In the forward pass, we get y=[2,4], L=.5*(2+4)=3
By the chain rule:
dL/dw=dL/dy0*dy0/dw+dL/dy1*dy1/dw
We see that there is a “batchSize” number of terms in this sum. As I understand you, all these factors are stored in the ‘graph’ until .backward() is called, is that correct? I was assuming that there would be averaging or something else going on as each layer was called in the network, but that was incorrect? Since autograd doesn’t know anything about mini batches, does that mean that x=1 and x=2 create two almost equal ‘arms’ in the graph, with a common root at L? If that picture makes sense.
So, the graph contains ALL computations that took place, even if a lot of them are linked to the same variables.
As in, I was expecting the graph to somehow make a rewriting like:
dL/dy0*dy0/dw+dL/dy1*dy1/dw=dL/dy0*(dy0/dw+dy1/dw)
to take advantage of symmetries, but from your ‘neither knows nor cares’ phrasing, I assume that autograd does no such thing, and instead keeps track of everything?
Is there a way to inspect this graph somehow? Or at least see its size? I am supposed to be including some of this in a lecture, and I think some way of visualizing how much is actually being saved in the graph would be instructive.
Regards