Hi,


In your image, the intermediate results will be z and a. And the correpsonding gradients for them are dL/dz and dL/da.
Note that in your image, all the boxes correspond to functions. The purple function is what you define in your forward and the red functions are our backward Node.
The Tensor containing values are on the arrows: the purple ones are the intermediate results you have when you write your forward.
The green are what is saved for backward (you don’t see these).
And the red are the gradients flowing back, containing the gradient for each intermediate result.
You can see a Tensor hook as observing one of these red edges.


nn.Module builds on top of the autograd to have a nice neural network library.
So a hook on the nn.Module is trying (and failing at the moment) to get the intermediate gradient for all the inputs/outputs of it’s forward function.
Hooking on a Tensor is using directly the autograd construct. And will give you the gradient computed for that specific Tensor.