Hi,
I need to track the computation and storage of different parts of my network training.
To be on the same page, let’s assume the simple following scenario (biases omitted)

image.jpg932×462 95.1 KB

Questions
Local Gradients

Are the intermediate gradients always stored in memory until they are consumed in the backward pass, being eliminated after that? Or is there an option to not storing but recompute them when needed during the backward pass?
How can I access those values right after they are computed in the forward pass? Is there anything similar to self.layer_i.weight.data.numpy()?

Back-Propagated Error

Is it right the theoretical understanding of back-propagated error?
In the Glorot and Bengio paper 2 (equations 13 and 14 in particular) they make a different analysis for weight gradients and back-propagated gradients and I want to make sure I got that right before analyzing anything.
Is it possible again to access directly these values from the networks attributes? Something like required_grad = True that could allow me to use self.layer_i.weight.data.numpy() for the input instead of for the parameters (weights)?

Thank you