Thank you for your answer,


We actually build the graph under the hook. The only place where you can see it is from the  .grad_fn  field. The  .grad  attribute exists for every Tensor, it is just None by default.




We accumulate in the .grad fields of all the leaf Tensors (t.is_leaf()) that require gradients yes. (Or the ones on which you called retain_grad() that can be used to get the .grad field populated for non-leaf Tensors).


So as I understand it, in every .forward() pass, we build the graph under the hook with the .grad_fn that connects every variable to the function and variables that creates it (this is done by the autograd).
When we do .backward(), we are calculating the .grad attribute for every leaf variables, from the .grad_fn attribute.
But, how we calculate the gradients for non-leaf variables? If we are not doing so, how does the weights for those variables change by the optimizer.step()?


In no_grad mode, no graph is actually built, new Tensors will have requires_grad=False and grad_fn=None.


So, in no_grad mode, .backward() and optimizer.step() will have no effect?
Thank you,