albanD:

For " the gradient information w.grad is treated as a number if the computation graph and not as a variable that one can differentiate" you want to use w.grad.detach() for further computations to achieve that.


After some thought I think I am confused about your suggestion because I was thinking how gradients work mathematically and they are based on partial derivatives. Even if w.grad.detach() is called, when we do a partial derivative with respect to an independent variable (i.e. a leaf node in the computation graph) all other variables are constants. Thus, wether the gradient variable (i.e. w.grad) collects gradients or not I would have expected is irrelevant.
But your comment seems to suggest otherwise. Why is that the case? What part of my reasoning is wrong?