Compute and report gradient norm of a hidden variable
Thanks a lot! EDIT: reformulated my question.
dima
Thank you. It works for parameters. As far as I understand, the gradient for non-leaf nodes is deallocated when it is not needed.
dima
Veril, unfortunately not: <SCODE>    x[0].requires_grad = True
RuntimeError: you can only change requires_grad flags of leaf variables.
<ECODE>
dima
apaszke
It’s not deallocated, it just hasn’t been allocated. We’ve changed the strategy for allocating gradients in the recent release.
apaszke
All optimizers I know about require gradients wrt parameters which are always leaf nodes. So, it shouldn’t be a problem for implementing an optimizer. In my case I need those gradients only for debugging purposes. For those are interested, I ended up doing like this: <SCODE>class LazyContainer(object):
    def __call__(self, g):
        self.grad_norm = g.norm(2)
container = LazyContainer()
h.register_hook(container)
# later
print(container.grad_norm)
<ECODE>
dima
you can also do like this: <SCODE>A = Variable(torch.ones(2), requires_grad = True)
B = A*2 
B.retain_grad()
C = B.norm()
C.backward()
print B.grad
<ECODE>
xylcbd