soulitzer:

I don’t see why you need to do this with hooks.


Thinking about it more, I don’t necessarily need to do it via hooks. It’s just that I use the hooks to reconstruct the gradients of a given layer for all samples so I thought it might be easily to do the same for the Hessian.



 soulitzer:

Hmm, could you just do torch.autograd.grad(loss, (out1, out2, ... ), create_graph=True) where the outs are the outputs of each of the layers.


When doing this, that would give the gradient of the loss with respect to the output of each layer (for all input samples?), right? If I wanted the Hessian, I assume I could do a similar thing as I stated above as do a nested-loop to get the Hessian for different gradients?,
first_grads = torch.autograd.grad(loss, (out1, out2, ... ), create_graph=True)

hessian_terms = []

for first_grad1 in first_grads:
  for first_grad2 in first_grads:
    hessian_terms.append( torch.autograd.grad(first_grad1, first_grad2, create_graph=False)[0] )

Is this something that’s feasible?