Hey!



 soulitzer:

If you want to compute second-order derivatives, I think you mean to find the derivative of your grad_output wrt to another output (not grad_output), because that grad_output would not be part of this grad_output’s graph.


Yes, that’s one way of saying it. But essentially compute the gradient of the loss with respect 2 grad_output of the i-th and j-th layer. Which might make more sense as the derivative of the grad_output with respect to another grad_output (in terms of calling autograd)



 soulitzer:

Have you tried registering a full backward hook on a module in which your hook function calls autograd.grad(grad_wrt_this_output, (other_output,))?


I guess a further would be how exactly could I get these derivatives efficiently? My current approach of doing it, is having this within an Optimizer (as my Optimizer requires the use of register_full_backward_hook).
I store all these backward hooks within a defaultdict of torch.optim.Optimizer (the state attribute to be preceise). Would it be as simple as iterating through that defaultdict within a nested-loop as just calling torch.autograd.grad on all pairs? (Like you’ve stated!)
Although, my question is how exactly could these two tensors be connected if no graph is created to represent a derivative of it? (In a similar way to compute second derivatives you need to enable create_graph=True in torch.autograd.grad, for example)