Yes, that makes sense.
Is there a way to somehow modify the way backprop works in PyTorch so that it computes the second derivative instead of the first? I know it might be a lot of work, but I feel it’s worth it since quite a few methods use diagonal hessian for pruning, e.g. optimal brain damage.