Is there any way to get second order derivative or Hessian Matrix?
I am trying to get the Hessian Matrix of weights in a convolutional kernel. However,  there is no API which can do the job like Tensorflow.
LiuzcEECS
smth
AFAIK TensorFlow will return you a Hessian-vector product like most automatic differentiation software.
apaszke
Any updates regarding second order derivatives in PyTorch?
Ilya_Kostrikov
jekbradbury
shijie-wu
Has there been any update on this? That is, how to get the Hessian (even if just the diagonal) in Pytorch? Would something like this work:
adrianalbert
vabh
Thanks, will look into that!
adrianalbert
It looks like torch.autograd does not have a “grad” function? I’m using the latest pytorch (0.1.12_2) ImportError: cannot import name grad
adrianalbert
vabh
Got it, thanks! I installed Pytorch from source and now I have access to the grad and backward functions in torch.autograd. However when I try to use grad my kernel just crashes. I use like this: Is this not intended to be used with non-scalar (multiple-dimensional) Tensors? The examples on the github work fine. Thanks!
adrianalbert
vabh
I see, that makes sense, given that the model is a CNN with skip connections. Thanks!
adrianalbert
Are there any updates on this? Its been some time. Whats wrong with just doing w.grad.backward()?
Brando_Miranda
fehiepsi
<SCODE>import torch
from torch import Tensor
from torch.autograd import Variable
from torch.autograd import grad
from torch import nn

torch.manual_seed(623)

x = Variable(torch.ones(2,1), requires_grad=True)
A = torch.FloatTensor([[1,2],[3,4]])

print(A)
print(x)

f = x.view(-1) @ A @ x
print(f)


x_1grad, = grad(f, x, create_graph=True)
print(x_1grad)
print(A @ x + A.t() @ x)

x_2grad0, = grad(x_1grad[0], x, create_graph=True)
x_2grad1, = grad(x_1grad[1], x, create_graph=True)

Hessian = torch.cat((x_2grad0, x_2grad1), dim=1)
print(Hessian)

print(A + A.t())
<ECODE>
Albert_Chen
I’ve worked on this issue for some time, plz feel free to use it below.
JIANG_GUOQING