Thanks Tom, I got the grad, but it is not correct. Like the following example, i want to get the second derivative of (2x)^2 at x0=0.5153, the final result could return the 1st order derivative correctly which is 8*x0=4.12221, but for the second derivative, it is not the expected 8, do you know why?
import torch
from torch import Tensor
from torch.autograd import Variable
from torch.autograd import grad
from torch import nn
torch.manual_seed(1)
x = Variable(Tensor([2.]), requires_grad=False)
model = nn.Linear(1, 1, bias=False)
x0 = [par.data for par in model.parameters()][0]
print(x0)
loss = torch.sum(model(x)**2)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_grads = grad(loss, model.parameters(), create_graph=True)
gn2 = sum([grd.norm()**2 for grd in loss_grads]) / 2 # 2nd derive
print(‘loss %f grad norm %f’ % (loss.data, gn2.data))
for grd in loss_grads:
grd = grd.retain_grad()
model.zero_grad()
gn2.backward(retain_graph=True)
for grd in loss_grads:
print 8 * x0, grd.data[0], grd.grad