@albanD @DiffEverything  Hi, thanks for your reply. I have a follow-up question. In my implementation the input is passed through two different networks. I am interested the gradient of the final output w.r.t. each of the input.(i.e. both dz/dx and dz/dy) I tried the following code, but the y.grad is always zero. Currently I can temperately fix it by converting y to numpy and reinitialize the Variable again in the same way as x, then pass through net2. I am wondering if it is possible to get y.grad in the very first pass. Am I missing anything?
net1.zero_grad(); net2.zero_grad()
x = Variable(torch.ones((1,20)), requires_grad=True)
y = net1(x)
z = net2(y)
z.backward(torch.ones(z.size()),retain_variables=True)
print(x.grad) # this will give non-zero value
print(y.grad) # this is all zero

net1.zero_grad(); net2.zero_grad()
y = torch.from_numpy(y.data.numpy())
y = Variable(y, requires_grad=True)
z = net2(y)
z.backward(torch.ones(z.size()),retain_variables=True)
print(y.grad) # this will give non-zero value