Thanks for you explanation. I guess I know what do you mean.
Sorry I am quite new to torch. In theano or tensorflow, I can get gradient of any node in the computational graph. But it seems that pytorch only save gradient of the leaf nodes?
from torch.autograd import Variable
x = Variable(torch.ones(2, 2), requires_grad = True)
y = x  + 2 
z = y * y * 3
out = z.mean()
out.backward(retain_variables=True)
print(x.grad)

I get
Variable containing:
 4.5000  4.5000
 4.5000  4.5000
[torch.FloatTensor of size 2x2]

But  print(y.grad)  gives me
Variable containing:
 0  0
 0  0
[torch.FloatTensor of size 2x2]

How do I get the gradient of the inner node ( y)?