Thanks that’s an excellent explanation! As you explained, in the second example when y.backward() was called the second time the variable ‘a’ was freed and the graph was not built again. That’s the reason why we can not let the gradient flow through. But I’m curious about, was ‘x’ also been freed? x is nn.Module so it is a collection of two leaf variables so in principle it is also an ‘intermediate variable’.
Besides, thanks to your comment I found another question that is not obvious to me. Consider the following code:
import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))
x_out = x(a)

for i in range(100):
    y = torch.sum(x_out)
    y.backward()

This code will fail, which is unexpected. I think in the second iteration the graph is recreated because we call y=torch.sum(x_out) , which recreates all the Variables associated with y again. The problem disappears if I move x_out in the loop:
import torch

x = torch.nn.Linear(3,3)
a = torch.autograd.Variable(torch.randn(2, 3))

for i in range(100):
    x_out = x(a)
    y = torch.sum(x_out)
    y.backward()

So my guess is, when you apply an operation like torch.sum it won’t recreate a graph. But x_out = x(a) will recreate a graph. Actually it is just a forward() call of the nn.Module object x. After all these experiences my crude conclusion is: the forward() call of nn.Module will recreate a graph associate with its return value. But this is insufficient…