Thank you for your reply Alban. On this simple example, can you tell me if I get it right ?
import torch

net = torch.nn.Sequential(
    torch.nn.Linear(1, 1, bias=False),
    torch.nn.Linear(1, 1, bias=False)
)

x = torch.tensor([3.])
y = torch.square(net(x))
z = 2 * y

Here, a single computational graph is computed “somewhere” by the autograd package. It contains Function objects. One of these function object can be accessed through y.grad_fn, and another can be accessed through z.grad_fn. The computational graph contains the intermediary node values obtained through the forward pass. When z.backward() is called, the computational graph is used to fill the grad attributes of the parameters in net. Intermediary values are then freed, and a second call to z.backward() will result in an error. Actually, even a call to y.backward() will result in an error, proving that the same computational graph is shared among those two values.
However, the graph is not completely destroyed after the call to backward, as z.grad_fn still exists and points to the MulBackward0 object. If, however, y and z both go out of scope later, then the graph will be completely freed.