Yes, here is a small sample. I use y = a*2 here because that way the op has a single input and single output and it is easier to show the graph.

a = torch.rand(10, requires_grad=True)
# no graph at this point

y = a * 2
# The backward graph linked to y is: y -> a

## Iterations 0
x_out0 = 2 * y
# The backward graph linked to x_out0 is: x_out0 -> y -> a
# Note that the backward graph linked to y is still: y -> a (the arrow here is the same as the one between y and a above, meaning that deleting the buffers in one will delete them in the other)

loss0 = measure(x_out0)
# The backward graph linked to loss0 is: loss0 -> x_out0 -> y -> a
loss0.backward()

## Iteration 1
x_out1 = 2 * y
# The backward graph linked to x_out1 is: x_out1 -> y -> a (the arrow between y and a here is the same as the one between y and a in iteration 0, meaning that deleting the buffers in one will delete them in the other)

loss1 = measure(x_out1)
# The backward graph linked to loss1 is: loss1 -> x_out1 -> y -> a
loss1.backward()
# This one will fail as the buffers for the array between y and a are already deleted.

# If you were printing the all the graphs linked to both loss0 and loss1, you would get:
# loss1 -> x_out1 -> y -> a
#    loss0 -> x_out0 ↑
# As you can see some part of the graph is shared.



It refers to both buffers and intermediary results. Without these, the “graph” cannot be used for anything and so it does not really exist anymore. We only use it to detect double usage.


These functions are autograd.Function that contain both the code to run for the backward pass and some intermediate variables needed to perform these computations. Each function handle it’s buffers and temporary variables in the way it wants.


Does that make things clearer?
Let me know if you have other doubts.