Thanks for the elaborate answer, the graph going out of scope with the output variable is the essential part I was missing here.
If these fixes 82 are what  you had in mind, then I’ll send a PR.
Let me know if you think it’s useful to make the  notebook with your answer into a full tutorial, I think these autograd graph/state mechanics are a bit underdocumented atm. Or maybe some explanation could be added to the autograd note in the docs.