It seems like to me that the autograd engine is recording reference of computation. Once you created the computation, it’s recorded in the engine. (The bad thing is you cannot dynamically change the graph you have already created)