Solved by albanD in post #5 
              
              
                Hey, 
Each “Node” in the graph implements a special functions to release all the resources it can here: pytorch/function.h at a46d56f988505547b0779838a022970b79123b3c · pytorch/pytorch · GitHub 
This is called from the execution engine when retain_graph=False here: pytorch/engine.cpp at a46d56f98850…