Solved by SimonW in post #2 
              
              
                As you said, the script you have keeps the computation graph alive until the output = ... in the second iteration. In the usual case where a loss is backproped in each iteration, the graph is freed after calling backward, and thus OOM can’t happen. 
The other way to do this is to wrap the iteration …