Solved by ptrblck in post #7 
              
              
                This might be the culprit, as the computation graph might be kept alive using this approach. 
Assuming you don’t want to backpropagate through the previous iterations, you could detach the tensor via op.detach_() in each iteration.