Given that you modify .grad with things that requires_grad=True, I guess the next backward keep the graph from the previous iterations? Can you try setting all the .grad to None at the end of your iteration (instead of calling .zero_grad().