mbsyed:

But when I update the model and try to do a backprop for the second iteration whose loss is the added loss of iteration 1 and iteration 2 even with retain_graph=True I get an error. I am not sure how to get around this and what is causing this issue.


I guess you are seeing an error, since you are trying to calculate the gradients (in iter2) from stale forward activations (calculated in iter1).
Have a look at this post 3 for more information and check, if you are hitting the same error.