mbsyed:

The Problem I am having right now is that once I detach the loss from the first iteration and add it in the second iteration it is just a constant and is not taken into account for the second iteration since the derivative is of the first loss which is a constant now is 0.


That is the expected behavior of detached tensors. If you want to add the loss into account in the second iteration, you would have to keep it attached to the computation graph and might need to set retain_graph=True in the backward operation.