Thank you for the response. That works but only if I do not update the weights using opt.step(). In my use case I need to update the weights before I run the next iteration.
Basically, I am working on a segmentation problem. My input are images with 4 channels (RGB and a fourth channel which is initialized to zeros). After the first iteration the 4th channel (zeros) is replaced with outputs from the updated model in the first iteration. Then this new input is fed into the same model again and we calculate the loss for the current iteration and add the loss from the last iteration. It would essentially look something like the following.
model = Net()
x         = (samples,4,height,width)
y         = (samples,1,height,width)
opt      = Adam(model.parameters())

#-------- iteration 1 -----------
opt.zero_grad()
outputs1 = model(x)
loss1    = loss_fn(outputs1,y)
loss = (loss1)*1
loss.backward(retain_graph=True)
opt.step()
x[samples,3,:,:] = model(inputs).detach()          #inputs updated for iteration 2

#-------- iteration 2 -----------
opt.zero_grad()
outputs2 = model(x)
loss2      = loss_fn(outputs2,y)
loss        = (2/3)*(loss2) + (1/3)*(loss1)           #loss weighted from previous iter and current iter
loss.backward(retain_graph=True)                      #This gives an error:  one of the variables needed for 
                                                       gradient computation has been modified by an inplace operation. 
                                                       #If I don't do opt.step() then there is no error
opt.step()
x[samples,3,:,:] = model(inputs).detach()          #inputs updated for iteration 3

#-------- iteration 3 -----------
opt.zero_grad()
outputs3 = model(x)
loss3      = loss_fn(outputs3,y)
loss        = (1/2)*(loss3) + (1/3)*(loss2) + (1/6)*(loss1)           #loss weighted from previous iters and current iter
loss.backward()             
opt.step()


I think the main issue I am having now is that I need to update the model to update my input for the next iteration using the outputs from the updated model. But when I update the model and try to do a backprop for the second iteration whose loss is the added loss of iteration 1 and iteration 2 even with retain_graph=True I get an error. I am not sure how to get around this and what is causing this issue.