Hi,
I am working on implementing an iterative loss for a project I am working on.
Basically what I am trying to do is the following.
model = toy_net().to(device)
opt = optim.Adam(model.parameters()) #optimizer
epochs = 50                          #epochs
k = 3                                #iterations
loss_fn = nn.BCELoss()

for epoch in range(epochs):
    for inputs, labels in dataloader:
        losses_iter = [0]*3
        x,y = inputs.to(device), labels.to(device)
        for i in range(k):
            Z = ((i+1)*(i+2))/2
            out = model(x)            #get model output
            loss = loss_fn(out,y)   #get loss
            losses_iter[i] = (loss)*(i+1)
            total_loss = sum((loss_iter)/Z) #total loss is summing losses
                                                   #and scaling by Z            
            total_loss.backward()
            opt.step()
            losses_iter[i] = (loss.detach())*(i+1)

The Problem I am having right now is that once I detach the loss from the first iteration and add it in the second iteration it is just a constant and is not taken into account for the second iteration since the derivative is of the first loss which is a constant now is 0.
Any help would be appreciated