I am training two losses for multi-label autoencoder. Both losses aim at different segments in the attention module of the model. Initially, I was thinking of combining the two losses into a single loss like;
loss = loss1 - coef. * loss2 // having coef. to give less attention to loss2.
But in this case, the total loss starts going negative and keeps increasing in negative after some epochs. So I am thinking of updating both the losses separately for the model like;
loss1.backward(retain_graph=true)
loss2.backward()
optimizer.step()

I am looking for more suggestions and if I am missing something in my first approach (combined loss).