I have a multi-output model in PyTorch when I train them using the same loss and then to backpropagate I combine the loss of both the output but when one output loss decreases others increase and so on. How can I fix the problem?
def forward(self, x):
    #neural network arch. forward pass
    x = dense1(x)
    x1 = dense2(x)
    x2 = dense2(x)
    x1 = F.log_softmax(x1) 
    x2 = F.log_softmax(x2)
    return x1, x2

out1, out2 = model(data)
loss1 = NLLL(out1, target1)
loss2 = NLLL(out2, target2)
loss = loss1 + loss2
loss.backward()

When loss1 decrease loss2 increase and when loss2 decrease loss1 increase how can i fix the issue.
Can any other operator other than ‘+’ be used to combine the loss or should I put weights to the different loss?