Summing the losses (can be done with weighting or equal proportions) is definitely different from running them separately. If you do loss.backward() with the (weighted) sum then everything is backpropagated, which is not necessarily what you want.

Thanks for clarifying this. I would like the weights of ModelA to be dependent on ModelB, too. Therefore I should be combining the losses to backward propagate.  I guess something like:
losses =  (a * lossA) + (b * lossB)
losses.backward()


In your case, I think it depends on whether you want to train weights in ModelA when you train ModelB , or if you keep them separate (i.e. the weights in A are fixed when training B). If you include parameters of A in the second loss as a (possibly weighted) sum, then those parameters are going to be updated by the optimizer step.

I was getting confused because I am basically calling lossA.backward() and lossB.backward() separately but the optimizer object gets list(modelA.parameters()) + list(modelB.parameters()). I thought that that would make the trick, but as your reply suggests, that is not correct. When I do that I am basically updating, for example, modelA's  parameters using gradients of  modelA.parameters() only.

Also if you simply take the A loss and add it to the B loss, then there are weights in A that contribute to the A loss but should not contribute to the B loss (in your drawing: layer A3). So I guess you would need an intermediate A loss that takes that into account.

This is an interesting point. I think I will have to explore because I have no idea.

Disclaimer: I’m not familiar with and have never trained an encoder/decoder network, but what I’ve described seems like the logical behaviour to me…

That is fine. Your comments make sense to me. I am not familiar with these models either . Thank you, this has been very useful. This community is very cool.