Ahh, i am not sure but i think my initial comment might be a bit misleading. Let’s see via an example.
Let’ss ay we are working on Image Classification / Bounding Box (locating where the object of interest lies within an image) + Generating the caption as well at the same time. Now these tow things will share a common backbone pretty much and then they will diverge into two separate “heads”. And naturally we will have two separate losses as well for the respective heads.
So, we have loss_from_classification_head and loss_from_captioning_head computed with us and then we can compute the total loss directly by loss_from_classification_head + loss_from_captioning_head and only calling backward once. PyTorch will take care of stuffs for you pretty much.
In your initial example, you are creating multiple optimizers, so do you plan to sue different learning_rate for them? if not, you can pretty much just do main_model.parameters()` and pass it to an optimizer and have that model output data as per the heads etc that you want to compute the loss for.
loss1 = Loss1(imgs, reconstructed)
loss2 = Loss2(prediction, labels)

total_loss = loss1 + loss2
total_loss.backward() # that's it!



Thanks! If doing backward() once, is there a way to update parameters according to each sub loss?

If by this, Do you mean that you want to use separate learning rate for each separate sub-model components that you have defined above? (e.g. encoder’s LR is 1e-3, decoder’s LR is 1e-4 and classifier’s LR is 1e-2?)
If that’s what you want to do, then just pass the LR in a single optimizer by separating the model parameters.
e.g.

# lrs is a list of 2 elements basically.
opt = optim.Adam([{'params': model.body.parameters(), 'lr':lrs[0]},
                {'params': model.head.parameters(), 'lr': lrs[1]}])

Please let me know if this helps!