@ptrblck Thank you very much!
I am thinking different architecture of multi-task network should have different BP approach. Here are 2 conditions:
Condition 1:
G(encoder  → decoder) → loss_G
G(encoder → decoder) → D → loss_D
In this condition, we should sum loss_G and loss_D up and do BP to update G.
Condition 2:
G(encoder → decoder) → loss_G
G(encoder) → G(decoder) → loss_G
|_____> D - >loss_D
In this condition, the two tasks like two branches of a tree. Only the G(encoder) is the common part. The two losses should be BP to update G(encoder). But for G(decoder) part, only loss_G should be used.
In this case, can we still feed loss_D + loss_G to optimizer?
Thank you very much!