One thing you could try is add d_loss1 and d_loss2 together and then use that loss for the backward step. Also it doesn’t seem like you run it for multiple epochs. You should do that if you want it to improve. If you do use epochs can you show the full code for that thanks.