Thank you for your advice! But I am wondering if it is correct to have optimizer.zero_grad() in between two backward(). Would zero_grad() clear out all the grad of loss1 and then optimizer.step() only update the gradient from loss2?