Solved by ptrblck in post #8 
              
              
                Yes, executing another forward pass should work. Another approach would be to compute the gradients for both losses and use optimizerX.step() afterwards, but it depends on your actual use case, if that’s possible. 
Zeroing out the gradients of optimizer12 looks valid, but note that the forward pass …