Yes this is somewhat close to what I want. But I want to use different optimizer for new_loss and a different one for original_loss. So essentially, I do not want to add the gradients from new_loss to ones from original_loss.
I have something like this:
op1 = optim.Adam(net.parameters(),...)
op2 = optim.Adam(subnet.parameters(),...)

Also, do I need create_graph=True since I am computing higher order derivatives later?