tom:

You can definitely avoid .backward through the nets several times by just summing the two loss components.


The issue is not calling .backward twice. The issue is redundant backward passes. Specifically: f_grad requires a backward pass and compute_f_loss().backward() requires a backward pass.
I can post my code but it is long and complicated. The small example I gave in an earlier post had all the same issues.
I don’t know if this is helpful, but if I were working in tensorflow, I would compute f_grad with tf.gradients and then I could feed those same gradients to the optimizer using opimizer.apply_gradients(variables, gradients). If torch has some kind of similar mechanism, it would solve my issue.