Adding on to @richard 's answer, if two losses are backwarded in each optimizer step, it might be easier to use register_hook since it avoids storing previous grad values. http://pytorch.org/docs/master/autograd.html 38