You can pass loss into your optimizer by using a closure. Couple of examples: look at the “step” function:

https://github.com/fmeirinhos/pytorch-hessianfree/blob/master/hessianfree.py 17
https://github.com/pytorch/pytorch/blob/master/torch/optim/lbfgs.py 11