You could define a new autograd.Function (tips here: http://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html 18 ). This would involve implementing forward() and backward() calls for your loss function.
Autograd only works with functions that are performed on Variables. If you send the output of your network to scipy/numpy functions, then autograd won’t be able to record the operations, which is why you’ll need to define your own backward().