if you have a scipy path, you generally have to write an autograd.Function, because

converting to numpy array has to be done with x.data.numpy() which detaches the gradient path.
pytorch doesn’t know the gradient of a scipy function

See this page for an example:
http://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html 254