I guess you should look into the following link -
https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html 412
This link explains how to incorporate scipy functions into the computational graph.