If you are leaving PyTorch and use 3rd party libs, such as numpy, you will detach these operations from the computation graph, and PyTorch won’t be able to calculate the gradients for them.
You could either implement custom autograd.Functions as described here 1 or use PyTorch operations instead.