Based on the error message it seems that F^(-1) might apply some inplace operations on the input tensor, which replace values needed to calculate the gradients. A .clone() operation might solve this issue.
Nevertheless, you could check, if the desired gradients are calculated by calling backward() on intermediate outputs or just the final loss and checking the .grad attributes of all parameters, which should get gradients.
If some of these .grad attributes are returning a None value, Autograd didn’t calculate any gradiets for them and your computation graph might have been broken.



 SEM:

I have converted them to numpy to do that.


That sounds risky, as this would cut the computation graph, since Autograd isn’t able to backpropagate through numpy operations. You would either need to stick to PyTorch methods or implement a custom autograd.Function as described here 7.