Autograd won’t interfere inside your custom forward and backward functions, as it’s disabled by default there.
If you are getting Inf gradients, you might want to debug your code using Anomaly Detection 2.