I would expect the code to run slower if the anomaly detection is enabled, as some additional checks are performed, e.g. here 36.
I personally just use it for debugging and thus think it shouldn’t match the performance of code with disabled anomaly detection.
Once you’ve found the the reason for the NaNs, you could disable it.