set_detect_anomaly(True) is used to explicitly raise an error with a stack trace to easier debug which operation might have created the invalid values. Without setting this global flag, the invalid values would just be created and the training might be broken (e.g. if you update any parameter to NaN).


NaN values can be raised through multiple operations and often it’s used when e.g. you are dividing by zero in the model (or a very small value close to zero) and forget to add a small eps value to the divisor.