detect_anomaly is already enabled which is how I discovered this problem to begin with.

You could also check for an overflow when scaling your gradient?

How do I do that? What class/method do I set a breakpoint in?

If you’re running with a low bit-wdith, scaling too high might cause an overflow which would create NaNs when backproping!

Okay, but I’m not the one deciding how high values are being scaled so if this kind of overflow happens in valid networks how do you deal with it? I was under the impression that gradient clipping was supposed to fix this but it does not.

Also, check for Infs as they can lead to NaNs under certain operations!

I already checked the input and output and neither contain NaN or Inf.