Are you using anomaly detection in your code?
If so, could you disable it if you are using amp, since NaN gradients are expected to occur sometimes (especially during the first iterations before the GradScaler stabilizes the scaling factor)?