The GradScaler in amp uses an initially high scaling value, which could create invalid gradients in the first iteration(s), which is expected. If the scaler sees these invalid gradients, the optimizer.step() will be skipped and the scaling value will be reduced.
If you add anomaly detection at the beginning of the training, this would raise this “false positive” issue, so you should either set the initial scale factor to a lower value or enable anomaly detection after the gradient scaler stabilized.