If your model outputs NaN values, the training is generally broken and I don’t know if the model can somehow recover from it. Additionally the scaler will reduce the loss scaling value, as the backward pass creates NaN gradients, and it thus assumes that the scaling factor is too high.
I would recommend to remove automatic mixed precision for now and make sure to fix the NaN issue.
Based on the error message, you might pass a negative values to a pow method, which would calculate a root?