Is your model generally working fine without using amp?
The loss scaler might run into this “death spiral” of decreasing the scale value, if the model output or loss contains NaN values.
These NaN values in the loss would thus create NaN gradients and the loss scaler will decrease the scale factor as it thinks the gradients are overflowing.
However, in fact the gradients are not overflowing, but your model yields invalid outputs.
Could you check the output and loss for NaNs and check, if they are also created without amp?