Is your model training correctly using FP32?
The loss scaler should reduce the loss scaling at the beginning, but should eventually stop decreasing the loss continuously.
I would still recommend to make sure the model trains in FP32 and the loss looks reasonable.