If the model output or losses do not contain NaN values and the scaler is constantly decreasing the scale value, some operation might cause large gradients, which are still overflowing. Could you post an executable code snippet, which reproduces this error?