Hi @valhalla
The fix seems not to be applied on the MT5 model:

Nan losses occur when fine-tuning the model with the latest (stable) version of transformers.
MT5 works well with fp16.

Do you see the same behavior?