After 3 days of training with fp16 on NaN loss happened. Created issue Pegasus pretraining in fp16 results in NaN loss · Issue #12225 · huggingface/transformers · GitHub 12, maybe someone knows how it can be fixed.