I also cross-posted this on stack overflow, in case anyone is helped by that: python - How to avoid huggingface t5-based seq to seq suddenly reaching a loss of `nan` and start predicting only `<pad>`? - Stack Overflow