Problem with mT5 and the official Summarization notebook
## Environment info

- `transformers` version: 4.6.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten @patil-suraj @sgugger 

## Information

I am using mT5-small on the [official summarization notebook](https://github.com/huggingface/transformers/tree/master/examples/pytorch). However when trained, the model gets nan loss values and outputs non-sense.

I made some changes to speed up the training such as loading 5% of the data, changing max input length to 256 from 1024 and the batch size to 8 from 16 however my settings work perfectly fine with t5-small and I get a high rouge score with sensible outputs and loss values. The problem seems to be mT5.

## To reproduce

Here is my [Colab notebook ](https://colab.research.google.com/drive/16-6yIHFQQ1Q8meVYqFn21Tw2eoG9wliU?usp=sharing)which you can see the output at the end.

TrainOutput(global_step=1276, training_loss=nan, metrics={'train_runtime': 340.4428, 'train_samples_per_second': 3.748, 'total_flos': 1196714720985600.0, 'epoch': 1.0, 'init_mem_cpu_alloc_delta': 1543725056, 'init_mem_gpu_alloc_delta': 1200707584, 'init_mem_cpu_peaked_delta': 0, 'init_mem_gpu_peaked_delta': 0, 'train_mem_cpu_alloc_delta': 10055680, 'train_mem_gpu_alloc_delta': 1203914240, 'train_mem_cpu_peaked_delta': 65540096, 'train_mem_gpu_peaked_delta': 4225469440})

## Expected behavior

Training loss should not be nan.

disabling fp16 seems to solve the issue of nan loss, but I wouldn't call this issue closed because this doubles the training time :(<|||||>Hey @demegire,

Sadly MT5 doesn't really work with fp16. There are a bunch of issues regarding this problem...see:
- https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139/5
- https://github.com/huggingface/transformers/issues/10830<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.