T5 fp16 forward yields nan
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

I use pytorch-lightning to manage fp16. This is the minimal example that reproduces the result.

```python
from transformers import T5Model, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5Model.from_pretrained("t5-base").cuda().half()
text = "hello world!"
inputs = tokenizer.encode(text, return_tensors="pt").cuda()
out = model(input_ids=inputs, decoder_input_ids=inputs)
print(out[0][:, :, :10])
```
output:
```
tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Get non-nan values.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.0
- Platform: Linux-4.15.0-88-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Thanks for the detailed error description @binshengliu! I linked a PR that should fix it :-) <|||||>Hi, there's still some chance we get `nan` values during training. It does not always happen but is still pretty easy to reproduce. My current version is `2.11.0`.

```
import torch
from transformers import T5Model, T5Tokenizer

torch.manual_seed(0)
tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5Model.from_pretrained("t5-base").cuda().half()
text = "hello world!"
model.train()
inputs = tokenizer.encode(text, return_tensors="pt").cuda()
for idx in range(1000):
    out = model(input_ids=inputs, decoder_input_ids=inputs)
    if torch.isnan(out[0]).any():
        print(idx)
        print(out[0][:, :, :10])
        exit()
```

Output:
```
143
tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, 0.],
         [nan, nan, nan, 0., nan, nan, nan, nan, nan, 0.],
         [nan, nan, 0., nan, nan, 0., 0., 0., nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)
```<|||||>Sorry I just noticed you commented https://github.com/huggingface/transformers/issues/4586#issuecomment-639704855. I can confirm the issue I encountered is caused by the same reason.<|||||>Hi, I have the same problem. I get NAN when using fp16. And when I set fp16=False, the NAN problem is gone. <|||||>Yeah that's still an open problem....not sure how easy it will be to solve it, see: https://github.com/huggingface/transformers/issues/4586#issuecomment-639704855<|||||>Same when fine-tuning GPT Neo.