T5ForConditionalGeneration fp16 issues
This is a continuation of  and very related to https://github.com/huggingface/transformers/issues/4586
but the issue here is `NaN` loss during finetuning, rather than `nan` in model outputs.

### Instructions to reproduce
From `examples/summarization`
Setup cnn tiny:

```bash
wget https://s3.amazonaws.com/datasets.huggingface.co/summarization/cnn_tiny.tgz
tar -xzvf cnn_tiny.tgz
rm cnn_tiny.tgz

export OUTPUT_DIR_NAME=bart_utest_output
export CURRENT_DIR=${PWD}
export OUTPUT_DIR=${CURRENT_DIR}/${OUTPUT_DIR_NAME}

# Make output directory if it doesn't exist
mkdir -p $OUTPUT_DIR

# Add parent directory to python path to access lightning_base.py and utils.py
export PYTHONPATH="../":"${PYTHONPATH}"
```

### Call finetune.py with 'O1'
```bash
python finetune.py \
--data_dir=cnn_tiny/ \
--model_name_or_path=t5-large \
--learning_rate=3e-5 \
--train_batch_size=1 \
--eval_batch_size=2 \
--output_dir=$OUTPUT_DIR \
--num_train_epochs=10  \
--n_gpu=1 \
--fp16 \
--fp16_opt_level=O1 \
--do_train $@

```
Uses 16GB, but loss is always NAN.



### Call finetune.py without fp16

```bash
mkdir t5_large_fp32
python finetune.py \
--data_dir=cnn_tiny/ \
--model_name_or_path=t5-large \
--learning_rate=3e-5 \
--train_batch_size=1 \
--eval_batch_size=1 \
--output_dir=t5_large_fp32 \
--num_train_epochs=10  \
--n_gpu=1 \
--do_train $@

```
Result: OOM on 16GB card, uses 19GB on RTX (24GB Card)


If you try `t5_base`, it seems to use 8GB of GPU ram in fp16, and no issues.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
