[Seq2Seq] (Byt5) zero loss
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master
- Platform: win + linux
- Python version: 3.8
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten @patil-suraj 

## Information

Model I am using (Bert, XLNet ...):
byt5

for comparison, e.g coding mistake on my side I used other seq2seq models like t5 too, these models are working as expected

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
![image](https://user-images.githubusercontent.com/47894090/138594846-3238ceb9-e634-449c-83a1-df7edd081ade.png)

Steps to reproduce the behavior:

1. running official seq2seq example script with trainer
2. train using byt5 models, any size with fp16 pytorch backend
3. see loss is going to zero after some steps (500 maximum to zero loss in my experiments)
4. do inference and see that every output is '' 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

training the model with reasonable loss and generating good text

<!-- A clear and concise description of what you would expect to happen. -->

Not sure if ByT5 supports fp16 training, cc @patrickvonplaten <|||||>Hi!

I am not sure if we have tested ByT5 with seq2seq scripts yet. Which script are you using, `run_translation.py` or `run_summarizaton.py` ? Would be nice if you could post a snippet to reproduce this.

Also, note that T5 (and ByT5 as well) models are trained with `bf16`, which may or may not work with `fp16`.  See this [discussion](https://discuss.huggingface.co/t/mixed-precision-for-bfloat16-pretrained-models/5315) or forum
However, this usually results in `nan` losses which isn't the case here. So I won't be sure without looking at the command that you are using.<|||||>removing the --fp16 argument fixes it, when using run translation script