fp16 flag silently fails
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0.dev0
- Platform: Linux-5.10.68+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.1 (True)
- Tensorflow version (GPU?): 2.6.2 (True)
- Flax version (CPU?/GPU?/TPU?): 0.3.6 (gpu)
- Jax version: 0.2.25
- JaxLib version: 0.1.70
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N

Models:

1. mT5-small
2. mT5-base

both have the same behavior


Library:
- Trainer: @sgugger


## Information
The problem arises when using: 
*  my own modified scripts: https://github.com/rumeshmadhusanka/mt5-simplification/blob/main/finetune.py derived from and very much  identical  to the official translation example

The tasks I am working on is:
* text simplification
* translation

## To reproduce

Steps to reproduce the behavior:

1. Run a translation/simplification task task turning fp16 flag on
My params(for simplification):
`model="google/mt5-base"
!python mt5-simplification/finetune.py \
    --model_name_or_path $model \
    --do_train \
    --fp16 \
    --do_eval \
    --adafactor \
    --source_lang com \
    --target_lang sim \
    --source_prefix "com-sim: " \
    --train_file train.json \
    --validation_file valid.json \
    --output_dir mt5-simplification \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --save_total_limit=1 \
    --adam_epsilon=1e-6 \
    --learning_rate=3e-5 \
    --save_strategy=epoch \
    --report_to="wandb" \
    --max_steps=1200 \
    --warmup_steps=250 \
    --overwrite_output_dir \
    --log_level debug \
    --output_dir saved \
    --predict_with_generate 
`

Some of the output logs:<br>
> 0%|                                                  | 0/1200 [00:00<?, ?it/s]/kaggle/working/transformers/src/transformers/trainer.py:1366: FutureWarning: Non-finite norm encountered in torch.nn.utils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
  args.max_grad_norm,
 14%|█████▌                                  | 166/1200 [00:39<03:58,  4.33it/s]/opt/conda/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
{'loss': 0.0, 'learning_rate': 2.7347368421052632e-05, 'epoch': 0.02}           
 83%|████████████████████████████████▌      | 1000/1200 [04:10<10:45,  3.23s/it]{'loss': 0.0, 'learning_rate': 1.1557894736842106e-05, 'epoch': 0.03}
100%|███████████████████████████████████████| 1200/1200 [04:56<00:00,  4.39it/s][INFO|trainer.py:2033] 2021-12-18 02:11:18,316 >> Saving model checkpoint to saved/checkpoint-1200


 > ***** train metrics *****
  epoch                    =       0.04
  train_loss               =        0.0
  train_runtime            = 0:05:11.02
  train_samples            =     120000
  train_samples_per_second =     15.433
  train_steps_per_second   =      3.858
[INFO|trainer.py:2281] 2021-12-18 02:11:46,330 >> ***** Running Evaluation *****
[INFO|trainer.py:2283] 2021-12-18 02:11:46,330 >>   Num examples = 2000
[INFO|trainer.py:2286] 2021-12-18 02:11:46,330 >>   Batch size = 4
100%|█████████████████████████████████████████| 500/500 [01:56<00:00,  4.31it/s]
***** eval metrics *****
  epoch                   =       0.04
  eval_bleu               =     0.0126
  eval_gen_len            =       8.16
  eval_loss               =        nan
  eval_runtime            = 0:01:56.24
  eval_samples            =       2000
  eval_samples_per_second =     17.205
  eval_steps_per_second   =      4.301

When I run a translation task on Kaggle's GPU(Tesla P100-PCIE) or AWS's T4 GPU the training  loss is always zero. This has been tried out multiple times with different training params.
## Expected behavior
Loss not to be zero while training<br>
Throw an error message if the GPU doesn't support fp16 <br> 
<!-- A clear and concise description of what you would expect to happen. -->

It's not an issue of GPU not supporting fp16. It's an issue of many models trained in bf16 and attempted to be used with fp16 and overflowing due to the incompatible numerical range. bf16-pretrained models use much bigger weight values than fp16 can accommodate so it overflows.

Please see: https://github.com/huggingface/transformers/pull/10956 for various workarounds.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.