loss 0.0 or NaN when training T5 or Flan-T5 models with bf16 on multiple GPUs
### System Info

- `transformers` version: 4.28.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-Ubuntu-22.04-jammy
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@stas00

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
1. Launch the example summarization training pipeline with `--bf16` and one of the Flan-T5 or T5 models such as `google/flan-t5-base` and `t5-large`. 
```
python3.7 examples/pytorch/summarization/run_summarization.py \
    --report_to none \
    --bf16 \
    --model_name_or_path google/flan-t5-base \
    --evaluation_strategy steps \
    --logging_strategy steps \
    --logging_steps 10 \
    --save_strategy steps \
    --save_steps 30000 \
    --num_train_epochs 3 \
    --do_train \
    --do_eval \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir /tmp/tst-summarization \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --max_train_samples=10000 \
    --max_eval_samples=100 \
    --overwrite_output_dir \
    --overwrite_cache
```
Then you will see the following logs:
```
{'loss': 0.0, 'learning_rate': 4.893503727369542e-05, 'epoch': 0.06}
{'eval_loss': nan, 'eval_runtime': 1.6413, 'eval_samples_per_second': 60.927, 'eval_steps_per_second': 2.437, 'epoch': 0.06}
```

### Expected behavior

Loss and eval_loss are wrong during training. Only T5 model works properly is `t5-small`, same as what was mentioned here: https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139, but other `T5` or `Flan-T5` models (including `flan-t5-small`) still suffer from this issue. I can train with FP32 without this problem, but would like to know if the fix (maybe not relevant here? since bf16 is different from fp16) mentioned has been incorporated.
BTW, if I use deepspeed to launch the script then `loss` and `eval_loss` are normal.
In addition, the 0.0 or NaN only happens when it is trained on multiple GPUs in the DataParallel mode, which is aligned with this issue: https://github.com/huggingface/transformers/issues/18899<|||||>Thank you for the good report and the link to the other similar issue, @cchen-dialpad 

I think hardly anybody uses DP since DDP was introduced. Is there a reason to use DP when DDP is by far more superior?

Switching to DDP was the resolution of the ticket you linked to: https://github.com/huggingface/transformers/issues/18899#issuecomment-1249262873

<|||||>@stas00 Oh, I guess I just got curious why that is the case, DDP works but DP doesn't  :)<|||||>You're more than welcome to try to figure it out, @cchen-dialpad - I haven't used DP in many years, perhaps it's not being well maintained because it's rarely used? It'd be an optimizer issue most likely if you want a place to start.<|||||>lol I see, thanks for the pointers!