eval_loss is nan for GPT2 trained with fp16 + deepseed on 8xA40s
Hi, 
I'm training GPT2 from scratch. After 5 epochs, the training loss decreased dramatically and the eval loss became nan. The last steps during training all produce the same logs:

```
[2021-12-19 16:54:45,098] [INFO] [unfused_optimizer.py:275:_update_scale] Grad overflow on iteration: 445299
[2021-12-19 16:54:45,098] [INFO] [unfused_optimizer.py:277:_update_scale] Reducing dynamic loss scale from 1 to 1
[2021-12-19 16:54:45,098] [INFO] [unfused_optimizer.py:202:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 1, reducing to 1
```

Now I wonder whether GPT2 was trained using bf16 like T5, and I shouldn't have used fp16+deepspeed.  Below you can find my command and deepspeed configuration:
`
deepspeed ./5.run_clm-post.py  --model_name_or_path gpt2-large --train_file dataset.txt --tokenizer_name tokenizer --do_train --do_eval --output_dir output --evaluation_strategy steps --eval_steps 10000 --save_steps 10000 --num_train_epochs 12 --per_device_train_batch_size 28 --cache_dir .cache2 --fp16 --deepspeed ds_config_zero1.json --save_total_limit 2`

ds_config_zero1.json:
```
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"}
```
![imagen](https://user-images.githubusercontent.com/50919790/146681634-c6b89390-570c-4237-a927-6d936ae2069f.png)


Any help is appreciated. I assume I have to start the training over, am I right?

Thanks.

Maybe of interest to @stas00 <|||||>Thank you, @LysandreJik, for the ping. The issue template does say to tag me on Deepspeed issues.

Yes, @aqred1, you're running into overflow and yes Deepspeed should assert there as I proposed here: https://github.com/microsoft/DeepSpeed/issues/1599 But it's not a bug per se, just not user-friendly.

Since you're on Amphere you can switch to bf16 which is currently in PR stage since Deepspeed hasn't merged their side yet. Please see this PR: https://github.com/huggingface/transformers/pull/14569 
and additionally you will need to use this Deepspeed branch: https://github.com/microsoft/DeepSpeed/pull/1453

A while back I started compiling the info on how different models were pretrained but I couldn't find any info on gpt2.
https://discuss.huggingface.co/t/compiling-data-on-how-models-were-pre-trained-fp16-fp32-bf16/5671
I suspect that the original wasn't trained in bf16.

I very much doubt the overflow issue itself has anything to do with deepspeed. You will most likely have the same issue if you remove deepspeed. You need to tweak your hparams and see where you're doing something incorrect, e.g. your learning rate could be too big.<|||||>Thank you, I didn't know I had to tag you, sorry about that, will read all templates next time.

I'll move to bf16+deepspeed and check hyperparameters as well.

As this is not really a bug or issue, feel free to close.