fp16 DDP training in 4.31.0
### System Info

pytorch 1.13.1
transformers==4.31.0

### Who can help?

Hi @sgugger ,

I used the 4.31.0 to train a Llama model with LoRA. I observe some problems with --fp16 training and I'm not sure if it is a bug in Trainer.py:

My model is like:

```
class MyModel(nn.Module):
    def __init__(self, model_name):
        super().__init__()
        self.model_name = model_name
        self.base_model = LlamaForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)
        self.base_model = get_peft_model(self.base_model, lora_config)
        self.other_modules = nn.Linear(4096, 4096)
```

I used the Trainer to train the model with the following command line:
`torchrun --nproc_per_node=4 main.py --max_steps 100000 --fp16
`
I find the model's gradients (in self.optimizer in the Trainer) are not fp16 but fp32. Is it correct?

Also, I find that no gradient_scaling is conducted during training since self.do_grad_scaling is always False (because self.sharded_ddp is None and args.half_precision_backend will be always "auto"). The current trainer.py will not correctly set up args.half_precision_backend and scaler if self.sharded_ddp is None. Are these observations expected? I'm a little confused why setting up args.half_precision_backend and scaler require sharded_ddp. As a result, I've found that during the training process, I often encounter the loss becoming NaN. I'm not sure whether it is because no gradient_scaling is conducted and half_precision_backend is not correctly set up during training.

Following are my grad_norm (before grad_clipping) with and without --fp16. (My base model here is "JackFram/llama-160m" for debugging) **The results are significantly different.**

Without --fp16:
step 1: grad_norm=0.059
Step 5: grad_norm=0.054
Step 10: grad_norm=0.048
Step 15: grad_norm=0.050
Step 20: grad_norm=0.050

With --fp16:
Step 1: grad_norm = nan
Step 5: grad_norm = 129.88
Step 10: grad_norm=126.98
Step 15: grad_norm=149.58
Step 20: grad_norm=80.7

```
def compute_grad_norm(optimizer): # the function to compute grad_norm
    total_norm = 0.0
    for group in optimizer.param_groups:
        for param in group['params']:
            if param.grad is not None:
                param_norm = param.grad.data.norm(2)
                total_norm += param_norm.item() ** 2
    total_norm = torch.sqrt(torch.tensor(total_norm))
    return total_norm
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Expected behavior

do_grad_scaling=True when --fp16 is enabled; rarely confronting loss becoming nan
Similar problem here. When using fp16 to train Llama2 with LoRA, always nan. Bf16 works well, but official Llama2 uses fp16.<|||||>Oh, I understand! It doesn't use the cuda.amp but uses accelerator to automatically handle the loss scale, right?

And the reason why fp16's gradient is large is that they're scaled gradients. 

However, it is still strange that gradients in the optimizer are fp32. Are they designed so for scaling? I'm sorry that I'm not very familiar with accelerator<|||||>This is **mixed precision** training. The gradients are computed in float16 but converted to float32 to do the optimizer update as we can't do the update in low precision. As for properly computing the gradient norm, you need to unscale the gradients first to compare them to a training without mixed precision (or bfloat16 as the training in bfloat16 does not require gradient scaling).<|||||>> This is **mixed precision** training. The gradients are computed in float16 but converted to float32 to do the optimizer update as we can't do the update in low precision. As for properly computing the gradient norm, you need to unscale the gradients first to compare them to a training without mixed precision (or bfloat16 as the training in bfloat16 does not require gradient scaling).

Many thanks for your answering. If I use --fp16 during training, what other arguments should I add for this? I am confused that using --fp16 to tune Llama2 always meets nan, but --bf16 works well.<|||||>I also observed the same thing that mixed precision training of llama-7b is very frequently resulting in nan losses. The issue does not exist for 13b for me. As you say, bfloat16 is more stable. I dont think there is anything wrong in the code base, rather some strange peculiarity with the 7b weights? Curious if someone has some insights on that.<|||||>> I also observed the same thing that mixed precision training of llama-7b is very frequently resulting in nan losses. The issue does not exist for 13b for me. As you say, bfloat16 is more stable. I dont think there is anything wrong in the code base, rather some strange peculiarity with the 7b weights? Curious if someone has some insights on that.

That's interesting findings. I haven't tried the 13b llama-2 yet.<|||||>I meet a more confused thing! When I torchrun/deepspeed fp16 train glm or baichuan with 1 or 2 gpus, the loss is ok. But when i use more than 2 gpus, like 3, the loss will overflow until fail! My gpus is V100, and i have try different version of transformers\torch\deepspeed<|||||>> I meet a more confused thing! When I torchrun/deepspeed fp16 train glm or baichuan with 1 or 2 gpus, the loss is ok. But when i use more than 2 gpus, like 3, the loss will overflow until fail! My gpus is V100, and i have try different version of transformers\torch\deepspeed

I observed the similar results. When using more gpus or larger gradient accumulatio steps, the result doesn't become better (as expected) but often becomes worse (using fp16 in v100)<|||||>so why? what is the difference between 2GPUs and 3(or more) GPUS when do training that causes the unexpected result. ps: I used to run the same train code in A100 with AMP fp16 which is ok.