TrainingArguments --learning_rate should not be used to set both "lr" and "warmup_max_lr" in DeepSpeed
According to this document: https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/deepspeed#shared-configuration, 
--learning_rate is used to set two parameters in DeepSpeed configuration: "lr" and "warmup_max_lr". This causes learning rate never changes, because the initial value is already the maximal value. 

Here is my DeepSpeed configuration:

```
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto"
    }
  },

  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto" ,
      "warmup_num_steps": "auto"
    }
  },

```

When running with this DeepSpeed configuration, I saw both "lr" and "warmup_max_lr" are set to 5e-5:

![image](https://user-images.githubusercontent.com/51274745/152467841-4c06b0bd-c882-4989-8370-60d09d3f70e3.png)

Unfortunately, the learning rate stayed at 5e-5 all the way because "lr" is bounded by "warmup_max_lr". When not using DeepSpeed, HF is in charge of learning rate, and I see learning rate changes.

I think --learning_rate should be used to set "warmup_min_lr", not the max.
@stas00 <|||||>Let's look at the WarmupLR docs: https://deepspeed.readthedocs.io/en/latest/schedulers.html#warmuplr

> Increase the learning rate of each parameter group from min lr to max lr over warmup_num_steps steps, **and then fix at max lr**.

So max lr is the same as `args.learning_rate` in this context. And you're suggesting that it's not.

Perhaps you shouldn't use the `WarmupLR` scheduler if you don't want its behavior? I'd say perhaps you want this one instead?
https://deepspeed.readthedocs.io/en/latest/schedulers.html#warmupdecaylr

Please help me understand where you think it's not doing the right thing.

<|||||>Ah, sorry. You're right. I misread the document. I thought it's saying learning rate increases from "lr" (not "min lr") to max lr. I also noticed that my warmup_step is 0. That may be the real cause why my rate is not changing at all.

This should probably go to another thread. But have you ever seen learning curve as odd as this one? The X is epoch and the Y is loss. At the 1st step of each epoch, training loss drops significantly while validation loss jumps. As early as 2nd epoch, the validation loss is overfitted. I've been investigating whole day today, but got no clue (and tried to blame LR scheduler :( ). This is from GPT-J model. If I switch the model to GPT-neo-1.3B, I have a perfectly fine learning curve and the validation loss doesn't overfit so quickly.

![image](https://user-images.githubusercontent.com/51274745/152485064-32c0ca45-c7d2-40e6-8b76-710d499f95b8.png)


<|||||>Yes, please definitely open a new issue as it's not related to Deepspeed anymore and surely others might be able to help as I have no experience finetuning GPT-J.

The only thing I can suggest is to add more data if you can so that you don't need to repeat it more than once.

I'm closing this Issue otherwise.