Is the position of the scheduler.step() correct?
## ❓ Questions & Help

In the [lm_finetuning_file](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L285-L319), the scheduler used is "get_linear_schedule_with_warmup" which in turn uses the underlying "LambdaLR" [ref](https://github.com/huggingface/transformers/blob/594ca6deadb6bb79451c3093641e3c9e5dcfa446/src/transformers/optimization.py#L47);

On a careful study of the file, the "scheduler" is being called for every batch, but that isn't what's as per the official PyTorch [docs](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.LambdaLR) to change LR since they are calling it at "epoch-level",

```
>>> lambda1 = lambda epoch: epoch // 30
>>> lambda2 = lambda epoch: 0.95 ** epoch
>>> scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
```

I meant it makes sense to me to do it at batch level change the lr etc but not sure why the PyTorch has it differently?

Thanks.
Why this issue were left without an answer? I don't know what is the answer, but it seems that this scheduler should be executed at batch-level since there are other examples using batch-level instead of epoch-level: https://github.com/huggingface/transformers/blob/8e8384663d716d4b5a4f510070ff954fc0ba4a52/examples/research_projects/bert-loses-patience/run_glue_with_pabee.py.

Anyway, LambdaLR is a generic LR scheduler which PyTorch provides in order to implement other LR schedulers. It is true that PyTorch docs talks about epoch-level, but there are other LR schedulers, like [CyclicLR scheduler](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CyclicLR.html), which explicitely indicates that has to be executed at batch-level. Since LambdaLR is a generic LR scheduler, this scheduler will need to be executed at batch-level or epoch-level depending on the specifid LR scheduler implemented. For the linear LR scheduler of the issue, I guess that the correct is to be executed at batch-level, or even could be adapted to epoch level if you want, but taking a look at the scripts of the repository, they use batch-level.