Trainer: Support scheduler that reduce LR on loss plateau.
# 🚀 Feature request

The HuggingFace Trainer currently only supports learning rate schedulers where the learning rate follows a fixed trajectory. Schedulers that adapt the learning rate to the state of the training procedure are (as far as I know) not supported.

As en example, take [ `ReduceLROnPlateau`](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html) from `torch.optim.lr_scheduler`. This scheduler reduced the learning rate when a given metric is stuck on a plateau. In its `step` method, this scheduler requires an input. This input, typically the validation metric, is used to decide whether or not to decrease the learning rate.

This behaviour could be supported by giving the scheduler access to the training/validation metrics. This would most likely involve changing [this piece of code](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/trainer.py#L1468).

I believe that [`TrainerState`](https://huggingface.co/docs/transformers/v4.17.0/en/main_classes/callback#transformers.TrainerState) holds the training/validation metrics in its `log_history` attribute. If the scheduler would have access to `TrainerState`, it would be easy to create a 'stateful' scheduler.

## Motivation

This change would allow user to easily use more complex learning rate schedulers together with the `Trainer` class.


Might be of interest to @sgugger <|||||>This is not a high-priority feature on our list, as no Transformers pretraining or fine-tuning is making use of such schedulers in the literature. If you want to make a PR with this, I'm happy to have a look though.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.