[Trainer] Add more schedules to trainer
Adds all available schedules to trainer and also add beta1 and beta2 for Adam to args.
@julien-c 
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=h1) Report
> Merging [#3988](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `33.33%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3988/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3988      +/-   ##
==========================================
- Coverage   78.45%   78.40%   -0.06%     
==========================================
  Files         111      111              
  Lines       18521    18533      +12     
==========================================
- Hits        14531    14530       -1     
- Misses       3990     4003      +13     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/3988/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `42.42% <9.09%> (-1.18%)` | :arrow_down: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/3988/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `88.40% <100.00%> (+0.71%)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/3988/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `91.89% <0.00%> (-8.11%)` | :arrow_down: |
| [src/transformers/configuration\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/3988/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3hsbmV0LnB5) | `93.87% <0.00%> (-2.05%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3988/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.59% <0.00%> (-0.17%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=footer). Last update [cb3c221...851b8f3](https://codecov.io/gh/huggingface/transformers/pull/3988?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>I'd rather let the user pass an optimizer and a scheduler in the Trainer's `__init__` and only keep the default one (which was the one used in all the scripts) in the implementation. Otherwise there's really a combinatorial explosion of possibilities.

What do you think?<|||||>Yeah it's true that this could lead to a "combinatorial explosion" but I don't really mind that because:
1. If we set good default values (like the beta ones for Adam), and have a good description I don't feel like the user is affected by more and more parameters
2. I think if a user does want to change very special params (like the beta ones), then it's very convenient to be able to do it directly

What I already started in this PR which can become messy is that there are now parameters that are only relevant if other parameters are set in a certain way `num_cycles` is only relevant if there is a cosine scheduler. And it's true that the trainer class could become messier this way as well.

In the end, for me the use case of the trainer class is more important I guess. If the idea is to have a very easy and fast way to train a model, then I would not mind introducing a bunch of parameters with good default values and description. If instead it should be a bit more "low-level" and the `run_language_modeling` script should wrap this class into a faster interface then it might be better to keep clean. <|||||>Closing this due to cafa6a9e29f3e99c67a1028f8ca779d439bc0689