Delay decay schedule until the end of warmup
The decay schedule should start at the end of the warmup steps. Without this simple edit, learning rate will drop at the end of the warmup steps, like the figure shown below:
<img width="378" alt="image" src="https://user-images.githubusercontent.com/15783079/84426617-c5820500-ac38-11ea-9579-2c0d5f374e33.png">

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=h1) Report
> Merging [#4940](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6293eb04dfef704e87a6e0b358848ffc41587b4f&el=desc) will **increase** coverage by `0.40%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4940/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4940      +/-   ##
==========================================
+ Coverage   77.20%   77.60%   +0.40%     
==========================================
  Files         128      128              
  Lines       21746    21746              
==========================================
+ Hits        16789    16877      +88     
+ Misses       4957     4869      -88     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `57.27% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.42% <0.00%> (+0.15%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `78.65% <0.00%> (+2.29%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `56.68% <0.00%> (+23.24%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=footer). Last update [6293eb0...96ae67a](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>@jplu what do you think of this?<|||||>Fix #5098