Add AdaFactor optimizer from fairseq
Tested for T5 finetuning and MLM -- reduced memory consumption compared to ADAM.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #1256

Hey @sshleifer -- here is belated PR for AdaFactor. Please let me know how to edit this properly, and what tests or examples we should add. Thanks!<|||||>We will integrate into examples/ in a separate PR I think.<|||||>Thanks @sshleifer -- let me try to make those changes. 

Agree that I should be able to add a single test -- appreciate the link -- and you can add examples in separate PR.

If I don't get this figure out soon, yes happy for you to make the changes yourself :-)<|||||>Hey @sshleifer -- think I got a test working finally. We can squash the commits. 

Still not sure what I need to clean up for the code standards/linter. 

Please advise, thanks!<|||||>For local style checking, you need: `pip install isort --upgrade` 
Then `make style` and `make quality` to both suggest you have no errors. 
They should autofix things or at least give error messages. My workflow is to define
```bash
sty () {
	make style
	flake8 examples templates tests src utils
}
```
and then run `sty` a lot.<|||||>Also squashing happens automatically at merge time, don't worry about that.<|||||>> For local style checking, you need: `pip install isort --upgrade`
> Then `make style` and `make quality` to both suggest you have no errors.
> They should autofix things or at least give error messages. My workflow is to define
> 
> ```shell
> sty () {
> 	make style
> 	flake8 examples templates tests src utils
> }
> ```
> 
> and then run `sty` a lot.

Hmm. Is there a way for `style` to tell me the location in offending file? Output seems pretty minimal.<|||||>if you also run the flake8 command it should just fix it.<|||||>I think I fixed the formatting, as requested. Took a sec to figure that all out...<|||||>@sshleifer  -- any idea what happened with the `black` / code quality changes overnite? I'm very confused. Seems as if the standard changed from yesterday... <|||||>Yes they did, sorry about that. I did some cleanup on this branch.
If you are curious about the style change: I tried to future proof it here https://github.com/huggingface/transformers/pull/6748<|||||># [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=h1) Report
> Merging [#6722](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a75c64d80c76c3dc71f735d9197a4a601847e0cd?el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `68.23%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6722/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6722      +/-   ##
==========================================
- Coverage   78.96%   78.94%   -0.03%     
==========================================
  Files         157      157              
  Lines       28486    28571      +85     
==========================================
+ Hits        22495    22555      +60     
- Misses       5991     6016      +25     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.28% <ø> (ø)` | |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `82.28% <68.23%> (-13.27%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6722/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `83.70% <0.00%> (+0.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=footer). Last update [a75c64d...8958b9f](https://codecov.io/gh/huggingface/transformers/pull/6722?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>Awesome. Thanks @sshleifer. I'll start working more on the other less mature PRs we discussed. And please ping me if/when you write tests or examples for this. Happy to contribute to that as well if you need.<|||||>I've added Adafactor to the docs and slightly changed the style of the docstrings in https://github.com/huggingface/transformers/pull/6765<|||||>Thanks! I'll add a `--adafactor` option lightning_base and trainer in 2 prs.