Could you please implement a Adafactor optimizer? :)
## 🚀 Feature

Could you please implement a Adafactor optimizer? :)

( https://arxiv.org/abs/1804.04235 )

## Motivation

In contrast to Adam it requires much less GPU memory. 
I tried to use the FairSeq implementation for the pytorch-transformers, but I'm no expert and I couldn't get it done.

Could you please do that? :)


## Additional context

<!-- Add any other context or screenshots about the feature request here. -->
What didn't work for you with the fairseq implementation?

It seems pretty self-contained: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py#L65-L213<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>FYI @sshleifer -- I was wrong -- able to train T5-large even batch==1 with FP32, no gradient check-pointing and ADAM. Given that T5 team strongly recommends AdaFactor -- giving it a try, other pieces perhaps being more difficult...