Rectified Adam + LARS
## 🚀 Feature
There has been a lot of buzz around the new Radam and Ralamb (Radam + LARS) optimizers, and I was wondering if it could also be implemented in pytorch-transformers.

## Motivation

It seems to have consistent performance improvements. It also seems to handle different learning rates a lot better. And with LARS it also allows for really large batch sizes without regressing.

## Additional context

https://gist.github.com/redknightlois/c4023d393eb8f92bb44b2ab582d7ec20
https://github.com/mgrankin/over9000
https://twitter.com/jeremyphoward/status/1162118545095852032
https://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b
From what I can tell, Radam makes automatic warmup and LARS is good but requires more calculations per batch. Before implementing it here it's worth do testing to tell if it's a good idea.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>any update?<|||||>For anyone interested in testing, I've created a fork that uses Radam+LARS+LookAhead, https://github.com/i404788/transformers<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
