Adding Stochastic Weight Averaging to transformer optimizers
# 🚀 Feature request

I would like to train my models with SWA optimizer. According to this [paper](https://arxiv.org/pdf/1803.05407.pdf), SWA leads to better models and wider optima. 

## Motivation

As humans, we are all willing to get better results :) . I think adding this feature will lead to better models without costing more and it may be easy to implement.

In the meantime, I have started to work on adding SWA to huggingface. After doing some experiments, If I get better results, I will create a PR. You can check my work  from [here](https://github.com/hasansalimkanmaz/transformers/tree/add-SWA-optimizer)

Any feedback will be appreciated.<|||||>Based on my custom experiments, I couldn't produce better results with SWA. So I am closing this issue. My implementation for SWA is so custom and I didn't go through all tests. So, I will not create a PR due to the lack of benefits. 