Support fairseq encoder-normalize-before in RoBERTa
### Feature request

Support Fairseq's `--encoder-normalize-before` variation of transformer-based models, (particularly RoBERTa, not sure how many it applies to) that "apply layernorm before each encoder block". See: https://fairseq.readthedocs.io/en/v0.7.0/models.html

### Motivation

There currently exist unofficial hacks of Huggingface's transformer model at https://github.com/princeton-nlp/dinkytrain which uses `--encoder-normalize-before` with fairseq and then applies a series of hacks to make it work with Huggingface's transformer library. See in particular [their custom version of RoBERTa](https://github.com/princeton-nlp/DinkyTrain/blob/main/huggingface/modeling_roberta_prelayernorm.py) which is a hack of Huggingface's RoBERTa implementation and depends on several internal components.

The weights for these hacked models are currently distributed on https://huggingface.co/princeton-nlp but are not actually compatible with the official transformer library. (https://arxiv.org/abs/2202.08005 is the related paper).

It would be great to implement feature parity for `--encoder-normalize-before` in Huggingface's transformer such that these hacks can be prevented.

Note that I have not affiliation with fairseq, the https://github.com/princeton-nlp/DinkyTrain authors, nor the https://arxiv.org/abs/2202.08005 authors.

### Your contribution

I'm happy to contribute a PR for RoBERTa that adds feature parity for the `--encoder-normalize-before` fairseq flag.

Note that I have no affiliation with fairseq nor the https://github.com/princeton-nlp/DinkyTrain authors.
Transformers is very opinionated in that regard and is not a building block library like fairseq. The only way to add support for those models in Transformers would be to add a new modeling file which adapts the code of RoBERTa to only include the code path corresponding to `--encoder-normalize-before`, as putting both in the same modeling file hurts readability.

You can learn more about our philosophy in this regard in [this blog post](https://huggingface.co/blog/transformers-design-philosophy) and if you're interested in making a PR to add this new model, we're looking forward to it!
<|||||>Thanks, I will look at adding the new model. Do you have a policy regarding its name? In this case, the authors did not give it a dedicated name as they see it as just RoBERTa.<|||||>Please see the implementation provided in https://github.com/huggingface/transformers/pull/20305<|||||>That's pretty cool! We had this kind of "problem" when adding support for XLM-R XL models:

https://github.com/huggingface/transformers/pull/12082#issue-665786049<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.