Convert Pre-LN Transformers into equivalent Pre-RMSNorm Transformers to accelerate inference and training
### Feature request

LayerNorm and RMSNorm are the top two normalization methods in Transformers. We unify them in Pre-Normalization Transformers in our paper https://arxiv.org/abs/2305.14858. The arithmetic equivalence allows us to convert Pre-LN Transformers into Pre-RMSNorm models without impact on the model functionality. Since RMSNorm offers superior efficiency compared to LayerNorm, our method enables faster equivalent inference and training for any Pre-LN Transformers, e.g., GPT, ViT.

Our implementation is at https://github.com/ZixuanJiang/pre-rmsnorm-transformer. As the first step, we can start by accelerating the deployment of the existing Pre-LN Transformers.

### Motivation

Related GitHub issue: https://github.com/pytorch/pytorch/issues/72643#issue

### Your contribution

We have provided our reference implementation at https://github.com/ZixuanJiang/pre-rmsnorm-transformer. We are open to submitting a related PR in the future.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.