Adding support for scaling rotary position embeddings
### Feature request

Hello,

I would like if possible for Rotary Position Embedding scaling factors to be usable in the library. Currently this can only be done by monkey-patching the library.

Namely, it requires modifying the:
- `max_position_embeddings`: This can already be done via the model's config class or `config.json`
- `position_scale`: This variable doesn't exist currently, and there is no way to incorporate this effect at the moment without monkey-patching the existing `LlamaRotaryEmbeddings` class. (I'd also like to not step over toes of a possible future XPos implementation which also uses it's own scale for different purposes)

### Motivation

Recently I demonstrated it is possible to drastically reduce training compute when fine-tuning pre-trained RoPE models with an adjusted scaling factor for the purpose of extending the context length of the model. This has the effect of interpolating the position embeddings making it easier to fine-tune the model using in-distribution positions as opposed to out-of-distribution positions typically used via pure extrapolation. There is an extended write-up with motivations here https://kaiokendev.github.io/context as well as the code I used (for the 8K example) can be found here https://huggingface.co/kaiokendev/superhot-13b-8k-no-rlhf-test/blob/main/llama_rope_scaled_monkey_patch.py

Some existing discussions and benchmarks can be found here: https://github.com/ggerganov/llama.cpp/discussions/1965

Several models currently use this scaling feature, but they will not produce coherent output unless the scale is applied correctly during inference (scale is a hyperparameter):
- https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test
- https://huggingface.co/Peeepy/Airoboros-13b-SuperHOT-8k
- https://huggingface.co/emozilla/open_llama_7b-scaled

EDIT: Meta has recently written a paper about it: https://arxiv.org/abs/2306.15595

### Your contribution

I would love to help in any way possible. While the basic implementation would be easy, I'm not sure what the best way could be for adding this modification (such as if users want to used a fixed scale versus having it dynamically applied based on the input sequence length)
any updates on this?<|||||>@lucasjinreal Yes, it is now an official part of the library
https://github.com/huggingface/transformers/pull/24653#issuecomment-1635324005

So I will close this issue<|||||>Here are the docs btw
https://huggingface.co/docs/transformers/main/en/model_doc/llama#transformers.LlamaConfig.rope_scaling<|||||>@kaiokendev So it actually support just same thing like longchat?
BTW, how to adopt it to Baichuan model properly?<|||||>Yes, for LongChat specifically, you would use "linear" method with factor of 8.

For Baichuan model you would not use this, as Baichuan uses ALiBi, not RoPE<|||||>@kaiokendev I was not aware of this issue, my bad 🙈 

Suggestion: tag someone when opening an issue; sometimes things fly under our radar