8 types of RoPE
text
8 types of RoPE
new_line
new_line
As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on.
text
As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on.
new_line
new_line
Here are 8 types of RoPE that can be implemented in different cases:
text
Here are 8 types of RoPE that can be implemented in different cases:
new_line
new_line
1. Original RoPE -> 
text
1. Original RoPE -> 
RoFormer: Enhanced Transformer with Rotary Position Embedding (2104.09864)
https://huggingface.co/papers/2104.09864
2104.09864
paper
resource
https://huggingface.co/papers/2104.09864
new_line
Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info.
text
Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info.
new_line
new_line
2. LongRoPE -> 
text
2. LongRoPE -> 
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2402.13753)
https://huggingface.co/papers/2402.13753
2402.13753
paper
resource
https://huggingface.co/papers/2402.13753
new_line
Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search.
text
Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search.
new_line
new_line
3. LongRoPE2 -> 
text
3. LongRoPE2 -> 
LongRoPE2: Near-Lossless LLM Context Window Scaling (2502.20082)
https://huggingface.co/papers/2502.20082
2502.20082
paper
resource
https://huggingface.co/papers/2502.20082
new_line
Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity.
text
Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity.
new_line
new_line
4. Multimodal RoPE (MRoPE) -> 
text
4. Multimodal RoPE (MRoPE) -> 
Qwen2.5-VL Technical Report (2502.13923)
https://huggingface.co/papers/2502.13923
2502.13923
paper
resource
https://huggingface.co/papers/2502.13923
new_line
Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos.
text
Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos.
new_line
new_line
5. Directional RoPE (DRoPE) -> 
text
5. Directional RoPE (DRoPE) -> 
DRoPE: Directional Rotary Position Embedding for Efficient Agent
  Interaction Modeling (2503.15029)
https://huggingface.co/papers/2503.15029
2503.15029
paper
resource
https://huggingface.co/papers/2503.15029
new_line
Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage.
text
Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage.
new_line
new_line
6. VideoRoPE -> 
text
6. VideoRoPE -> 
VideoRoPE: What Makes for Good Video Rotary Position Embedding? (2502.05173)
https://huggingface.co/papers/2502.05173
2502.05173
paper
resource
https://huggingface.co/papers/2502.05173
new_line
Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing.
text
Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing.
new_line
new_line
7. VRoPE -> 
text
7. VRoPE -> 
VRoPE: Rotary Position Embedding for Video Large Language Models (2502.11664)
https://huggingface.co/papers/2502.11664
2502.11664
paper
resource
https://huggingface.co/papers/2502.11664
new_line
An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus.
text
An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus.
new_line
new_line
8. XPos (Extrapolatable Position Embedding) -> 
text
8. XPos (Extrapolatable Position Embedding) -> 
https://huggingface.co/papers/2212.10
https://huggingface.co/papers/2212.10
link
new_line
Introduces an exponential decay factor into the rotation matrix​, improving stability on long sequences.
text
Introduces an exponential decay factor into the rotation matrix​, improving stability on long sequences.
8 types of RoPE

As we always use Transformers, it's helpful to understand RoPE—Rotary Position Embedding. Since token order matters, RoPE encodes it by rotating token embeddings based on their position, so the model knows how to interpret which token comes first, second, and so on.

Here are 8 types of RoPE that can be implemented in different cases:

1. Original RoPE -> https://huggingface.co/papers/2104.09864
Encodes token positions by rotating token embeddings in the complex plane via a position-based rotation matrix, thereby providing the self-attention mechanism with relative positional info.

2. LongRoPE -> https://huggingface.co/papers/2402.13753
Extends the context window of pre-trained LLMs to 2048k tokens, leveraging non-uniformities in positional interpolation with an efficient search.

3. LongRoPE2 -> https://huggingface.co/papers/2502.20082
Extends the effective context window of pre-trained LLMs to the target! length, rescaling RoPE guided by “needle-driven” perplexity.

4. Multimodal RoPE (MRoPE) -> https://huggingface.co/papers/2502.13923
Decomposes positional embedding into 3 components: temporal, height and width, so that positional features are aligned across modalities: text, images and videos.

5. Directional RoPE (DRoPE) -> https://huggingface.co/papers/2503.15029
Adds an identity scalar, improving how angles are handled without extra complexity. It helps balance accuracy, speed, and memory usage.

6. VideoRoPE -> https://huggingface.co/papers/2502.05173
Adapts RoPE for video, featuring 3D structure, low-frequency temporal allocation, diagonal layout, and adjustable spacing.

7. VRoPE -> https://huggingface.co/papers/2502.11664
An another RoPE for video, which restructures positional indices and balances encoding for uniform spatial focus.

8. XPos (Extrapolatable Position Embedding) -> https://huggingface.co/papers/2212.10
Introduces an exponential decay factor into the rotation matrix​, improving stability on long sequences.
👀
tolgacangoz John6666 xi0v mkurman damerajee theainerd busekoseoglu Kseniase xinnn63 Takugen emredeveloper dantezxcd SueK
❤️
tolgacangoz stevenbucaille cs-mshah damerajee Kseniase DrBlue42 dantezxcd
🚀
Cosmosandy Kseniase dantezxcd DrishtiSharma
64838b28c235ef76b63e4999
https://cdn-avatars.huggingface.co/v1/production/uploads/64838b28c235ef76b63e4999/ZhQCYoU3vps71Ag7Jezj6.jpeg
Ksenia Se
Kseniase
user