Resonance RoPE: Improving Context Length Generalization of Large
  Language Models
This paper addresses the challenge of train-short-test-long (TSTL) scenarios
in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
where models pre-trained on shorter sequences face difficulty with
out-of-distribution (OOD) token positions in longer sequences. We introduce
Resonance RoPE, a novel approach designed to narrow the generalization gap in
TSTL scenarios by refining the interpolation of RoPE features for OOD
positions, significantly improving the model performance without additional
online computational costs. Furthermore, we present PosGen, a new synthetic
benchmark specifically designed for fine-grained behavior analysis in TSTL
scenarios, aiming to isolate the constantly increasing difficulty of token
generation on long contexts from the challenges of recognizing new token
positions. Our experiments on synthetic tasks show that after applying
Resonance RoPE, Transformers recognize OOD position better and more robustly.
Our extensive LLM experiments also show superior performance after applying
Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
both upstream language modeling tasks and a variety of downstream long-text
applications.
65e5591aad42606f6f3ecaab
Suyuchen Wang
admin_assigned
2024-03-04T10:16:41.890Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/XuQKiRxyq9mncX5QrVLy7.jpeg
Suyuchen Wang
user
sheryc
65e5591aad42606f6f3ecaac
Ivan Kobyzev
65e5591aad42606f6f3ecaad
Peng Lu
65e5591aad42606f6f3ecaae
Mehdi Rezagholizadeh
admin_assigned
2024-03-04T10:15:46.753Z
6348322f0744c555b6292073
https://cdn-avatars.huggingface.co/v1/production/uploads/1665675864273-6348322f0744c555b6292073.jpeg
Mehdi Rezagholizadeh
user
mrgzadeh
65e5591aad42606f6f3ecaaf
Bang Liu
admin_assigned
2024-03-04T10:16:22.275Z
654a97282d2fcd6bf2851173
https://cdn-avatars.huggingface.co/v1/production/uploads/654a97282d2fcd6bf2851173/9zXf940gr4WNt4e-oOt4k.png
Bang Liu
user
Bang-UdeM-Mila
Resonance RoPE: Improving Context Length Generalization of Large
  Language Models
This paper addresses the challenge of train-short-test-long (TSTL) scenarios
in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
where models pre-trained on shorter sequences face difficulty with
out-of-distribution (OOD) token positions in longer sequences. We introduce
Resonance RoPE, a novel approach designed to narrow the generalization gap in
TSTL scenarios by refining the interpolation of RoPE features for OOD
positions, significantly improving the model performance without additional
online computational costs. Furthermore, we present PosGen, a new synthetic
benchmark specifically designed for fine-grained behavior analysis in TSTL
scenarios, aiming to isolate the constantly increasing difficulty of token
generation on long contexts from the challenges of recognizing new token
positions. Our experiments on synthetic tasks show that after applying
Resonance RoPE, Transformers recognize OOD position better and more robustly.
Our extensive LLM experiments also show superior performance after applying
Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
both upstream language modeling tasks and a variety of downstream long-text
applications.
Resonance RoPE enhances LLM generalization in TSTL scenarios by improving RoPE interpolation for OOD token positions, outperforming existing methods on both synthetic and real-world tasks.
Rotary Position Embedding Resonance RoPE out-of-distribution TSTL generalization gap interpolation PosGen YaRN Transformer language modeling long-text applications