title: Resonance RoPE: Improving Context Length Generalization of Large
    Language Models
summary: This paper addresses the challenge of train-short-test-long (TSTL) scenarios
  in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
  where models pre-trained on shorter sequences face difficulty with
  out-of-distribution (OOD) token positions in longer sequences. We introduce
  Resonance RoPE, a novel approach designed to narrow the generalization gap in
  TSTL scenarios by refining the interpolation of RoPE features for OOD
  positions, significantly improving the model performance without additional
  online computational costs. Furthermore, we present PosGen, a new synthetic
  benchmark specifically designed for fine-grained behavior analysis in TSTL
  scenarios, aiming to isolate the constantly increasing difficulty of token
  generation on long contexts from the challenges of recognizing new token
  positions. Our experiments on synthetic tasks show that after applying
  Resonance RoPE, Transformers recognize OOD position better and more robustly.
  Our extensive LLM experiments also show superior performance after applying
  Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
  both upstream language modeling tasks and a variety of downstream long-text
  applications.
paper_authors._id: 65e5591aad42606f6f3ecaab
paper_authors.name: Suyuchen Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-04T10:16:41.890Z
paper_authors._id: 62bb1e0f3ff437e49a3088e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/bcUQmH8tKfI6DIWH9IcYp.jpeg
paper_authors.fullname: Suyuchen Wang
paper_authors.type: user
paper_authors.user: sheryc
paper_authors._id: 65e5591aad42606f6f3ecaac
paper_authors.name: Ivan Kobyzev
paper_authors._id: 65e5591aad42606f6f3ecaad
paper_authors.name: Peng Lu
paper_authors._id: 65e5591aad42606f6f3ecaae
paper_authors.name: Mehdi Rezagholizadeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-04T10:15:46.753Z
paper_authors._id: 6348322f0744c555b6292073
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665675864273-6348322f0744c555b6292073.jpeg
paper_authors.fullname: Mehdi Rezagholizadeh
paper_authors.type: user
paper_authors.user: mrgzadeh
paper_authors._id: 65e5591aad42606f6f3ecaaf
paper_authors.name: Bang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-04T10:16:22.275Z
paper_authors._id: 654a97282d2fcd6bf2851173
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654a97282d2fcd6bf2851173/9zXf940gr4WNt4e-oOt4k.png
paper_authors.fullname: Bang Liu
paper_authors.type: user
paper_authors.user: Bang-UdeM-Mila
paper_title: Resonance RoPE: Improving Context Length Generalization of Large
    Language Models
paper_summary: This paper addresses the challenge of train-short-test-long (TSTL) scenarios
  in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE),
  where models pre-trained on shorter sequences face difficulty with
  out-of-distribution (OOD) token positions in longer sequences. We introduce
  Resonance RoPE, a novel approach designed to narrow the generalization gap in
  TSTL scenarios by refining the interpolation of RoPE features for OOD
  positions, significantly improving the model performance without additional
  online computational costs. Furthermore, we present PosGen, a new synthetic
  benchmark specifically designed for fine-grained behavior analysis in TSTL
  scenarios, aiming to isolate the constantly increasing difficulty of token
  generation on long contexts from the challenges of recognizing new token
  positions. Our experiments on synthetic tasks show that after applying
  Resonance RoPE, Transformers recognize OOD position better and more robustly.
  Our extensive LLM experiments also show superior performance after applying
  Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on
  both upstream language modeling tasks and a variety of downstream long-text
  applications.
paper_ai_keywords: Rotary Position Embedding (RoPE) Resonance RoPE out-of-distribution (OOD) train-short-test-long (TSTL) PosGen fine-grained behavior analysis token generation downstream long-text applications language modeling tasks YaRN