title: LongRoPE2: Near-Lossless LLM Context Window Scaling
summary: LongRoPE2 is a novel approach that extends the effective context window of
  pre-trained large language models (LLMs) to the target length, while preserving
  the performance on the original shorter context window. This is achieved by
  three contributions: (1) a hypothesis that insufficient training in higher RoPE
  dimensions contributes to the persistent out-of-distribution (OOD) issues
  observed in existing methods; (2) an effective RoPE rescaling algorithm that
  adopts evolutionary search guided by "needle-driven" perplexity to address the
  insufficient training problem; (3) a mixed context window training approach
  that fine-tunes model weights to adopt rescaled RoPE for long-context sequences
  while preserving the short-context performance with the original RoPE.
  Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks
  validate the hypothesis and demonstrate the effectiveness of LongRoPE2.
  Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context
  length while retaining over 98.5% of short-context performance, using only 10B
  tokens -- 80x fewer than Meta's approach, which fails to reach the target
  effective context length. Code will be available at
  https://github.com/microsoft/LongRoPE.
paper_authors._id: 67c12b6d25c74ee5b6e2ce8e
paper_authors.name: Ning Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:28:26.117Z
paper_authors._id: 632bc663eafe8eca5e9bfdbc
paper_authors.avatarUrl: /avatars/787553c73e9a96adc5219e67acd29c00.svg
paper_authors.fullname: Ning Shang
paper_authors.type: user
paper_authors.user: J-shang
paper_authors._id: 67c12b6d25c74ee5b6e2ce8f
paper_authors.name: Li Lyna Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:26:58.131Z
paper_authors._id: 62b0009c72043b05d29492b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
paper_authors.fullname: Li Lyna Zhang
paper_authors.type: user
paper_authors.user: lynazhang
paper_authors._id: 67c12b6d25c74ee5b6e2ce90
paper_authors.name: Siyuan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:14:20.687Z
paper_authors._id: 6495b0b844bc2e9ce6cc849b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/j6aucl_tefMHwtD-bdUAw.jpeg
paper_authors.fullname: Siyuan Wang
paper_authors.type: user
paper_authors.user: OldKingMeister
paper_authors._id: 67c12b6d25c74ee5b6e2ce91
paper_authors.name: Gaokai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:28:05.040Z
paper_authors._id: 65efe691ccef3501d586bb62
paper_authors.avatarUrl: /avatars/c4716c532754b487359e77e43afe09bc.svg
paper_authors.fullname: Gaokai Zhang
paper_authors.type: user
paper_authors.user: gaokaiz2
paper_authors._id: 67c12b6d25c74ee5b6e2ce92
paper_authors.name: Gilsinia Lopez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:27:52.164Z
paper_authors._id: 60c790f1accf7da31ed8240d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60c790f1accf7da31ed8240d/YDohCmgf9OUeWqZIs3Thh.jpeg
paper_authors.fullname: Gilsinia Lopez
paper_authors.type: user
paper_authors.user: lgg
paper_authors._id: 67c12b6d25c74ee5b6e2ce93
paper_authors.name: Fan Yang
paper_authors._id: 67c12b6d25c74ee5b6e2ce94
paper_authors.name: Weizhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:27:17.122Z
paper_authors._id: 64da876370446182be5b608d
paper_authors.avatarUrl: /avatars/e412fdc71404ecdf638e416846e3ebfb.svg
paper_authors.fullname: Weizhu Chen
paper_authors.type: user
paper_authors.user: chenweizhu
paper_authors._id: 67c12b6d25c74ee5b6e2ce95
paper_authors.name: Mao Yang
paper_title: LongRoPE2: Near-Lossless LLM Context Window Scaling
paper_summary: LongRoPE2 is a novel approach that extends the effective context window of
  pre-trained large language models (LLMs) to the target length, while preserving
  the performance on the original shorter context window. This is achieved by
  three contributions: (1) a hypothesis that insufficient training in higher RoPE
  dimensions contributes to the persistent out-of-distribution (OOD) issues
  observed in existing methods; (2) an effective RoPE rescaling algorithm that
  adopts evolutionary search guided by "needle-driven" perplexity to address the
  insufficient training problem; (3) a mixed context window training approach
  that fine-tunes model weights to adopt rescaled RoPE for long-context sequences
  while preserving the short-context performance with the original RoPE.
  Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks
  validate the hypothesis and demonstrate the effectiveness of LongRoPE2.
  Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context
  length while retaining over 98.5% of short-context performance, using only 10B
  tokens -- 80x fewer than Meta's approach, which fails to reach the target
  effective context length. Code will be available at
  https://github.com/microsoft/LongRoPE.
paper_ai_keywords: pre-trained large language models (LLMs) effective context window hypothesis insufficient training higher RoPE dimensions out-of-distribution (OOD) issues RoPE rescaling algorithm evolutionary search needle-driven perplexity mixed context window training approach fine-tuning rescaled RoPE short-context performance original RoPE LLaMA3-8B Phi3-mini-3.8B benchmarks effective context length