title: LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
summary: Large context window is a desirable feature in large language models (LLMs).
  However, due to high fine-tuning costs, scarcity of long texts, and
  catastrophic values introduced by new token positions, current extended context
  windows are limited to around 128k tokens. This paper introduces LongRoPE that,
  for the first time, extends the context window of pre-trained LLMs to an
  impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k
  training lengths, while maintaining performance at the original short context
  window. This is achieved by three key innovations: (i) we identify and exploit
  two forms of non-uniformities in positional interpolation through an efficient
  search, providing a better initialization for fine-tuning and enabling an 8x
  extension in non-fine-tuning scenarios; (ii) we introduce a progressive
  extension strategy that first fine-tunes a 256k length LLM and then conducts a
  second positional interpolation on the fine-tuned extended LLM to achieve a
  2048k context window; (iii) we readjust LongRoPE on 8k length to recover the
  short context window performance. Extensive experiments on LLaMA2 and Mistral
  across various tasks demonstrate the effectiveness of our method. Models
  extended via LongRoPE retain the original architecture with minor modifications
  to the positional embedding, and can reuse most pre-existing optimizations.
paper_authors._id: 65d6b88c550694a180f2468f
paper_authors.name: Yiran Ding
paper_authors._id: 65d6b88c550694a180f24690
paper_authors.name: Li Lyna Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-22T22:27:40.320Z
paper_authors._id: 62b0009c72043b05d29492b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
paper_authors.fullname: Li Lyna Zhang
paper_authors.type: user
paper_authors.user: lynazhang
paper_authors._id: 65d6b88c550694a180f24691
paper_authors.name: Chengruidong Zhang
paper_authors._id: 65d6b88c550694a180f24692
paper_authors.name: Yuanyuan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T12:02:12.244Z
paper_authors._id: 64c0ea7fb746fe515454ee3d
paper_authors.avatarUrl: /avatars/1eeb06bcf6e2845715fabca7f88dcd25.svg
paper_authors.fullname: Yuanyuan Xu
paper_authors.type: user
paper_authors.user: Micacle
paper_authors._id: 65d6b88c550694a180f24693
paper_authors.name: Ning Shang
paper_authors._id: 65d6b88c550694a180f24694
paper_authors.name: Jiahang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T12:02:52.271Z
paper_authors._id: 62abdf657b037eafffc48808
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655430982462-noauth.jpeg
paper_authors.fullname: Jiahang Xu
paper_authors.type: user
paper_authors.user: Jiahang
paper_authors._id: 65d6b88c550694a180f24695
paper_authors.name: Fan Yang
paper_authors._id: 65d6b88c550694a180f24696
paper_authors.name: Mao Yang
paper_title: LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
paper_summary: Large context window is a desirable feature in large language models (LLMs).
  However, due to high fine-tuning costs, scarcity of long texts, and
  catastrophic values introduced by new token positions, current extended context
  windows are limited to around 128k tokens. This paper introduces LongRoPE that,
  for the first time, extends the context window of pre-trained LLMs to an
  impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k
  training lengths, while maintaining performance at the original short context
  window. This is achieved by three key innovations: (i) we identify and exploit
  two forms of non-uniformities in positional interpolation through an efficient
  search, providing a better initialization for fine-tuning and enabling an 8x
  extension in non-fine-tuning scenarios; (ii) we introduce a progressive
  extension strategy that first fine-tunes a 256k length LLM and then conducts a
  second positional interpolation on the fine-tuned extended LLM to achieve a
  2048k context window; (iii) we readjust LongRoPE on 8k length to recover the
  short context window performance. Extensive experiments on LLaMA2 and Mistral
  across various tasks demonstrate the effectiveness of our method. Models
  extended via LongRoPE retain the original architecture with minor modifications
  to the positional embedding, and can reuse most pre-existing optimizations.
paper_ai_keywords: large context window large language models (LLMs) fine-tuning long texts token positions extended context windows LongRoPE positional interpolation non-uniformities multivariate distribution initialization fine-tuning steps simulated scenarios 256k training lengths 2048k context window progressive extension strategy second positional interpolation 8k length LLaMA2 Mistral positional embedding pre-existing optimizations