title: LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
summary: This work elicits LLMs' inherent ability to handle long contexts without
  fine-tuning. The limited length of the training sequence during training may
  limit the application of Large Language Models (LLMs) on long input sequences
  for inference. In this work, we argue that existing LLMs themselves have
  inherent capabilities for handling long contexts. Based on this argument, we
  suggest extending LLMs' context window by themselves to fully utilize the
  inherent ability.We propose Self-Extend to stimulate LLMs' long context
  handling potential. The basic idea is to construct bi-level attention
  information: the group level and the neighbor level. The two levels are
  computed by the original model's self-attention, which means the proposed does
  not require any training. With only four lines of code modification, the
  proposed method can effortlessly extend existing LLMs' context window without
  any fine-tuning. We conduct comprehensive experiments and the results show that
  the proposed method can effectively extend existing LLMs' context window's
  length.
paper_authors._id: 6594c4caa41c3cbad541cb8d
paper_authors.name: Hongye Jin
paper_authors._id: 6594c4caa41c3cbad541cb8e
paper_authors.name: Xiaotian Han
paper_authors._id: 6594c4caa41c3cbad541cb8f
paper_authors.name: Jingfeng Yang
paper_authors._id: 6594c4caa41c3cbad541cb90
paper_authors.name: Zhimeng Jiang
paper_authors._id: 6594c4caa41c3cbad541cb91
paper_authors.name: Zirui Liu
paper_authors._id: 6594c4caa41c3cbad541cb92
paper_authors.name: Chia-Yuan Chang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T12:55:38.894Z
paper_authors._id: 645be213f6ce7202d777bde9
paper_authors.avatarUrl: /avatars/49aef757f826f6945fac05f3718d2d9b.svg
paper_authors.fullname: Scott Chang
paper_authors.type: user
paper_authors.user: z7631614
paper_authors._id: 6594c4caa41c3cbad541cb93
paper_authors.name: Huiyuan Chen
paper_authors._id: 6594c4caa41c3cbad541cb94
paper_authors.name: Xia Hu
paper_title: LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
paper_summary: This work elicits LLMs' inherent ability to handle long contexts without
  fine-tuning. The limited length of the training sequence during training may
  limit the application of Large Language Models (LLMs) on long input sequences
  for inference. In this work, we argue that existing LLMs themselves have
  inherent capabilities for handling long contexts. Based on this argument, we
  suggest extending LLMs' context window by themselves to fully utilize the
  inherent ability.We propose Self-Extend to stimulate LLMs' long context
  handling potential. The basic idea is to construct bi-level attention
  information: the group level and the neighbor level. The two levels are
  computed by the original model's self-attention, which means the proposed does
  not require any training. With only four lines of code modification, the
  proposed method can effortlessly extend existing LLMs' context window without
  any fine-tuning. We conduct comprehensive experiments and the results show that
  the proposed method can effectively extend existing LLMs' context window's
  length.
paper_ai_summary: Bi-level attention mechanism extends the context window of LLMs without additional training or fine-tuning.
paper_ai_keywords: LLMs self-extend bi-level attention group level neighbor level self-attention