LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
This work elicits LLMs' inherent ability to handle long contexts without
fine-tuning. The limited length of the training sequence during training may
limit the application of Large Language Models (LLMs) on long input sequences
for inference. In this work, we argue that existing LLMs themselves have
inherent capabilities for handling long contexts. Based on this argument, we
suggest extending LLMs' context window by themselves to fully utilize the
inherent ability.We propose Self-Extend to stimulate LLMs' long context
handling potential. The basic idea is to construct bi-level attention
information: the group level and the neighbor level. The two levels are
computed by the original model's self-attention, which means the proposed does
not require any training. With only four lines of code modification, the
proposed method can effortlessly extend existing LLMs' context window without
any fine-tuning. We conduct comprehensive experiments and the results show that
the proposed method can effectively extend existing LLMs' context window's
length.
6594c4caa41c3cbad541cb8d
Hongye Jin
6594c4caa41c3cbad541cb8e
Xiaotian Han
6594c4caa41c3cbad541cb8f
Jingfeng Yang
6594c4caa41c3cbad541cb90
Zhimeng Jiang
6594c4caa41c3cbad541cb91
Zirui Liu
6594c4caa41c3cbad541cb92
Chia-Yuan Chang
claimed_verified
2025-02-27T12:55:38.894Z
645be213f6ce7202d777bde9
/avatars/49aef757f826f6945fac05f3718d2d9b.svg
Scott Chang
user
z7631614
6594c4caa41c3cbad541cb93
Huiyuan Chen
6594c4caa41c3cbad541cb94
Xia Hu
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
This work elicits LLMs' inherent ability to handle long contexts without
fine-tuning. The limited length of the training sequence during training may
limit the application of Large Language Models (LLMs) on long input sequences
for inference. In this work, we argue that existing LLMs themselves have
inherent capabilities for handling long contexts. Based on this argument, we
suggest extending LLMs' context window by themselves to fully utilize the
inherent ability.We propose Self-Extend to stimulate LLMs' long context
handling potential. The basic idea is to construct bi-level attention
information: the group level and the neighbor level. The two levels are
computed by the original model's self-attention, which means the proposed does
not require any training. With only four lines of code modification, the
proposed method can effortlessly extend existing LLMs' context window without
any fine-tuning. We conduct comprehensive experiments and the results show that
the proposed method can effectively extend existing LLMs' context window's
length.
Bi-level attention mechanism extends the context window of LLMs without additional training or fine-tuning.
LLMs self-extend bi-level attention group level neighbor level self-attention