Why Does the Effective Context Length of LLMs Fall Short?
Advancements in distributed training and efficient attention mechanisms have
significantly expanded the context window sizes of large language models
(LLMs). However, recent work reveals that the effective context lengths of
open-source LLMs often fall short, typically not exceeding half of their
training lengths. In this work, we attribute this limitation to the left-skewed
frequency distribution of relative positions formed in LLMs pretraining and
post-training stages, which impedes their ability to effectively gather distant
information. To address this challenge, we introduce ShifTed Rotray position
embeddING (STRING). STRING shifts well-trained positions to overwrite the
original ineffective positions during inference, enhancing performance within
their existing training lengths. Experimental results show that without
additional training, STRING dramatically improves the performance of the latest
large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on
popular long-context benchmarks RULER and InfiniteBench, establishing new
state-of-the-art results for open-source LLMs. Compared to commercial models,
Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and
clearly surpasses Claude 2 and Kimi-chat.
671aff8b4073e48ef6aa25d8
Chenxin An
671aff8b4073e48ef6aa25d9
Jun Zhang
671aff8b4073e48ef6aa25da
Ming Zhong
671aff8b4073e48ef6aa25db
Lei Li
claimed_verified
2024-12-09T14:48:25.361Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
671aff8b4073e48ef6aa25dc
Shansan Gong
671aff8b4073e48ef6aa25dd
Yao Luo
671aff8b4073e48ef6aa25de
Jingjing Xu
671aff8b4073e48ef6aa25df
Lingpeng Kong
Why Does the Effective Context Length of LLMs Fall Short?
Advancements in distributed training and efficient attention mechanisms have
significantly expanded the context window sizes of large language models
(LLMs). However, recent work reveals that the effective context lengths of
open-source LLMs often fall short, typically not exceeding half of their
training lengths. In this work, we attribute this limitation to the left-skewed
frequency distribution of relative positions formed in LLMs pretraining and
post-training stages, which impedes their ability to effectively gather distant
information. To address this challenge, we introduce ShifTed Rotray position
embeddING (STRING). STRING shifts well-trained positions to overwrite the
original ineffective positions during inference, enhancing performance within
their existing training lengths. Experimental results show that without
additional training, STRING dramatically improves the performance of the latest
large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on
popular long-context benchmarks RULER and InfiniteBench, establishing new
state-of-the-art results for open-source LLMs. Compared to commercial models,
Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and
clearly surpasses Claude 2 and Kimi-chat.
STRING, a position embedding technique that shifts well-trained positions during inference, enhances the performance of large language models without additional training, surpassing both open-source and commercial models on long-context benchmarks.
distributed training efficient attention mechanisms large language models context window sizes effective context lengths relative positions position embeddING ShifTed Rotray position embeddING STRING RULER InfiniteBench GPT-4-128K Claude 2 Kimi-chat