title: Why Does the Effective Context Length of LLMs Fall Short?
summary: Advancements in distributed training and efficient attention mechanisms have
  significantly expanded the context window sizes of large language models
  (LLMs). However, recent work reveals that the effective context lengths of
  open-source LLMs often fall short, typically not exceeding half of their
  training lengths. In this work, we attribute this limitation to the left-skewed
  frequency distribution of relative positions formed in LLMs pretraining and
  post-training stages, which impedes their ability to effectively gather distant
  information. To address this challenge, we introduce ShifTed Rotray position
  embeddING (STRING). STRING shifts well-trained positions to overwrite the
  original ineffective positions during inference, enhancing performance within
  their existing training lengths. Experimental results show that without
  additional training, STRING dramatically improves the performance of the latest
  large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on
  popular long-context benchmarks RULER and InfiniteBench, establishing new
  state-of-the-art results for open-source LLMs. Compared to commercial models,
  Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and
  clearly surpasses Claude 2 and Kimi-chat.
paper_authors._id: 671aff8b4073e48ef6aa25d8
paper_authors.name: Chenxin An
paper_authors._id: 671aff8b4073e48ef6aa25d9
paper_authors.name: Jun Zhang
paper_authors._id: 671aff8b4073e48ef6aa25da
paper_authors.name: Ming Zhong
paper_authors._id: 671aff8b4073e48ef6aa25db
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:48:25.361Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 671aff8b4073e48ef6aa25dc
paper_authors.name: Shansan Gong
paper_authors._id: 671aff8b4073e48ef6aa25dd
paper_authors.name: Yao Luo
paper_authors._id: 671aff8b4073e48ef6aa25de
paper_authors.name: Jingjing Xu
paper_authors._id: 671aff8b4073e48ef6aa25df
paper_authors.name: Lingpeng Kong
paper_title: Why Does the Effective Context Length of LLMs Fall Short?
paper_summary: Advancements in distributed training and efficient attention mechanisms have
  significantly expanded the context window sizes of large language models
  (LLMs). However, recent work reveals that the effective context lengths of
  open-source LLMs often fall short, typically not exceeding half of their
  training lengths. In this work, we attribute this limitation to the left-skewed
  frequency distribution of relative positions formed in LLMs pretraining and
  post-training stages, which impedes their ability to effectively gather distant
  information. To address this challenge, we introduce ShifTed Rotray position
  embeddING (STRING). STRING shifts well-trained positions to overwrite the
  original ineffective positions during inference, enhancing performance within
  their existing training lengths. Experimental results show that without
  additional training, STRING dramatically improves the performance of the latest
  large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on
  popular long-context benchmarks RULER and InfiniteBench, establishing new
  state-of-the-art results for open-source LLMs. Compared to commercial models,
  Llama 3.1 70B with \method even achieves better performance than GPT-4-128K and
  clearly surpasses Claude 2 and Kimi-chat.
paper_ai_keywords: distributed training efficient attention mechanisms context window sizes large language models (LLMs) effective context lengths left-skewed frequency distribution relative positions pretraining post-training ShifTed Rotray position embeddING (STRING) long-context benchmarks RULER InfiniteBench Llama3.1 70B Qwen2 72B GPT-4-128K Claude 2 Kimi-chat