title: Extending LLMs' Context Window with 100 Samples
summary: Large Language Models (LLMs) are known to have limited extrapolation ability
  beyond their pre-trained context window, constraining their application in
  downstream tasks with lengthy inputs. Recent studies have sought to extend
  LLMs' context window by modifying rotary position embedding (RoPE), a popular
  position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and
  GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are
  resource-intensive and lack comparative experiments to assess their
  applicability. In this work, we identify the inherent need for LLMs' attention
  entropy (i.e. the information entropy of attention scores) to maintain
  stability and introduce a novel extension to RoPE which combines adjusting
  RoPE's base frequency and scaling the attention logits to help LLMs efficiently
  adapt to a larger context window. We validate the superiority of our method in
  both fine-tuning performance and robustness across different context window
  sizes on various context-demanding tasks. Notably, our method extends the
  context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6
  training steps, showcasing extraordinary efficiency. Finally, we also explore
  how data compositions and training curricula affect context window extension
  for specific downstream tasks, suggesting fine-tuning LLMs with lengthy
  conversations as a good starting point. We release our code and SFT data at
  https://github.com/GAIR-NLP/Entropy-ABF.
paper_authors._id: 65a77c2921943858bde10da1
paper_authors.name: Yikai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:44:57.954Z
paper_authors._id: 63e8b792ca4fc7d30de6975b
paper_authors.avatarUrl: /avatars/57237f54d61d479df15209497a3f531e.svg
paper_authors.fullname: Yikai Zhang
paper_authors.type: user
paper_authors.user: Arist12
paper_authors._id: 65a77c2921943858bde10da2
paper_authors.name: Junlong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:46:21.036Z
paper_authors._id: 621e40ac944c7e36aaec2369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
paper_authors.fullname: Junlong Li
paper_authors.type: user
paper_authors.user: lockon
paper_authors._id: 65a77c2921943858bde10da3
paper_authors.name: Pengfei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:46:04.774Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: Extending LLMs' Context Window with 100 Samples
paper_summary: Large Language Models (LLMs) are known to have limited extrapolation ability
  beyond their pre-trained context window, constraining their application in
  downstream tasks with lengthy inputs. Recent studies have sought to extend
  LLMs' context window by modifying rotary position embedding (RoPE), a popular
  position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and
  GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are
  resource-intensive and lack comparative experiments to assess their
  applicability. In this work, we identify the inherent need for LLMs' attention
  entropy (i.e. the information entropy of attention scores) to maintain
  stability and introduce a novel extension to RoPE which combines adjusting
  RoPE's base frequency and scaling the attention logits to help LLMs efficiently
  adapt to a larger context window. We validate the superiority of our method in
  both fine-tuning performance and robustness across different context window
  sizes on various context-demanding tasks. Notably, our method extends the
  context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6
  training steps, showcasing extraordinary efficiency. Finally, we also explore
  how data compositions and training curricula affect context window extension
  for specific downstream tasks, suggesting fine-tuning LLMs with lengthy
  conversations as a good starting point. We release our code and SFT data at
  https://github.com/GAIR-NLP/Entropy-ABF.
paper_ai_keywords: rotary position embedding (RoPE) Position Interpolation (PI) YaRN attention entropy information entropy attention scores attention logits context window fine-tuning performance robustness context-demanding tasks LLaMA-2-7B-Chat context window extension data compositions training curricula lengthy conversations