title: Extending Context Window of Large Language Models via Positional
    Interpolation
summary: We present Position Interpolation (PI) that extends the context window sizes
  of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal
  fine-tuning (within 1000 steps), while demonstrating strong empirical results
  on various tasks that require long context, including passkey retrieval,
  language modeling, and long document summarization from LLaMA 7B to 65B.
  Meanwhile, the extended model by Position Interpolation preserve quality
  relatively well on tasks within its original context window. To achieve this
  goal, Position Interpolation linearly down-scales the input position indices to
  match the original context window size, rather than extrapolating beyond the
  trained context length which may lead to catastrophically high attention scores
  that completely ruin the self-attention mechanism. Our theoretical study shows
  that the upper bound of interpolation is at least sim 600 times smaller
  than that of extrapolation, further demonstrating its stability. Models
  extended via Position Interpolation retain its original architecture and can
  reuse most pre-existing optimization and infrastructure.
paper_authors._id: 649b963fb45a4d4bd3ae2deb
paper_authors.name: Shouyuan Chen
paper_authors._id: 649b963fb45a4d4bd3ae2dec
paper_authors.name: Sherman Wong
paper_authors._id: 649b963fb45a4d4bd3ae2ded
paper_authors.name: Liangjian Chen
paper_authors._id: 649b963fb45a4d4bd3ae2dee
paper_authors.name: Yuandong Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:16:00.161Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_title: Extending Context Window of Large Language Models via Positional
    Interpolation
paper_summary: We present Position Interpolation (PI) that extends the context window sizes
  of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal
  fine-tuning (within 1000 steps), while demonstrating strong empirical results
  on various tasks that require long context, including passkey retrieval,
  language modeling, and long document summarization from LLaMA 7B to 65B.
  Meanwhile, the extended model by Position Interpolation preserve quality
  relatively well on tasks within its original context window. To achieve this
  goal, Position Interpolation linearly down-scales the input position indices to
  match the original context window size, rather than extrapolating beyond the
  trained context length which may lead to catastrophically high attention scores
  that completely ruin the self-attention mechanism. Our theoretical study shows
  that the upper bound of interpolation is at least sim 600 times smaller
  than that of extrapolation, further demonstrating its stability. Models
  extended via Position Interpolation retain its original architecture and can
  reuse most pre-existing optimization and infrastructure.
paper_ai_keywords: RoPE-based pretrained LLMs Position Interpolation (PI) context window sizes fine-tuning passkey retrieval language modeling long document summarization position indices self-attention catastrophically high attention scores upper bound of interpolation extrapolation stability pre-existing optimization infrastructure