title: YaRN: Efficient Context Window Extension of Large Language Models
summary: Rotary Position Embeddings (RoPE) have been shown to effectively encode
  positional information in transformer-based language models. However, these
  models fail to generalize past the sequence length they were trained on. We
  present YaRN (Yet another RoPE extensioN method), a compute-efficient method to
  extend the context window of such models, requiring 10x less tokens and 2.5x
  less training steps than previous methods. Using YaRN, we show that LLaMA
  models can effectively utilize and extrapolate to context lengths much longer
  than their original pre-training would allow, while also surpassing previous
  the state-of-the-art at context window extension. In addition, we demonstrate
  that YaRN exhibits the capability to extrapolate beyond the limited context of
  a fine-tuning dataset. We publish the checkpoints of Llama 2 7B/13B fine-tuned
  using YaRN with 64k and 128k context windows at
  https://github.com/jquesnelle/yarn
paper_authors._id: 64f5285ab9a80a053124bb5c
paper_authors.name: Bowen Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T19:45:14.164Z
paper_authors._id: 62cf262026c94b143173ef65
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cf262026c94b143173ef65/FFcC-XhG5hMjApSvfk_he.png
paper_authors.fullname: Bowen Peng
paper_authors.type: user
paper_authors.user: bloc97
paper_authors._id: 64f5285ab9a80a053124bb5d
paper_authors.name: Jeffrey Quesnelle
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-04T07:31:10.157Z
paper_authors._id: 630581db99870e13d3e0006f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676652577978-630581db99870e13d3e0006f.jpeg
paper_authors.fullname: Jeffrey Quesnelle
paper_authors.type: user
paper_authors.user: emozilla
paper_authors._id: 64f5285ab9a80a053124bb5e
paper_authors.name: Honglu Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-04T08:09:49.430Z
paper_authors._id: 635bdaf7dc371b8f91029184
paper_authors.avatarUrl: /avatars/7e3769763c0e3f3ff83f3020aef60768.svg
paper_authors.fullname: Honglu
paper_authors.type: user
paper_authors.user: quintic
paper_authors._id: 64f5285ab9a80a053124bb5f
paper_authors.name: Enrico Shippole
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T07:52:28.796Z
paper_authors._id: 6276ba3c2d26ac639e5a2b01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6276ba3c2d26ac639e5a2b01/ULDyT9ijOye7rfXqTWFVG.jpeg
paper_authors.fullname: Enrico Shippole
paper_authors.type: user
paper_authors.user: conceptofmind
paper_title: YaRN: Efficient Context Window Extension of Large Language Models
paper_summary: Rotary Position Embeddings (RoPE) have been shown to effectively encode
  positional information in transformer-based language models. However, these
  models fail to generalize past the sequence length they were trained on. We
  present YaRN (Yet another RoPE extensioN method), a compute-efficient method to
  extend the context window of such models, requiring 10x less tokens and 2.5x
  less training steps than previous methods. Using YaRN, we show that LLaMA
  models can effectively utilize and extrapolate to context lengths much longer
  than their original pre-training would allow, while also surpassing previous
  the state-of-the-art at context window extension. In addition, we demonstrate
  that YaRN exhibits the capability to extrapolate beyond the limited context of
  a fine-tuning dataset. We publish the checkpoints of Llama 2 7B/13B fine-tuned
  using YaRN with 64k and 128k context windows at
  https://github.com/jquesnelle/yarn
paper_ai_summary: YaRN extends the context window of transformer-based language models like LLaMA with improved efficiency and performance.
paper_ai_keywords: Rotary Position Embeddings RoPE compute-efficient context window LLaMA fine-tuning checkpoints