YaRN: Efficient Context Window Extension of Large Language Models
Rotary Position Embeddings (RoPE) have been shown to effectively encode
positional information in transformer-based language models. However, these
models fail to generalize past the sequence length they were trained on. We
present YaRN (Yet another RoPE extensioN method), a compute-efficient method to
extend the context window of such models, requiring 10x less tokens and 2.5x
less training steps than previous methods. Using YaRN, we show that LLaMA
models can effectively utilize and extrapolate to context lengths much longer
than their original pre-training would allow, while also surpassing previous
the state-of-the-art at context window extension. In addition, we demonstrate
that YaRN exhibits the capability to extrapolate beyond the limited context of
a fine-tuning dataset. We publish the checkpoints of Llama 2 7B/13B fine-tuned
using YaRN with 64k and 128k context windows at
https://github.com/jquesnelle/yarn
64f5285ab9a80a053124bb5c
Bowen Peng
admin_assigned
2023-09-04T19:45:14.164Z
62cf262026c94b143173ef65
https://cdn-avatars.huggingface.co/v1/production/uploads/62cf262026c94b143173ef65/FFcC-XhG5hMjApSvfk_he.png
Bowen Peng
user
bloc97
64f5285ab9a80a053124bb5d
Jeffrey Quesnelle
claimed_verified
2023-09-04T07:31:10.157Z
630581db99870e13d3e0006f
https://cdn-avatars.huggingface.co/v1/production/uploads/1676652577978-630581db99870e13d3e0006f.jpeg
Jeffrey Quesnelle
user
emozilla
64f5285ab9a80a053124bb5e
Honglu Fan
claimed_verified
2023-09-04T08:09:49.430Z
635bdaf7dc371b8f91029184
/avatars/7e3769763c0e3f3ff83f3020aef60768.svg
Honglu
user
quintic
64f5285ab9a80a053124bb5f
Enrico Shippole
admin_assigned
2023-09-04T07:52:28.796Z
6276ba3c2d26ac639e5a2b01
https://cdn-avatars.huggingface.co/v1/production/uploads/6276ba3c2d26ac639e5a2b01/ULDyT9ijOye7rfXqTWFVG.jpeg
Enrico Shippole
user
conceptofmind
YaRN: Efficient Context Window Extension of Large Language Models
Rotary Position Embeddings (RoPE) have been shown to effectively encode
positional information in transformer-based language models. However, these
models fail to generalize past the sequence length they were trained on. We
present YaRN (Yet another RoPE extensioN method), a compute-efficient method to
extend the context window of such models, requiring 10x less tokens and 2.5x
less training steps than previous methods. Using YaRN, we show that LLaMA
models can effectively utilize and extrapolate to context lengths much longer
than their original pre-training would allow, while also surpassing previous
the state-of-the-art at context window extension. In addition, we demonstrate
that YaRN exhibits the capability to extrapolate beyond the limited context of
a fine-tuning dataset. We publish the checkpoints of Llama 2 7B/13B fine-tuned
using YaRN with 64k and 128k context windows at
https://github.com/jquesnelle/yarn
YaRN extends the context window of transformer-based language models like LLaMA with improved efficiency and performance.
Rotary Position Embeddings RoPE compute-efficient context window LLaMA fine-tuning checkpoints