Focused Transformer: Contrastive Training for Context Scaling
Large language models have an exceptional capability to incorporate new
information in a contextual manner. However, the full potential of such an
approach is often restrained due to a limitation in the effective context
length. One solution to this issue is to endow an attention layer with access
to an external memory, which comprises of (key, value) pairs. Yet, as the
number of documents increases, the proportion of relevant keys to irrelevant
ones decreases, leading the model to focus more on the irrelevant keys. We
identify a significant challenge, dubbed the distraction issue, where keys
linked to different semantic values might overlap, making them hard to
distinguish. To tackle this problem, we introduce the Focused Transformer
(FoT), a technique that employs a training process inspired by contrastive
learning. This novel approach enhances the structure of the (key, value) space,
enabling an extension of the context length. Our method allows for fine-tuning
pre-existing, large-scale models to lengthen their effective context. This is
demonstrated by our fine-tuning of 3B and 7B OpenLLaMA checkpoints. The
resulting models, which we name LongLLaMA, exhibit advancements in tasks
requiring a long context. We further illustrate that our LongLLaMA models
adeptly manage a 256 k context length for passkey retrieval.
64a7610f1fc2c199e62c8cf3
Szymon Tworkowski
claimed_verified
2023-07-07T07:34:16.421Z
63fa0a77d38275b44358448b
https://cdn-avatars.huggingface.co/v1/production/uploads/63fa0a77d38275b44358448b/bHaHZ-rxsOXmKsvbITitl.jpeg
Szymon Tworkowski
user
syzymon
64a7610f1fc2c199e62c8cf4
Konrad Staniszewski
64a7610f1fc2c199e62c8cf5
Mikołaj Pacek
64a7610f1fc2c199e62c8cf6
Yuhuai Wu
64a7610f1fc2c199e62c8cf7
Henryk Michalewski
64a7610f1fc2c199e62c8cf8
Piotr Miłoś
claimed_verified
2023-07-07T13:31:09.829Z
64a811577027600ca38630ae
/avatars/b70f1b40fb0e0cb9db0fda870aa9e5da.svg
Piotr Miłoś
user
lyzell2
Focused Transformer: Contrastive Training for Context Scaling
Large language models have an exceptional capability to incorporate new
information in a contextual manner. However, the full potential of such an
approach is often restrained due to a limitation in the effective context
length. One solution to this issue is to endow an attention layer with access
to an external memory, which comprises of (key, value) pairs. Yet, as the
number of documents increases, the proportion of relevant keys to irrelevant
ones decreases, leading the model to focus more on the irrelevant keys. We
identify a significant challenge, dubbed the distraction issue, where keys
linked to different semantic values might overlap, making them hard to
distinguish. To tackle this problem, we introduce the Focused Transformer
(FoT), a technique that employs a training process inspired by contrastive
learning. This novel approach enhances the structure of the (key, value) space,
enabling an extension of the context length. Our method allows for fine-tuning
pre-existing, large-scale models to lengthen their effective context. This is
demonstrated by our fine-tuning of 3B and 7B OpenLLaMA checkpoints. The
resulting models, which we name LongLLaMA, exhibit advancements in tasks
requiring a long context. We further illustrate that our LongLLaMA models
adeptly manage a 256 k context length for passkey retrieval.
A new method called Focused Transformer (FoT) enhances large language models by extending their context length through contrastive learning, demonstrated with LongLLaMA models that handle up to 256k context length.
attention layer external memory (key value) pairs distraction issue contrastive learning Focused Transformer (FoT) fine-tuning OpenLLaMA checkpoints LongLLaMA context length passkey retrieval