title: Focused Transformer: Contrastive Training for Context Scaling
summary: Large language models have an exceptional capability to incorporate new
  information in a contextual manner. However, the full potential of such an
  approach is often restrained due to a limitation in the effective context
  length. One solution to this issue is to endow an attention layer with access
  to an external memory, which comprises of (key, value) pairs. Yet, as the
  number of documents increases, the proportion of relevant keys to irrelevant
  ones decreases, leading the model to focus more on the irrelevant keys. We
  identify a significant challenge, dubbed the distraction issue, where keys
  linked to different semantic values might overlap, making them hard to
  distinguish. To tackle this problem, we introduce the Focused Transformer
  (FoT), a technique that employs a training process inspired by contrastive
  learning. This novel approach enhances the structure of the (key, value) space,
  enabling an extension of the context length. Our method allows for fine-tuning
  pre-existing, large-scale models to lengthen their effective context. This is
  demonstrated by our fine-tuning of 3B and 7B OpenLLaMA checkpoints. The
  resulting models, which we name LongLLaMA, exhibit advancements in tasks
  requiring a long context. We further illustrate that our LongLLaMA models
  adeptly manage a 256 k context length for passkey retrieval.
paper_authors._id: 64a7610f1fc2c199e62c8cf3
paper_authors.name: Szymon Tworkowski
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T07:34:16.421Z
paper_authors._id: 63fa0a77d38275b44358448b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63fa0a77d38275b44358448b/bHaHZ-rxsOXmKsvbITitl.jpeg
paper_authors.fullname: Szymon Tworkowski
paper_authors.type: user
paper_authors.user: syzymon
paper_authors._id: 64a7610f1fc2c199e62c8cf4
paper_authors.name: Konrad Staniszewski
paper_authors._id: 64a7610f1fc2c199e62c8cf5
paper_authors.name: Mikołaj Pacek
paper_authors._id: 64a7610f1fc2c199e62c8cf6
paper_authors.name: Yuhuai Wu
paper_authors._id: 64a7610f1fc2c199e62c8cf7
paper_authors.name: Henryk Michalewski
paper_authors._id: 64a7610f1fc2c199e62c8cf8
paper_authors.name: Piotr Miłoś
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T13:31:09.829Z
paper_authors._id: 64a811577027600ca38630ae
paper_authors.avatarUrl: /avatars/b70f1b40fb0e0cb9db0fda870aa9e5da.svg
paper_authors.fullname: Piotr Miłoś
paper_authors.type: user
paper_authors.user: lyzell2
paper_title: Focused Transformer: Contrastive Training for Context Scaling
paper_summary: Large language models have an exceptional capability to incorporate new
  information in a contextual manner. However, the full potential of such an
  approach is often restrained due to a limitation in the effective context
  length. One solution to this issue is to endow an attention layer with access
  to an external memory, which comprises of (key, value) pairs. Yet, as the
  number of documents increases, the proportion of relevant keys to irrelevant
  ones decreases, leading the model to focus more on the irrelevant keys. We
  identify a significant challenge, dubbed the distraction issue, where keys
  linked to different semantic values might overlap, making them hard to
  distinguish. To tackle this problem, we introduce the Focused Transformer
  (FoT), a technique that employs a training process inspired by contrastive
  learning. This novel approach enhances the structure of the (key, value) space,
  enabling an extension of the context length. Our method allows for fine-tuning
  pre-existing, large-scale models to lengthen their effective context. This is
  demonstrated by our fine-tuning of 3B and 7B OpenLLaMA checkpoints. The
  resulting models, which we name LongLLaMA, exhibit advancements in tasks
  requiring a long context. We further illustrate that our LongLLaMA models
  adeptly manage a 256 k context length for passkey retrieval.
paper_ai_summary: A new method called Focused Transformer (FoT) enhances large language models by extending their context length through contrastive learning, demonstrated with LongLLaMA models that handle up to 256k context length.
paper_ai_keywords: attention layer external memory (key value) pairs distraction issue contrastive learning Focused Transformer (FoT) fine-tuning OpenLLaMA checkpoints LongLLaMA context length passkey retrieval