title: Multi-Token Attention
summary: Soft attention is a critical mechanism powering LLMs to locate relevant parts
  within a given context. However, individual attention weights are determined by
  the similarity of only a single query and key token vector. This "single token
  attention" bottlenecks the amount of information used in distinguishing a
  relevant part from the rest of the context. To address this issue, we propose a
  new attention method, Multi-Token Attention (MTA), which allows LLMs to
  condition their attention weights on multiple query and key vectors
  simultaneously. This is achieved by applying convolution operations over
  queries, keys and heads, allowing nearby queries and keys to affect each
  other's attention weights for more precise attention. As a result, our method
  can locate relevant context using richer, more nuanced information that can
  exceed a single vector's capacity. Through extensive evaluations, we
  demonstrate that MTA achieves enhanced performance on a range of popular
  benchmarks. Notably, it outperforms Transformer baseline models on standard
  language modeling tasks, and on tasks that require searching for information
  within long contexts, where our method's ability to leverage richer information
  proves particularly beneficial.
paper_authors._id: 67ecc23b6a2bc6abdc2e9183
paper_authors.name: Olga Golovneva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:56:07.426Z
paper_authors._id: 6318b1e06e6bd039440871d6
paper_authors.avatarUrl: /avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
paper_authors.fullname: Olga Golovneva
paper_authors.type: user
paper_authors.user: Golovneva
paper_authors._id: 67ecc23b6a2bc6abdc2e9184
paper_authors.name: Tianlu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:56:14.148Z
paper_authors._id: 63f6a0e6b4c9a104f4b95d86
paper_authors.avatarUrl: /avatars/660c3e44bea976f3e0560bf95466b03c.svg
paper_authors.fullname: Tianlu Wang
paper_authors.type: user
paper_authors.user: Tianlu
paper_authors._id: 67ecc23b6a2bc6abdc2e9185
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:56:21.002Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 67ecc23b6a2bc6abdc2e9186
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-02T11:56:26.631Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_title: Multi-Token Attention
paper_summary: Soft attention is a critical mechanism powering LLMs to locate relevant parts
  within a given context. However, individual attention weights are determined by
  the similarity of only a single query and key token vector. This "single token
  attention" bottlenecks the amount of information used in distinguishing a
  relevant part from the rest of the context. To address this issue, we propose a
  new attention method, Multi-Token Attention (MTA), which allows LLMs to
  condition their attention weights on multiple query and key vectors
  simultaneously. This is achieved by applying convolution operations over
  queries, keys and heads, allowing nearby queries and keys to affect each
  other's attention weights for more precise attention. As a result, our method
  can locate relevant context using richer, more nuanced information that can
  exceed a single vector's capacity. Through extensive evaluations, we
  demonstrate that MTA achieves enhanced performance on a range of popular
  benchmarks. Notably, it outperforms Transformer baseline models on standard
  language modeling tasks, and on tasks that require searching for information
  within long contexts, where our method's ability to leverage richer information
  proves particularly beneficial.
paper_ai_summary: A new attention mechanism, Multi-Token Attention (MTA), enhances LLM performance by conditioning attention weights on multiple query and key vectors, improving context search and standard language modeling tasks.
paper_ai_keywords: soft attention LLMs single token attention Multi-Token Attention (MTA) convolution operations attention weights queries keys heads language modeling long contexts