Multi-Token Attention
Soft attention is a critical mechanism powering LLMs to locate relevant parts
within a given context. However, individual attention weights are determined by
the similarity of only a single query and key token vector. This "single token
attention" bottlenecks the amount of information used in distinguishing a
relevant part from the rest of the context. To address this issue, we propose a
new attention method, Multi-Token Attention (MTA), which allows LLMs to
condition their attention weights on multiple query and key vectors
simultaneously. This is achieved by applying convolution operations over
queries, keys and heads, allowing nearby queries and keys to affect each
other's attention weights for more precise attention. As a result, our method
can locate relevant context using richer, more nuanced information that can
exceed a single vector's capacity. Through extensive evaluations, we
demonstrate that MTA achieves enhanced performance on a range of popular
benchmarks. Notably, it outperforms Transformer baseline models on standard
language modeling tasks, and on tasks that require searching for information
within long contexts, where our method's ability to leverage richer information
proves particularly beneficial.
67ecc23b6a2bc6abdc2e9183
Olga Golovneva
admin_assigned
2025-04-02T11:56:07.426Z
6318b1e06e6bd039440871d6
/avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
Olga Golovneva
user
Golovneva
67ecc23b6a2bc6abdc2e9184
Tianlu Wang
admin_assigned
2025-04-02T11:56:14.148Z
63f6a0e6b4c9a104f4b95d86
/avatars/660c3e44bea976f3e0560bf95466b03c.svg
Tianlu Wang
user
Tianlu
67ecc23b6a2bc6abdc2e9185
Jason Weston
admin_assigned
2025-04-02T11:56:21.002Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
67ecc23b6a2bc6abdc2e9186
Sainbayar Sukhbaatar
admin_assigned
2025-04-02T11:56:26.631Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
Multi-Token Attention
Soft attention is a critical mechanism powering LLMs to locate relevant parts
within a given context. However, individual attention weights are determined by
the similarity of only a single query and key token vector. This "single token
attention" bottlenecks the amount of information used in distinguishing a
relevant part from the rest of the context. To address this issue, we propose a
new attention method, Multi-Token Attention (MTA), which allows LLMs to
condition their attention weights on multiple query and key vectors
simultaneously. This is achieved by applying convolution operations over
queries, keys and heads, allowing nearby queries and keys to affect each
other's attention weights for more precise attention. As a result, our method
can locate relevant context using richer, more nuanced information that can
exceed a single vector's capacity. Through extensive evaluations, we
demonstrate that MTA achieves enhanced performance on a range of popular
benchmarks. Notably, it outperforms Transformer baseline models on standard
language modeling tasks, and on tasks that require searching for information
within long contexts, where our method's ability to leverage richer information
proves particularly beneficial.
A new attention mechanism, Multi-Token Attention (MTA), enhances LLM performance by conditioning attention weights on multiple query and key vectors, improving context search and standard language modeling tasks.
soft attention LLMs single token attention Multi-Token Attention (MTA) convolution operations attention weights queries keys heads language modeling long contexts