RetrievalAttention: Accelerating Long-Context LLM Inference via Vector
  Retrieval
Transformer-based large Language Models (LLMs) become increasingly important
in various domains. However, the quadratic time complexity of attention
operation poses a significant challenge for scaling to longer contexts due to
the extremely high inference latency and GPU memory consumption for caching
key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free
approach to accelerate attention computation. To leverage the dynamic sparse
property of attention, RetrievalAttention builds approximate nearest neighbor
search (ANNS) indexes upon KV vectors in CPU memory and retrieves the most
relevant ones via vector search during generation. Due to the
out-of-distribution (OOD) between query vectors and key vectors, off-the-shelf
ANNS indexes still need to scan O(N) (usually 30% of all keys) data for
accurate retrieval, which fails to exploit the high sparsity.
RetrievalAttention first identifies the OOD challenge of ANNS-based attention,
and addresses it via an attention-aware vector search algorithm that can adapt
to queries and only access 1--3% of data, thus achieving a sub-linear time
complexity. RetrievalAttention greatly reduces the inference cost of
long-context LLM with much lower GPU memory requirements while maintaining the
model accuracy. Especially, RetrievalAttention only needs 16GB GPU memory for
serving 128K tokens in LLMs with 8B parameters, which is capable of generating
one token in 0.188 seconds on a single NVIDIA RTX4090 (24GB).
66e8f30d6646c8f0e8b11d93
Di Liu
66e8f30d6646c8f0e8b11d94
Meng Chen
claimed_verified
2024-09-17T09:30:37.669Z
64ace004b2f83dec96121d9e
/avatars/0b85db4bdf41f627026b0f1aef132b85.svg
Meng Chen
user
Matchyc
66e8f30d6646c8f0e8b11d95
Baotong Lu
admin_assigned
2024-09-17T07:39:33.609Z
667135bdcca06def1c2599a6
/avatars/d94ab99265e1970852605d344b4d69e9.svg
Baotong Lu
user
baotonglu
66e8f30d6646c8f0e8b11d96
Huiqiang Jiang
admin_assigned
2024-09-17T07:39:21.178Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
66e8f30d6646c8f0e8b11d97
Zhenhua Han
admin_assigned
2024-09-17T07:39:59.765Z
64cc9b96e60d2cddfadca2c8
/avatars/a07755847ec8d05052221d351a3ae20f.svg
Zhenhua Han
user
hzhua
66e8f30d6646c8f0e8b11d98
Qianxi Zhang
claimed_verified
2024-09-25T07:35:07.212Z
66e96f58de9ebeee86f5e27f
/avatars/e7d692aa47f02f1858186f47186166ad.svg
Qianxi Zhang
user
qianxizhang
66e8f30d6646c8f0e8b11d99
Qi Chen
66e8f30d6646c8f0e8b11d9a
Chengruidong Zhang
admin_assigned
2024-09-17T07:40:30.129Z
64646896884f2e3e1ced3cd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
Zhang
user
Chengruidong
66e8f30d6646c8f0e8b11d9b
Bailu Ding
66e8f30d6646c8f0e8b11d9c
Kai Zhang
66e8f30d6646c8f0e8b11d9d
Chen Chen
66e8f30d6646c8f0e8b11d9e
Fan Yang
66e8f30d6646c8f0e8b11d9f
Yuqing Yang
66e8f30d6646c8f0e8b11da0
Lili Qiu
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector
  Retrieval
Transformer-based large Language Models (LLMs) become increasingly important
in various domains. However, the quadratic time complexity of attention
operation poses a significant challenge for scaling to longer contexts due to
the extremely high inference latency and GPU memory consumption for caching
key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free
approach to accelerate attention computation. To leverage the dynamic sparse
property of attention, RetrievalAttention builds approximate nearest neighbor
search (ANNS) indexes upon KV vectors in CPU memory and retrieves the most
relevant ones via vector search during generation. Due to the
out-of-distribution (OOD) between query vectors and key vectors, off-the-shelf
ANNS indexes still need to scan O(N) (usually 30% of all keys) data for
accurate retrieval, which fails to exploit the high sparsity.
RetrievalAttention first identifies the OOD challenge of ANNS-based attention,
and addresses it via an attention-aware vector search algorithm that can adapt
to queries and only access 1--3% of data, thus achieving a sub-linear time
complexity. RetrievalAttention greatly reduces the inference cost of
long-context LLM with much lower GPU memory requirements while maintaining the
model accuracy. Especially, RetrievalAttention only needs 16GB GPU memory for
serving 128K tokens in LLMs with 8B parameters, which is capable of generating
one token in 0.188 seconds on a single NVIDIA RTX4090 (24GB).
RetrievalAttention is a training-free method that accelerates attention computation in Transformer-based LLMs by using attention-aware vector search to reduce retrieval costs and GPU memory usage.
Transformer-based large Language Models LLMs attention operation quadratic time complexity key-value (KV) vectors RetrievalAttention approximate nearest neighbor search ANNS out-of-distribution (OOD) attention-aware vector search sub-linear time complexity