title: RetrievalAttention: Accelerating Long-Context LLM Inference via Vector
    Retrieval
summary: Transformer-based large Language Models (LLMs) become increasingly important
  in various domains. However, the quadratic time complexity of attention
  operation poses a significant challenge for scaling to longer contexts due to
  the extremely high inference latency and GPU memory consumption for caching
  key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free
  approach to accelerate attention computation. To leverage the dynamic sparse
  property of attention, RetrievalAttention builds approximate nearest neighbor
  search (ANNS) indexes upon KV vectors in CPU memory and retrieves the most
  relevant ones via vector search during generation. Due to the
  out-of-distribution (OOD) between query vectors and key vectors, off-the-shelf
  ANNS indexes still need to scan O(N) (usually 30% of all keys) data for
  accurate retrieval, which fails to exploit the high sparsity.
  RetrievalAttention first identifies the OOD challenge of ANNS-based attention,
  and addresses it via an attention-aware vector search algorithm that can adapt
  to queries and only access 1--3% of data, thus achieving a sub-linear time
  complexity. RetrievalAttention greatly reduces the inference cost of
  long-context LLM with much lower GPU memory requirements while maintaining the
  model accuracy. Especially, RetrievalAttention only needs 16GB GPU memory for
  serving 128K tokens in LLMs with 8B parameters, which is capable of generating
  one token in 0.188 seconds on a single NVIDIA RTX4090 (24GB).
paper_authors._id: 66e8f30d6646c8f0e8b11d93
paper_authors.name: Di Liu
paper_authors._id: 66e8f30d6646c8f0e8b11d94
paper_authors.name: Meng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T09:30:37.669Z
paper_authors._id: 64ace004b2f83dec96121d9e
paper_authors.avatarUrl: /avatars/0b85db4bdf41f627026b0f1aef132b85.svg
paper_authors.fullname: Meng Chen
paper_authors.type: user
paper_authors.user: Matchyc
paper_authors._id: 66e8f30d6646c8f0e8b11d95
paper_authors.name: Baotong Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:39:33.609Z
paper_authors._id: 667135bdcca06def1c2599a6
paper_authors.avatarUrl: /avatars/d94ab99265e1970852605d344b4d69e9.svg
paper_authors.fullname: Baotong Lu
paper_authors.type: user
paper_authors.user: baotonglu
paper_authors._id: 66e8f30d6646c8f0e8b11d96
paper_authors.name: Huiqiang Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:39:21.178Z
paper_authors._id: 6278bd42541f3d2dfa77ea70
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
paper_authors.fullname: Huiqiang Jiang
paper_authors.type: user
paper_authors.user: iofu728
paper_authors._id: 66e8f30d6646c8f0e8b11d97
paper_authors.name: Zhenhua Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:39:59.765Z
paper_authors._id: 64cc9b96e60d2cddfadca2c8
paper_authors.avatarUrl: /avatars/a07755847ec8d05052221d351a3ae20f.svg
paper_authors.fullname: Zhenhua Han
paper_authors.type: user
paper_authors.user: hzhua
paper_authors._id: 66e8f30d6646c8f0e8b11d98
paper_authors.name: Qianxi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:35:07.212Z
paper_authors._id: 66e96f58de9ebeee86f5e27f
paper_authors.avatarUrl: /avatars/e7d692aa47f02f1858186f47186166ad.svg
paper_authors.fullname: Qianxi Zhang
paper_authors.type: user
paper_authors.user: qianxizhang
paper_authors._id: 66e8f30d6646c8f0e8b11d99
paper_authors.name: Qi Chen
paper_authors._id: 66e8f30d6646c8f0e8b11d9a
paper_authors.name: Chengruidong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:40:30.129Z
paper_authors._id: 64646896884f2e3e1ced3cd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chengruidong
paper_authors._id: 66e8f30d6646c8f0e8b11d9b
paper_authors.name: Bailu Ding
paper_authors._id: 66e8f30d6646c8f0e8b11d9c
paper_authors.name: Kai Zhang
paper_authors._id: 66e8f30d6646c8f0e8b11d9d
paper_authors.name: Chen Chen
paper_authors._id: 66e8f30d6646c8f0e8b11d9e
paper_authors.name: Fan Yang
paper_authors._id: 66e8f30d6646c8f0e8b11d9f
paper_authors.name: Yuqing Yang
paper_authors._id: 66e8f30d6646c8f0e8b11da0
paper_authors.name: Lili Qiu
paper_title: RetrievalAttention: Accelerating Long-Context LLM Inference via Vector
    Retrieval
paper_summary: Transformer-based large Language Models (LLMs) become increasingly important
  in various domains. However, the quadratic time complexity of attention
  operation poses a significant challenge for scaling to longer contexts due to
  the extremely high inference latency and GPU memory consumption for caching
  key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free
  approach to accelerate attention computation. To leverage the dynamic sparse
  property of attention, RetrievalAttention builds approximate nearest neighbor
  search (ANNS) indexes upon KV vectors in CPU memory and retrieves the most
  relevant ones via vector search during generation. Due to the
  out-of-distribution (OOD) between query vectors and key vectors, off-the-shelf
  ANNS indexes still need to scan O(N) (usually 30% of all keys) data for
  accurate retrieval, which fails to exploit the high sparsity.
  RetrievalAttention first identifies the OOD challenge of ANNS-based attention,
  and addresses it via an attention-aware vector search algorithm that can adapt
  to queries and only access 1--3% of data, thus achieving a sub-linear time
  complexity. RetrievalAttention greatly reduces the inference cost of
  long-context LLM with much lower GPU memory requirements while maintaining the
  model accuracy. Especially, RetrievalAttention only needs 16GB GPU memory for
  serving 128K tokens in LLMs with 8B parameters, which is capable of generating
  one token in 0.188 seconds on a single NVIDIA RTX4090 (24GB).
paper_ai_summary: RetrievalAttention is a training-free method that accelerates attention computation in Transformer-based LLMs by using attention-aware vector search to reduce retrieval costs and GPU memory usage.
paper_ai_keywords: Transformer-based large Language Models LLMs attention operation quadratic time complexity key-value (KV) vectors RetrievalAttention approximate nearest neighbor search ANNS out-of-distribution (OOD) attention-aware vector search sub-linear time complexity