title: RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM
    Inference
summary: The growing context lengths of large language models (LLMs) pose significant
  challenges for efficient inference, primarily due to GPU memory and bandwidth
  constraints. We present RetroInfer, a novel system that reconceptualizes the
  key-value (KV) cache as a vector storage system which exploits the inherent
  attention sparsity to accelerate long-context LLM inference. At its core is the
  wave index, an Attention-aWare VEctor index that enables efficient and accurate
  retrieval of critical tokens through techniques such as tripartite attention
  approximation, accuracy-bounded attention estimation, and segmented clustering.
  Complementing this is the wave buffer, which coordinates KV cache placement and
  overlaps computation and data transfer across GPU and CPU to sustain high
  throughput. Unlike prior sparsity-based methods that struggle with token
  selection and hardware coordination, RetroInfer delivers robust performance
  without compromising model accuracy. Experiments on long-context benchmarks
  show up to 4.5X speedup over full attention within GPU memory limits and up to
  10.5X over sparse attention baselines when KV cache is extended to CPU memory,
  all while preserving full-attention-level accuracy.
paper_authors._id: 681abdbef8feaef23b543877
paper_authors.name: Yaoqi Chen
paper_authors._id: 681abdbef8feaef23b543878
paper_authors.name: Jinkai Zhang
paper_authors._id: 681abdbef8feaef23b543879
paper_authors.name: Baotong Lu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-07T01:56:16.240Z
paper_authors._id: 667135bdcca06def1c2599a6
paper_authors.avatarUrl: /avatars/d94ab99265e1970852605d344b4d69e9.svg
paper_authors.fullname: Baotong Lu
paper_authors.type: user
paper_authors.user: baotonglu
paper_authors._id: 681abdbef8feaef23b54387a
paper_authors.name: Qianxi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-07T08:09:00.341Z
paper_authors._id: 66e96f58de9ebeee86f5e27f
paper_authors.avatarUrl: /avatars/e7d692aa47f02f1858186f47186166ad.svg
paper_authors.fullname: Qianxi Zhang
paper_authors.type: user
paper_authors.user: qianxizhang
paper_authors._id: 681abdbef8feaef23b54387b
paper_authors.name: Chengruidong Zhang
paper_authors._id: 681abdbef8feaef23b54387c
paper_authors.name: Jingjia Luo
paper_authors._id: 681abdbef8feaef23b54387d
paper_authors.name: Di Liu
paper_authors._id: 681abdbef8feaef23b54387e
paper_authors.name: Huiqiang Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T08:59:39.061Z
paper_authors._id: 6278bd42541f3d2dfa77ea70
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
paper_authors.fullname: Huiqiang Jiang
paper_authors.type: user
paper_authors.user: iofu728
paper_authors._id: 681abdbef8feaef23b54387f
paper_authors.name: Qi Chen
paper_authors._id: 681abdbef8feaef23b543880
paper_authors.name: Jing Liu
paper_authors._id: 681abdbef8feaef23b543881
paper_authors.name: Bailu Ding
paper_authors._id: 681abdbef8feaef23b543882
paper_authors.name: Xiao Yan
paper_authors._id: 681abdbef8feaef23b543883
paper_authors.name: Jiawei Jiang
paper_authors._id: 681abdbef8feaef23b543884
paper_authors.name: Chen Chen
paper_authors._id: 681abdbef8feaef23b543885
paper_authors.name: Mingxing Zhang
paper_authors._id: 681abdbef8feaef23b543886
paper_authors.name: Yuqing Yang
paper_authors._id: 681abdbef8feaef23b543887
paper_authors.name: Fan Yang
paper_authors._id: 681abdbef8feaef23b543888
paper_authors.name: Mao Yang
paper_title: RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM
    Inference
paper_summary: The growing context lengths of large language models (LLMs) pose significant
  challenges for efficient inference, primarily due to GPU memory and bandwidth
  constraints. We present RetroInfer, a novel system that reconceptualizes the
  key-value (KV) cache as a vector storage system which exploits the inherent
  attention sparsity to accelerate long-context LLM inference. At its core is the
  wave index, an Attention-aWare VEctor index that enables efficient and accurate
  retrieval of critical tokens through techniques such as tripartite attention
  approximation, accuracy-bounded attention estimation, and segmented clustering.
  Complementing this is the wave buffer, which coordinates KV cache placement and
  overlaps computation and data transfer across GPU and CPU to sustain high
  throughput. Unlike prior sparsity-based methods that struggle with token
  selection and hardware coordination, RetroInfer delivers robust performance
  without compromising model accuracy. Experiments on long-context benchmarks
  show up to 4.5X speedup over full attention within GPU memory limits and up to
  10.5X over sparse attention baselines when KV cache is extended to CPU memory,
  all while preserving full-attention-level accuracy.
paper_ai_summary: RetroInfer, a system that uses a vector storage approach based on attention sparsity, significantly accelerates inference for large language models with long contexts without reducing accuracy.
paper_ai_keywords: large language models KV cache vector storage system attention sparsity wave index attention-aware tripartite attention approximation accuracy-bounded attention estimation segmented clustering wave buffer KV cache placement GPU memory CPU memory long-context benchmarks full attention sparse attention