RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM
  Inference
The growing context lengths of large language models (LLMs) pose significant
challenges for efficient inference, primarily due to GPU memory and bandwidth
constraints. We present RetroInfer, a novel system that reconceptualizes the
key-value (KV) cache as a vector storage system which exploits the inherent
attention sparsity to accelerate long-context LLM inference. At its core is the
wave index, an Attention-aWare VEctor index that enables efficient and accurate
retrieval of critical tokens through techniques such as tripartite attention
approximation, accuracy-bounded attention estimation, and segmented clustering.
Complementing this is the wave buffer, which coordinates KV cache placement and
overlaps computation and data transfer across GPU and CPU to sustain high
throughput. Unlike prior sparsity-based methods that struggle with token
selection and hardware coordination, RetroInfer delivers robust performance
without compromising model accuracy. Experiments on long-context benchmarks
show up to 4.5X speedup over full attention within GPU memory limits and up to
10.5X over sparse attention baselines when KV cache is extended to CPU memory,
all while preserving full-attention-level accuracy.
681abdbef8feaef23b543877
Yaoqi Chen
681abdbef8feaef23b543878
Jinkai Zhang
681abdbef8feaef23b543879
Baotong Lu
extracted_pending
2025-05-07T01:56:16.240Z
667135bdcca06def1c2599a6
/avatars/d94ab99265e1970852605d344b4d69e9.svg
Baotong Lu
user
baotonglu
681abdbef8feaef23b54387a
Qianxi Zhang
claimed_verified
2025-05-07T08:09:00.341Z
66e96f58de9ebeee86f5e27f
/avatars/e7d692aa47f02f1858186f47186166ad.svg
Qianxi Zhang
user
qianxizhang
681abdbef8feaef23b54387b
Chengruidong Zhang
681abdbef8feaef23b54387c
Jingjia Luo
681abdbef8feaef23b54387d
Di Liu
681abdbef8feaef23b54387e
Huiqiang Jiang
claimed_verified
2025-05-08T08:59:39.061Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
681abdbef8feaef23b54387f
Qi Chen
681abdbef8feaef23b543880
Jing Liu
681abdbef8feaef23b543881
Bailu Ding
681abdbef8feaef23b543882
Xiao Yan
681abdbef8feaef23b543883
Jiawei Jiang
681abdbef8feaef23b543884
Chen Chen
681abdbef8feaef23b543885
Mingxing Zhang
681abdbef8feaef23b543886
Yuqing Yang
681abdbef8feaef23b543887
Fan Yang
681abdbef8feaef23b543888
Mao Yang
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM
  Inference
The growing context lengths of large language models (LLMs) pose significant
challenges for efficient inference, primarily due to GPU memory and bandwidth
constraints. We present RetroInfer, a novel system that reconceptualizes the
key-value (KV) cache as a vector storage system which exploits the inherent
attention sparsity to accelerate long-context LLM inference. At its core is the
wave index, an Attention-aWare VEctor index that enables efficient and accurate
retrieval of critical tokens through techniques such as tripartite attention
approximation, accuracy-bounded attention estimation, and segmented clustering.
Complementing this is the wave buffer, which coordinates KV cache placement and
overlaps computation and data transfer across GPU and CPU to sustain high
throughput. Unlike prior sparsity-based methods that struggle with token
selection and hardware coordination, RetroInfer delivers robust performance
without compromising model accuracy. Experiments on long-context benchmarks
show up to 4.5X speedup over full attention within GPU memory limits and up to
10.5X over sparse attention baselines when KV cache is extended to CPU memory,
all while preserving full-attention-level accuracy.
RetroInfer, a system that uses a vector storage approach based on attention sparsity, significantly accelerates inference for large language models with long contexts without reducing accuracy.
large language models KV cache vector storage system attention sparsity wave index attention-aware tripartite attention approximation accuracy-bounded attention estimation segmented clustering wave buffer KV cache placement GPU memory CPU memory long-context benchmarks full attention sparse attention