DuoAttention: Efficient Long-Context LLM Inference with Retrieval and
  Streaming Heads
Deploying long-context large language models (LLMs) is essential but poses
significant computational and memory challenges. Caching all Key and Value (KV)
states across all attention heads consumes substantial memory. Existing KV
cache pruning methods either damage the long-context capabilities of LLMs or
offer only limited efficiency improvements. In this paper, we identify that
only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for
processing long contexts and require full attention across all tokens. In
contrast, all other heads, which primarily focus on recent tokens and attention
sinks--referred to as Streaming Heads--do not require full attention. Based on
this insight, we introduce DuoAttention, a framework that only applies a full
KV cache to retrieval heads while using a light-weight, constant-length KV
cache for streaming heads, which reduces both LLM's decoding and pre-filling
memory and latency without compromising its long-context abilities.
DuoAttention uses a lightweight, optimization-based algorithm with synthetic
data to identify retrieval heads accurately. Our method significantly reduces
long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models
while speeding up decoding by up to 2.18x and 1.50x and accelerating
pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with
minimal accuracy loss compared to full attention. Notably, combined with
quantization, DuoAttention enables Llama-3-8B decoding with 3.3 million context
length on a single A100 GPU. Code is provided in
https://github.com/mit-han-lab/duo-attention.
670f0ee0125e4ced02d514e4
Guangxuan Xiao
claimed_verified
2024-10-18T08:25:16.976Z
6362fefe19cf373a5fc5b39e
https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
Guangxuan Xiao
user
Guangxuan-Xiao
670f0ee0125e4ced02d514e5
Jiaming Tang
claimed_verified
2024-10-21T07:48:42.023Z
63f30d28f4e30ffd2bda9aff
https://cdn-avatars.huggingface.co/v1/production/uploads/1676872959244-noauth.jpeg
Jiaming Tang
user
Sakits
670f0ee0125e4ced02d514e6
Jingwei Zuo
670f0ee0125e4ced02d514e7
Junxian Guo
670f0ee0125e4ced02d514e8
Shang Yang
claimed_verified
2025-02-21T10:00:33.554Z
641d8bacd526196afc12766d
/avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
Shang Yang
user
Shangy
670f0ee0125e4ced02d514e9
Haotian Tang
670f0ee0125e4ced02d514ea
Yao Fu
670f0ee0125e4ced02d514eb
Song Han
DuoAttention: Efficient Long-Context LLM Inference with Retrieval and
  Streaming Heads
Deploying long-context large language models (LLMs) is essential but poses
significant computational and memory challenges. Caching all Key and Value (KV)
states across all attention heads consumes substantial memory. Existing KV
cache pruning methods either damage the long-context capabilities of LLMs or
offer only limited efficiency improvements. In this paper, we identify that
only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for
processing long contexts and require full attention across all tokens. In
contrast, all other heads, which primarily focus on recent tokens and attention
sinks--referred to as Streaming Heads--do not require full attention. Based on
this insight, we introduce DuoAttention, a framework that only applies a full
KV cache to retrieval heads while using a light-weight, constant-length KV
cache for streaming heads, which reduces both LLM's decoding and pre-filling
memory and latency without compromising its long-context abilities.
DuoAttention uses a lightweight, optimization-based algorithm with synthetic
data to identify retrieval heads accurately. Our method significantly reduces
long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models
while speeding up decoding by up to 2.18x and 1.50x and accelerating
pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with
minimal accuracy loss compared to full attention. Notably, combined with
quantization, DuoAttention enables Llama-3-8B decoding with 3.3 million context
length on a single A100 GPU. Code is provided in
https://github.com/mit-han-lab/duo-attention.
DuoAttention reduces memory and latency for long-context LLMs by using a full KV cache only for critical retrieval heads while employing a lightweight cache for streaming heads, improving efficiency without sacrificing performance.
long-context large language models LLMs Key and Value states KV cache pruning attention heads Retrieval Heads Streaming Heads DuoAttention full KV cache lightweight KV cache optimization-based algorithm synthetic data long-context inference memory reduction decoding speed pre-filling speed accuracy loss quantization Llama-3-8B A100 GPU