LServe: Efficient Long-sequence LLM Serving with Unified Sparse
  Attention
Large language models (LLMs) have shown remarkable potential in processing
long sequences, yet efficiently serving these long-context models remains
challenging due to the quadratic computational complexity of attention in the
prefilling stage and the large memory footprint of the KV cache in the decoding
stage. To address these issues, we introduce LServe, an efficient system that
accelerates long-sequence LLM serving via hybrid sparse attention. This method
unifies different hardware-friendly, structured sparsity patterns for both
prefilling and decoding attention into a single framework, where computations
on less important tokens are skipped block-wise. LServe demonstrates the
compatibility of static and dynamic sparsity in long-context LLM attention.
This design enables multiplicative speedups by combining these optimizations.
Specifically, we convert half of the attention heads to nearly free streaming
heads in both the prefilling and decoding stages. Additionally, we find that
only a constant number of KV pages is required to preserve long-context
capabilities, irrespective of context length. We then design a hierarchical KV
page selection policy that dynamically prunes KV pages based on query-centric
similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and
decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is
released at https://github.com/mit-han-lab/omniserve.
67b7f46218d8b6a80a14220b
Shang Yang
claimed_verified
2025-02-21T09:58:16.384Z
641d8bacd526196afc12766d
/avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
Shang Yang
user
Shangy
67b7f46218d8b6a80a14220c
Junxian Guo
67b7f46218d8b6a80a14220d
Haotian Tang
admin_assigned
2025-02-21T15:17:54.758Z
646791c5374fe5728d403369
https://cdn-avatars.huggingface.co/v1/production/uploads/646791c5374fe5728d403369/2kyCP48za0T3PTj-IR-J0.jpeg
Haotian Tang
user
kentang1998
67b7f46218d8b6a80a14220e
Qinghao Hu
admin_assigned
2025-02-21T15:18:00.291Z
67a6d5f424a2ace09c62640d
/avatars/0360e2543d791c4d046dd516eb70ced1.svg
Qinghao Hu
user
huqinghao
67b7f46218d8b6a80a14220f
Guangxuan Xiao
admin_assigned
2025-02-21T15:18:07.219Z
6362fefe19cf373a5fc5b39e
https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
Guangxuan Xiao
user
Guangxuan-Xiao
67b7f46218d8b6a80a142210
Jiaming Tang
admin_assigned
2025-02-21T15:18:13.068Z
65ac13385860f06ff21c9a8a
/avatars/5c4c151e90c0dfc0e321623013594bbe.svg
Jiaming Tang
user
Dudep
67b7f46218d8b6a80a142211
Yujun Lin
admin_assigned
2025-02-21T15:18:19.276Z
66a156136609d2b2b0f6353a
/avatars/fc6850b5fc437269bf0870f6a6cdcf40.svg
Yujun Lin
user
synxlin
67b7f46218d8b6a80a142212
Zhijian Liu
admin_assigned
2025-02-21T15:18:26.473Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
67b7f46218d8b6a80a142213
Yao Lu
67b7f46218d8b6a80a142214
Song Han
admin_assigned
2025-02-21T15:18:34.872Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
LServe: Efficient Long-sequence LLM Serving with Unified Sparse
  Attention
Large language models (LLMs) have shown remarkable potential in processing
long sequences, yet efficiently serving these long-context models remains
challenging due to the quadratic computational complexity of attention in the
prefilling stage and the large memory footprint of the KV cache in the decoding
stage. To address these issues, we introduce LServe, an efficient system that
accelerates long-sequence LLM serving via hybrid sparse attention. This method
unifies different hardware-friendly, structured sparsity patterns for both
prefilling and decoding attention into a single framework, where computations
on less important tokens are skipped block-wise. LServe demonstrates the
compatibility of static and dynamic sparsity in long-context LLM attention.
This design enables multiplicative speedups by combining these optimizations.
Specifically, we convert half of the attention heads to nearly free streaming
heads in both the prefilling and decoding stages. Additionally, we find that
only a constant number of KV pages is required to preserve long-context
capabilities, irrespective of context length. We then design a hierarchical KV
page selection policy that dynamically prunes KV pages based on query-centric
similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and
decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is
released at https://github.com/mit-han-lab/omniserve.
LServe accelerates long-context LLM serving using hybrid sparse attention, statically and dynamically reducing computational complexity and memory usage.
large language models long sequences attention quadratic computational complexity KV cache hybrid sparse attention prefilling decoding static sparsity dynamic sparsity token skipping streaming heads hierarchical KV page selection query-centric similarity