title: LServe: Efficient Long-sequence LLM Serving with Unified Sparse
    Attention
summary: Large language models (LLMs) have shown remarkable potential in processing
  long sequences, yet efficiently serving these long-context models remains
  challenging due to the quadratic computational complexity of attention in the
  prefilling stage and the large memory footprint of the KV cache in the decoding
  stage. To address these issues, we introduce LServe, an efficient system that
  accelerates long-sequence LLM serving via hybrid sparse attention. This method
  unifies different hardware-friendly, structured sparsity patterns for both
  prefilling and decoding attention into a single framework, where computations
  on less important tokens are skipped block-wise. LServe demonstrates the
  compatibility of static and dynamic sparsity in long-context LLM attention.
  This design enables multiplicative speedups by combining these optimizations.
  Specifically, we convert half of the attention heads to nearly free streaming
  heads in both the prefilling and decoding stages. Additionally, we find that
  only a constant number of KV pages is required to preserve long-context
  capabilities, irrespective of context length. We then design a hierarchical KV
  page selection policy that dynamically prunes KV pages based on query-centric
  similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and
  decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is
  released at https://github.com/mit-han-lab/omniserve.
paper_authors._id: 67b7f46218d8b6a80a14220b
paper_authors.name: Shang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:58:16.384Z
paper_authors._id: 641d8bacd526196afc12766d
paper_authors.avatarUrl: /avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
paper_authors.fullname: Shang Yang
paper_authors.type: user
paper_authors.user: Shangy
paper_authors._id: 67b7f46218d8b6a80a14220c
paper_authors.name: Junxian Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:17:47.441Z
paper_authors._id: 64e702982eba1760dfb0166c
paper_authors.avatarUrl: /avatars/3291112513914d823cd524dafec66c87.svg
paper_authors.fullname: Junxian Guo
paper_authors.type: user
paper_authors.user: JerryGJX
paper_authors._id: 67b7f46218d8b6a80a14220d
paper_authors.name: Haotian Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:17:54.758Z
paper_authors._id: 646791c5374fe5728d403369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646791c5374fe5728d403369/2kyCP48za0T3PTj-IR-J0.jpeg
paper_authors.fullname: Haotian Tang
paper_authors.type: user
paper_authors.user: kentang1998
paper_authors._id: 67b7f46218d8b6a80a14220e
paper_authors.name: Qinghao Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:18:00.291Z
paper_authors._id: 67a6d5f424a2ace09c62640d
paper_authors.avatarUrl: /avatars/0360e2543d791c4d046dd516eb70ced1.svg
paper_authors.fullname: Qinghao Hu
paper_authors.type: user
paper_authors.user: huqinghao
paper_authors._id: 67b7f46218d8b6a80a14220f
paper_authors.name: Guangxuan Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:18:07.219Z
paper_authors._id: 6362fefe19cf373a5fc5b39e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362fefe19cf373a5fc5b39e/v4uJ5bzjpZJOxqGUHYPM2.jpeg
paper_authors.fullname: Guangxuan Xiao
paper_authors.type: user
paper_authors.user: Guangxuan-Xiao
paper_authors._id: 67b7f46218d8b6a80a142210
paper_authors.name: Jiaming Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:18:13.068Z
paper_authors._id: 65ac13385860f06ff21c9a8a
paper_authors.avatarUrl: /avatars/5c4c151e90c0dfc0e321623013594bbe.svg
paper_authors.fullname: Jiaming Tang
paper_authors.type: user
paper_authors.user: Dudep
paper_authors._id: 67b7f46218d8b6a80a142211
paper_authors.name: Yujun Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:18:19.276Z
paper_authors._id: 66a156136609d2b2b0f6353a
paper_authors.avatarUrl: /avatars/fc6850b5fc437269bf0870f6a6cdcf40.svg
paper_authors.fullname: Yujun Lin
paper_authors.type: user
paper_authors.user: synxlin
paper_authors._id: 67b7f46218d8b6a80a142212
paper_authors.name: Zhijian Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:18:26.473Z
paper_authors._id: 650dac79b959b0e1d41d7378
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
paper_authors.fullname: Zhijian Liu
paper_authors.type: user
paper_authors.user: zhijianliu
paper_authors._id: 67b7f46218d8b6a80a142213
paper_authors.name: Yao Lu
paper_authors._id: 67b7f46218d8b6a80a142214
paper_authors.name: Song Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:18:34.872Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_title: LServe: Efficient Long-sequence LLM Serving with Unified Sparse
    Attention
paper_summary: Large language models (LLMs) have shown remarkable potential in processing
  long sequences, yet efficiently serving these long-context models remains
  challenging due to the quadratic computational complexity of attention in the
  prefilling stage and the large memory footprint of the KV cache in the decoding
  stage. To address these issues, we introduce LServe, an efficient system that
  accelerates long-sequence LLM serving via hybrid sparse attention. This method
  unifies different hardware-friendly, structured sparsity patterns for both
  prefilling and decoding attention into a single framework, where computations
  on less important tokens are skipped block-wise. LServe demonstrates the
  compatibility of static and dynamic sparsity in long-context LLM attention.
  This design enables multiplicative speedups by combining these optimizations.
  Specifically, we convert half of the attention heads to nearly free streaming
  heads in both the prefilling and decoding stages. Additionally, we find that
  only a constant number of KV pages is required to preserve long-context
  capabilities, irrespective of context length. We then design a hierarchical KV
  page selection policy that dynamically prunes KV pages based on query-centric
  similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and
  decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is
  released at https://github.com/mit-han-lab/omniserve.
paper_ai_summary: LServe accelerates long-context LLM serving using hybrid sparse attention, statically and dynamically reducing computational complexity and memory usage.
paper_ai_keywords: large language models long sequences attention quadratic computational complexity KV cache hybrid sparse attention prefilling decoding static sparsity dynamic sparsity token skipping streaming heads hierarchical KV page selection query-centric similarity