title: Efficient Memory Management for Large Language Model Serving with
    PagedAttention
summary: High throughput serving of large language models (LLMs) requires batching
  sufficiently many requests at a time. However, existing systems struggle
  because the key-value cache (KV cache) memory for each request is huge and
  grows and shrinks dynamically. When managed inefficiently, this memory can be
  significantly wasted by fragmentation and redundant duplication, limiting the
  batch size. To address this problem, we propose PagedAttention, an attention
  algorithm inspired by the classical virtual memory and paging techniques in
  operating systems. On top of it, we build vLLM, an LLM serving system that
  achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV
  cache within and across requests to further reduce memory usage. Our
  evaluations show that vLLM improves the throughput of popular LLMs by
  2-4times with the same level of latency compared to the state-of-the-art
  systems, such as FasterTransformer and Orca. The improvement is more pronounced
  with longer sequences, larger models, and more complex decoding algorithms.
  vLLM's source code is publicly available at
  https://github.com/vllm-project/vllm
paper_authors._id: 650114d996ec8a5218886621
paper_authors.name: Woosuk Kwon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-13T07:28:58.381Z
paper_authors._id: 6458733caa426bae5e079b7b
paper_authors.avatarUrl: /avatars/990d0c21e28f973290ce429f62156031.svg
paper_authors.fullname: Woosuk Kwon
paper_authors.type: user
paper_authors.user: wskwon
paper_authors._id: 650114d996ec8a5218886622
paper_authors.name: Zhuohan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-13T07:29:05.538Z
paper_authors._id: 649dd1fae16abbd8fd7a3bdf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2t1p0Ku7qmNcSapRdzxv9.jpeg
paper_authors.fullname: Zhuohan  Li
paper_authors.type: user
paper_authors.user: zhuohan123
paper_authors._id: 650114d996ec8a5218886623
paper_authors.name: Siyuan Zhuang
paper_authors._id: 650114d996ec8a5218886624
paper_authors.name: Ying Sheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-13T07:29:40.177Z
paper_authors._id: 6455b969654d8bccae50736c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
paper_authors.fullname: Ying Sheng
paper_authors.type: user
paper_authors.user: ying1123
paper_authors._id: 650114d996ec8a5218886625
paper_authors.name: Lianmin Zheng
paper_authors._id: 650114d996ec8a5218886626
paper_authors.name: Cody Hao Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T07:31:46.459Z
paper_authors._id: 64763d68288784ce9aba1cc7
paper_authors.avatarUrl: /avatars/84033af4466f5469c6264e7bff9f6819.svg
paper_authors.fullname: Cody Yu
paper_authors.type: user
paper_authors.user: comaniac
paper_authors._id: 650114d996ec8a5218886627
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-13T07:30:30.707Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 650114d996ec8a5218886628
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-01T20:15:04.984Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_authors._id: 650114d996ec8a5218886629
paper_authors.name: Ion Stoica
paper_title: Efficient Memory Management for Large Language Model Serving with
    PagedAttention
paper_summary: High throughput serving of large language models (LLMs) requires batching
  sufficiently many requests at a time. However, existing systems struggle
  because the key-value cache (KV cache) memory for each request is huge and
  grows and shrinks dynamically. When managed inefficiently, this memory can be
  significantly wasted by fragmentation and redundant duplication, limiting the
  batch size. To address this problem, we propose PagedAttention, an attention
  algorithm inspired by the classical virtual memory and paging techniques in
  operating systems. On top of it, we build vLLM, an LLM serving system that
  achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV
  cache within and across requests to further reduce memory usage. Our
  evaluations show that vLLM improves the throughput of popular LLMs by
  2-4times with the same level of latency compared to the state-of-the-art
  systems, such as FasterTransformer and Orca. The improvement is more pronounced
  with longer sequences, larger models, and more complex decoding algorithms.
  vLLM's source code is publicly available at
  https://github.com/vllm-project/vllm
paper_ai_keywords: key-value cache KV cache PagedAttention virtual memory paging attention algorithm vLLM LLM serving system near-zero waste flexible sharing latency FasterTransformer Orca throughput sequences models decoding algorithms