Efficient Memory Management for Large Language Model Serving with
  PagedAttention
High throughput serving of large language models (LLMs) requires batching
sufficiently many requests at a time. However, existing systems struggle
because the key-value cache (KV cache) memory for each request is huge and
grows and shrinks dynamically. When managed inefficiently, this memory can be
significantly wasted by fragmentation and redundant duplication, limiting the
batch size. To address this problem, we propose PagedAttention, an attention
algorithm inspired by the classical virtual memory and paging techniques in
operating systems. On top of it, we build vLLM, an LLM serving system that
achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV
cache within and across requests to further reduce memory usage. Our
evaluations show that vLLM improves the throughput of popular LLMs by
2-4times with the same level of latency compared to the state-of-the-art
systems, such as FasterTransformer and Orca. The improvement is more pronounced
with longer sequences, larger models, and more complex decoding algorithms.
vLLM's source code is publicly available at
https://github.com/vllm-project/vllm
650114d996ec8a5218886621
Woosuk Kwon
admin_assigned
2023-09-13T07:28:58.381Z
6458733caa426bae5e079b7b
/avatars/990d0c21e28f973290ce429f62156031.svg
Woosuk Kwon
user
wskwon
650114d996ec8a5218886622
Zhuohan Li
admin_assigned
2023-09-13T07:29:05.538Z
649dd1fae16abbd8fd7a3bdf
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/2t1p0Ku7qmNcSapRdzxv9.jpeg
Zhuohan  Li
user
zhuohan123
650114d996ec8a5218886623
Siyuan Zhuang
650114d996ec8a5218886624
Ying Sheng
admin_assigned
2023-09-13T07:29:40.177Z
6455b969654d8bccae50736c
https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
Ying Sheng
user
ying1123
650114d996ec8a5218886625
Lianmin Zheng
650114d996ec8a5218886626
Cody Hao Yu
claimed_verified
2023-09-14T07:31:46.459Z
64763d68288784ce9aba1cc7
/avatars/84033af4466f5469c6264e7bff9f6819.svg
Cody Yu
user
comaniac
650114d996ec8a5218886627
Joseph E. Gonzalez
admin_assigned
2023-09-13T07:30:30.707Z
645d2e8401f4eaab2a0878ce
/avatars/1273c5fb607b4b622a746a42692fa632.svg
Joseph E. Gonzalez
user
ProfJoeyG
650114d996ec8a5218886628
Hao Zhang
claimed_verified
2025-01-01T20:15:04.984Z
62d363143eebd640a4fa41fa
https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
Hao Zhang
user
zhisbug
650114d996ec8a5218886629
Ion Stoica
Efficient Memory Management for Large Language Model Serving with
  PagedAttention
High throughput serving of large language models (LLMs) requires batching
sufficiently many requests at a time. However, existing systems struggle
because the key-value cache (KV cache) memory for each request is huge and
grows and shrinks dynamically. When managed inefficiently, this memory can be
significantly wasted by fragmentation and redundant duplication, limiting the
batch size. To address this problem, we propose PagedAttention, an attention
algorithm inspired by the classical virtual memory and paging techniques in
operating systems. On top of it, we build vLLM, an LLM serving system that
achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV
cache within and across requests to further reduce memory usage. Our
evaluations show that vLLM improves the throughput of popular LLMs by
2-4times with the same level of latency compared to the state-of-the-art
systems, such as FasterTransformer and Orca. The improvement is more pronounced
with longer sequences, larger models, and more complex decoding algorithms.
vLLM's source code is publicly available at
https://github.com/vllm-project/vllm
PagedAttention algorithm and vLLM system enhance the throughput of large language models by efficiently managing memory and reducing waste in the key-value cache.
PagedAttention key-value cache KV cache vLLM virtual memory paging techniques attention algorithm memory management throughput improvement FasterTransformer Orca