H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large
  Language Models
Large Language Models (LLMs), despite their recent impressive
accomplishments, are notably cost-prohibitive to deploy, particularly for
applications involving long-content generation, such as dialogue systems and
story writing. Often, a large amount of transient state information, referred
to as the KV cache, is stored in GPU memory in addition to model parameters,
scaling linearly with the sequence length and batch size. In this paper, we
introduce a novel approach for implementing the KV cache which significantly
reduces its memory footprint. Our approach is based on the noteworthy
observation that a small portion of tokens contributes most of the value when
computing attention scores. We call these tokens Heavy Hitters (H_2). Through
a comprehensive investigation, we find that (i) the emergence of H_2 is
natural and strongly correlates with the frequent co-occurrence of tokens in
the text, and (ii) removing them results in significant performance
degradation. Based on these insights, we propose Heavy Hitter Oracle (H_2O),
a KV cache eviction policy that dynamically retains a balance of recent and
H_2 tokens. We formulate the KV cache eviction as a dynamic submodular
problem and prove (under mild assumptions) a theoretical guarantee for our
novel eviction algorithm which could help guide future work. We validate the
accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of
tasks. Our implementation of H_2O with 20% heavy hitters improves the
throughput over three leading inference systems DeepSpeed Zero-Inference,
Hugging Face Accelerate, and FlexGen by up to 29times, 29times, and
3times on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the
latency by up to 1.9times. The code is available at
https://github.com/FMInference/H2O.
649a44e48f74b5aa984d8e27
Zhenyu Zhang
claimed_verified
2023-06-28T19:26:04.194Z
649c888f67fd6c6aa97e5f85
/avatars/9967b729916d1128773102797fed1673.svg
Zhenyu Zhang
user
Kyriection
649a44e48f74b5aa984d8e28
Ying Sheng
claimed_verified
2023-06-27T12:55:27.824Z
6455b969654d8bccae50736c
https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
Ying Sheng
user
ying1123
649a44e48f74b5aa984d8e29
Tianyi Zhou
649a44e48f74b5aa984d8e2a
Tianlong Chen
649a44e48f74b5aa984d8e2b
Lianmin Zheng
extracted_confirmed
2023-09-11T16:44:39.442Z
62d35f3ceaf3858ce253ab7a
https://cdn-avatars.huggingface.co/v1/production/uploads/62d35f3ceaf3858ce253ab7a/3yiiMriltXJBi242FZY-W.jpeg
Lianmin
user
lmzheng
649a44e48f74b5aa984d8e2c
Ruisi Cai
claimed_verified
2024-10-30T09:57:15.773Z
63c99e2ddfac8071d01b89da
/avatars/196af8c8498aa7a930e77c0a169da678.svg
Ruisi Cai
user
CCCCRS
649a44e48f74b5aa984d8e2d
Zhao Song
649a44e48f74b5aa984d8e2e
Yuandong Tian
extracted_confirmed
2023-06-28T13:57:50.681Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
649a44e48f74b5aa984d8e2f
Christopher Ré
649a44e48f74b5aa984d8e30
Clark Barrett
649a44e48f74b5aa984d8e31
Zhangyang Wang
649a44e48f74b5aa984d8e32
Beidi Chen
H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large
  Language Models
Large Language Models (LLMs), despite their recent impressive
accomplishments, are notably cost-prohibitive to deploy, particularly for
applications involving long-content generation, such as dialogue systems and
story writing. Often, a large amount of transient state information, referred
to as the KV cache, is stored in GPU memory in addition to model parameters,
scaling linearly with the sequence length and batch size. In this paper, we
introduce a novel approach for implementing the KV cache which significantly
reduces its memory footprint. Our approach is based on the noteworthy
observation that a small portion of tokens contributes most of the value when
computing attention scores. We call these tokens Heavy Hitters (H_2). Through
a comprehensive investigation, we find that (i) the emergence of H_2 is
natural and strongly correlates with the frequent co-occurrence of tokens in
the text, and (ii) removing them results in significant performance
degradation. Based on these insights, we propose Heavy Hitter Oracle (H_2O),
a KV cache eviction policy that dynamically retains a balance of recent and
H_2 tokens. We formulate the KV cache eviction as a dynamic submodular
problem and prove (under mild assumptions) a theoretical guarantee for our
novel eviction algorithm which could help guide future work. We validate the
accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of
tasks. Our implementation of H_2O with 20% heavy hitters improves the
throughput over three leading inference systems DeepSpeed Zero-Inference,
Hugging Face Accelerate, and FlexGen by up to 29times, 29times, and
3times on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the
latency by up to 1.9times. The code is available at
https://github.com/FMInference/H2O.
A novel KV cache management technique, Heavy Hitter Oracle (H2O), optimizes memory usage in Large Language Models by prioritizing significant tokens, enhancing throughput and reducing latency.
Large Language Models KV cache Heavy Hitters Heavy Hitter Oracle attention scores submodular problem DeepSpeed Zero-Inference Hugging Face Accelerate FlexGen