title: H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large
    Language Models
summary: Large Language Models (LLMs), despite their recent impressive
  accomplishments, are notably cost-prohibitive to deploy, particularly for
  applications involving long-content generation, such as dialogue systems and
  story writing. Often, a large amount of transient state information, referred
  to as the KV cache, is stored in GPU memory in addition to model parameters,
  scaling linearly with the sequence length and batch size. In this paper, we
  introduce a novel approach for implementing the KV cache which significantly
  reduces its memory footprint. Our approach is based on the noteworthy
  observation that a small portion of tokens contributes most of the value when
  computing attention scores. We call these tokens Heavy Hitters (H_2). Through
  a comprehensive investigation, we find that (i) the emergence of H_2 is
  natural and strongly correlates with the frequent co-occurrence of tokens in
  the text, and (ii) removing them results in significant performance
  degradation. Based on these insights, we propose Heavy Hitter Oracle (H_2O),
  a KV cache eviction policy that dynamically retains a balance of recent and
  H_2 tokens. We formulate the KV cache eviction as a dynamic submodular
  problem and prove (under mild assumptions) a theoretical guarantee for our
  novel eviction algorithm which could help guide future work. We validate the
  accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of
  tasks. Our implementation of H_2O with 20% heavy hitters improves the
  throughput over three leading inference systems DeepSpeed Zero-Inference,
  Hugging Face Accelerate, and FlexGen by up to 29times, 29times, and
  3times on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the
  latency by up to 1.9times. The code is available at
  https://github.com/FMInference/H2O.
paper_authors._id: 649a44e48f74b5aa984d8e27
paper_authors.name: Zhenyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T19:26:04.194Z
paper_authors._id: 649c888f67fd6c6aa97e5f85
paper_authors.avatarUrl: /avatars/9967b729916d1128773102797fed1673.svg
paper_authors.fullname: Zhenyu Zhang
paper_authors.type: user
paper_authors.user: Kyriection
paper_authors._id: 649a44e48f74b5aa984d8e28
paper_authors.name: Ying Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-27T12:55:27.824Z
paper_authors._id: 6455b969654d8bccae50736c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
paper_authors.fullname: Ying Sheng
paper_authors.type: user
paper_authors.user: ying1123
paper_authors._id: 649a44e48f74b5aa984d8e29
paper_authors.name: Tianyi Zhou
paper_authors._id: 649a44e48f74b5aa984d8e2a
paper_authors.name: Tianlong Chen
paper_authors._id: 649a44e48f74b5aa984d8e2b
paper_authors.name: Lianmin Zheng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-11T16:44:39.442Z
paper_authors._id: 62d35f3ceaf3858ce253ab7a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d35f3ceaf3858ce253ab7a/3yiiMriltXJBi242FZY-W.jpeg
paper_authors.fullname: Lianmin
paper_authors.type: user
paper_authors.user: lmzheng
paper_authors._id: 649a44e48f74b5aa984d8e2c
paper_authors.name: Ruisi Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:57:15.773Z
paper_authors._id: 63c99e2ddfac8071d01b89da
paper_authors.avatarUrl: /avatars/196af8c8498aa7a930e77c0a169da678.svg
paper_authors.fullname: Ruisi Cai
paper_authors.type: user
paper_authors.user: CCCCRS
paper_authors._id: 649a44e48f74b5aa984d8e2d
paper_authors.name: Zhao Song
paper_authors._id: 649a44e48f74b5aa984d8e2e
paper_authors.name: Yuandong Tian
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-28T13:57:50.681Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 649a44e48f74b5aa984d8e2f
paper_authors.name: Christopher Ré
paper_authors._id: 649a44e48f74b5aa984d8e30
paper_authors.name: Clark Barrett
paper_authors._id: 649a44e48f74b5aa984d8e31
paper_authors.name: Zhangyang Wang
paper_authors._id: 649a44e48f74b5aa984d8e32
paper_authors.name: Beidi Chen
paper_title: H_2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large
    Language Models
paper_summary: Large Language Models (LLMs), despite their recent impressive
  accomplishments, are notably cost-prohibitive to deploy, particularly for
  applications involving long-content generation, such as dialogue systems and
  story writing. Often, a large amount of transient state information, referred
  to as the KV cache, is stored in GPU memory in addition to model parameters,
  scaling linearly with the sequence length and batch size. In this paper, we
  introduce a novel approach for implementing the KV cache which significantly
  reduces its memory footprint. Our approach is based on the noteworthy
  observation that a small portion of tokens contributes most of the value when
  computing attention scores. We call these tokens Heavy Hitters (H_2). Through
  a comprehensive investigation, we find that (i) the emergence of H_2 is
  natural and strongly correlates with the frequent co-occurrence of tokens in
  the text, and (ii) removing them results in significant performance
  degradation. Based on these insights, we propose Heavy Hitter Oracle (H_2O),
  a KV cache eviction policy that dynamically retains a balance of recent and
  H_2 tokens. We formulate the KV cache eviction as a dynamic submodular
  problem and prove (under mild assumptions) a theoretical guarantee for our
  novel eviction algorithm which could help guide future work. We validate the
  accuracy of our algorithm with OPT, LLaMA, and GPT-NeoX across a wide range of
  tasks. Our implementation of H_2O with 20% heavy hitters improves the
  throughput over three leading inference systems DeepSpeed Zero-Inference,
  Hugging Face Accelerate, and FlexGen by up to 29times, 29times, and
  3times on OPT-6.7B and OPT-30B. With the same batch size, H2O can reduce the
  latency by up to 1.9times. The code is available at
  https://github.com/FMInference/H2O.
paper_ai_summary: A novel KV cache management technique, Heavy Hitter Oracle (H2O), optimizes memory usage in Large Language Models by prioritizing significant tokens, enhancing throughput and reducing latency.
paper_ai_keywords: Large Language Models KV cache Heavy Hitters Heavy Hitter Oracle attention scores submodular problem DeepSpeed Zero-Inference Hugging Face Accelerate FlexGen