title: Hydragen: High-Throughput LLM Inference with Shared Prefixes
summary: Transformer-based large language models (LLMs) are now deployed to hundreds
  of millions of users. LLM inference is commonly performed on batches of
  sequences that share a prefix, such as few-shot examples or a chatbot system
  prompt. Decoding in this large-batch setting can be bottlenecked by the
  attention operation, which reads large key-value (KV) caches from memory and
  computes inefficient matrix-vector products for every sequence in the batch. In
  this work, we introduce Hydragen, a hardware-aware exact implementation of
  attention with shared prefixes. Hydragen computes attention over the shared
  prefix and unique suffixes separately. This decomposition enables efficient
  prefix attention by batching queries together across sequences, reducing
  redundant memory reads and enabling the use of hardware-friendly matrix
  multiplications. Our method can improve end-to-end LLM throughput by up to 32x
  against competitive baselines, with speedup growing with the batch size and
  shared prefix length. Hydragen also enables the use of very long shared
  contexts: with a high batch size, increasing the prefix length from 1K to 16K
  tokens decreases Hydragen throughput by less than 15%, while the throughput of
  baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix
  decomposition and can be applied to tree-based prompt sharing patterns,
  allowing us to further reduce inference time on competitive programming
  problems by 55%.
paper_authors._id: 65c45954de0cb5b1c4dd1a69
paper_authors.name: Jordan Juravsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:52:33.038Z
paper_authors._id: 61a851b29c853be451cbf2ec
paper_authors.avatarUrl: /avatars/00f8f4f1a865de7bf6cec35b1726e269.svg
paper_authors.fullname: Jordan Juravsky
paper_authors.type: user
paper_authors.user: jordan-benjamin
paper_authors._id: 65c45954de0cb5b1c4dd1a6a
paper_authors.name: Bradley Brown
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T12:05:13.624Z
paper_authors._id: 60799bed489fc71534e91bf3
paper_authors.avatarUrl: /avatars/0f57ee357b29fed39f253f28e39abf6b.svg
paper_authors.fullname: Brown
paper_authors.type: user
paper_authors.user: Bradley
paper_authors._id: 65c45954de0cb5b1c4dd1a6b
paper_authors.name: Ryan Ehrlich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:53:39.236Z
paper_authors._id: 64bb0315af05eb17c79f6a2a
paper_authors.avatarUrl: /avatars/18351d9f8984d8500e9f04ea7bcbedc8.svg
paper_authors.fullname: Ryan Ehrlich
paper_authors.type: user
paper_authors.user: ryanse
paper_authors._id: 65c45954de0cb5b1c4dd1a6c
paper_authors.name: Daniel Y. Fu
paper_authors._id: 65c45954de0cb5b1c4dd1a6d
paper_authors.name: Christopher Ré
paper_authors._id: 65c45954de0cb5b1c4dd1a6e
paper_authors.name: Azalia Mirhoseini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:54:20.987Z
paper_authors._id: 6424b61d96f39ebc0415aba0
paper_authors.avatarUrl: /avatars/9687a540db73c673f80107dec5dad5bd.svg
paper_authors.fullname: Azalia Mirhoseini
paper_authors.type: user
paper_authors.user: azm1
paper_title: Hydragen: High-Throughput LLM Inference with Shared Prefixes
paper_summary: Transformer-based large language models (LLMs) are now deployed to hundreds
  of millions of users. LLM inference is commonly performed on batches of
  sequences that share a prefix, such as few-shot examples or a chatbot system
  prompt. Decoding in this large-batch setting can be bottlenecked by the
  attention operation, which reads large key-value (KV) caches from memory and
  computes inefficient matrix-vector products for every sequence in the batch. In
  this work, we introduce Hydragen, a hardware-aware exact implementation of
  attention with shared prefixes. Hydragen computes attention over the shared
  prefix and unique suffixes separately. This decomposition enables efficient
  prefix attention by batching queries together across sequences, reducing
  redundant memory reads and enabling the use of hardware-friendly matrix
  multiplications. Our method can improve end-to-end LLM throughput by up to 32x
  against competitive baselines, with speedup growing with the batch size and
  shared prefix length. Hydragen also enables the use of very long shared
  contexts: with a high batch size, increasing the prefix length from 1K to 16K
  tokens decreases Hydragen throughput by less than 15%, while the throughput of
  baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix
  decomposition and can be applied to tree-based prompt sharing patterns,
  allowing us to further reduce inference time on competitive programming
  problems by 55%.
paper_ai_keywords: Transformer-based large language models (LLMs) few-shot examples chatbot system prompt attention operation key-value (KV) caches matrix-vector products Hydragen hardware-aware exact implementation prefix attention matrix multiplications tree-based prompt sharing patterns competitive programming problems