Hydragen: High-Throughput LLM Inference with Shared Prefixes
Transformer-based large language models (LLMs) are now deployed to hundreds
of millions of users. LLM inference is commonly performed on batches of
sequences that share a prefix, such as few-shot examples or a chatbot system
prompt. Decoding in this large-batch setting can be bottlenecked by the
attention operation, which reads large key-value (KV) caches from memory and
computes inefficient matrix-vector products for every sequence in the batch. In
this work, we introduce Hydragen, a hardware-aware exact implementation of
attention with shared prefixes. Hydragen computes attention over the shared
prefix and unique suffixes separately. This decomposition enables efficient
prefix attention by batching queries together across sequences, reducing
redundant memory reads and enabling the use of hardware-friendly matrix
multiplications. Our method can improve end-to-end LLM throughput by up to 32x
against competitive baselines, with speedup growing with the batch size and
shared prefix length. Hydragen also enables the use of very long shared
contexts: with a high batch size, increasing the prefix length from 1K to 16K
tokens decreases Hydragen throughput by less than 15%, while the throughput of
baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix
decomposition and can be applied to tree-based prompt sharing patterns,
allowing us to further reduce inference time on competitive programming
problems by 55%.
65c45954de0cb5b1c4dd1a69
Jordan Juravsky
admin_assigned
2024-02-08T11:52:33.038Z
61a851b29c853be451cbf2ec
/avatars/00f8f4f1a865de7bf6cec35b1726e269.svg
Jordan Juravsky
user
jordan-benjamin
65c45954de0cb5b1c4dd1a6a
Bradley Brown
claimed_verified
2024-02-08T12:05:13.624Z
60799bed489fc71534e91bf3
/avatars/0f57ee357b29fed39f253f28e39abf6b.svg
Brown
user
Bradley
65c45954de0cb5b1c4dd1a6b
Ryan Ehrlich
admin_assigned
2024-02-08T11:53:39.236Z
64bb0315af05eb17c79f6a2a
/avatars/18351d9f8984d8500e9f04ea7bcbedc8.svg
Ryan Ehrlich
user
ryanse
65c45954de0cb5b1c4dd1a6c
Daniel Y. Fu
65c45954de0cb5b1c4dd1a6d
Christopher Ré
65c45954de0cb5b1c4dd1a6e
Azalia Mirhoseini
admin_assigned
2024-02-08T11:54:20.987Z
6424b61d96f39ebc0415aba0
/avatars/9687a540db73c673f80107dec5dad5bd.svg
Azalia Mirhoseini
user
azm1
Hydragen: High-Throughput LLM Inference with Shared Prefixes
Transformer-based large language models (LLMs) are now deployed to hundreds
of millions of users. LLM inference is commonly performed on batches of
sequences that share a prefix, such as few-shot examples or a chatbot system
prompt. Decoding in this large-batch setting can be bottlenecked by the
attention operation, which reads large key-value (KV) caches from memory and
computes inefficient matrix-vector products for every sequence in the batch. In
this work, we introduce Hydragen, a hardware-aware exact implementation of
attention with shared prefixes. Hydragen computes attention over the shared
prefix and unique suffixes separately. This decomposition enables efficient
prefix attention by batching queries together across sequences, reducing
redundant memory reads and enabling the use of hardware-friendly matrix
multiplications. Our method can improve end-to-end LLM throughput by up to 32x
against competitive baselines, with speedup growing with the batch size and
shared prefix length. Hydragen also enables the use of very long shared
contexts: with a high batch size, increasing the prefix length from 1K to 16K
tokens decreases Hydragen throughput by less than 15%, while the throughput of
baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix
decomposition and can be applied to tree-based prompt sharing patterns,
allowing us to further reduce inference time on competitive programming
problems by 55%.
Hydragen improves LLM inference efficiency by decomposing attention operations into shared prefix and unique suffix computations, reducing memory reads and accelerating matrix multiplications, achieving significant throughput improvements.
Transformer-based large language models LLMs attention operation key-value (KV) caches matrix-vector products hardware-aware implementation prefix attention matrix multiplications throughput shared contexts competitive programming problems