and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong
et al., 2022). The memory bandwidth from loading
keys and values can be sharply reduced through
multi-query attention (Shazeer, 2019), which uses
multiple query heads but single key and value
heads.
However, multi-query attention ( MQA ) can lead
to quality degradation and training instability, and
it may not be feasible to train separate models
optimized for quality and inference. Moreover,
while some language models already use multiquery attention, such as PaLM (Chowdhery et al.,
2022), many do not, including publicly available
language models such as T5 (Raffel et al., 2020)
and LLaMA (Touvron et al., 2023).
This work contains two contributions for faster
inference with large language models. First, we
Equal contribution.
yUniversity of Southern California. Work done at Google
Research.show that language model checkpoints with multihead attention ( MHA ) can be uptrained (Komatsuzaki et al., 2022) to use MQA with a small fraction of original training compute. This presents a
cost-effective method to obtain fast multi-query as
well as high-quality MHA checkpoints.
Second, we propose grouped-query attention
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai