is inspired by Komatsuzaki et al. (2022), which
uptrains standard T5 checkpoints into sparsely activated Mixture-of-Experts models.
5 Conclusion
Language models are expensive for inference primarily due to the memory bandwidth overhead
from loading keys and values. Multi-query attention reduces this overhead at the cost of decreased
model capacity and quality. We propose to convert
multi-head attention models to multi-query models
with a small fraction of original pre-training compute. Moreover, we introduce grouped-query attention, an interpolation of multi-query and multi-head
attention that achieves quality close to multi-head
at comparable speed to multi-query attention.
Acknowlegements
We thank Santiago Ontañón, Afroz Mohiuddin,
William Cohen and others at Google Research for
insightful advice and discussion.
References
James Bradbury, Roy Frostig, Peter Hawkins,
Matthew James Johnson, Chris Leary, Dougal
Maclaurin, George Necula, Adam Paszke, Jake
VanderPlas, Skye Wanderman-Milne, and Qiao
Zhang. 2018. JAX: composable transformations of
Python+NumPy programs.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and
John Jumper. 2023. Accelerating large language
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai