57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July
28- August 2, 2019, Volume 1: Long Papers , pages
1074–1084. Association for Computational Linguistics.
Elias Frantar, Saleh Ashkboos, Torsten Hoeﬂer, and
Dan Alistarh. 2022. GPTQ: accurate post-training
quantization for generative pre-trained transformers.
CoRR , abs/2210.17323.
Google. 2020. Proﬁle your model with cloud tpu
tools. https://cloud.google.com/tpu/docs/
cloud-tpu-tools . Accessed: 2022-11-11.
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and
Dacheng Tao. 2021. Knowledge distillation: A survey. Int. J. Comput. Vis. , 129(6):1789–1819.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner,
and Marc van Zee. 2020. Flax: A neural network
library and ecosystem for JAX.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.
2015. Distilling the knowledge in a neural network.
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai