library and ecosystem for JAX.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.
2015. Distilling the knowledge in a neural network.
CoRR , abs/1503.02531.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke
Zettlemoyer. 2017. Triviaqa: A large scale distantly
supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of
the Association for Computational Linguistics , Vancouver, Canada. Association for Computational Linguistics.
Sehoon Kim, Karttikeya Mangalam, Jitendra Malik,
Michael W. Mahoney, Amir Gholami, and Kurt
Keutzer. 2023. Big little transformer decoder.
CoRR , abs/2302.07863.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp,
Carlos Riquelme Ruiz, Basil Mustafa, Joshua
Ainslie, Yi Tay, Mostafa Dehghani, and Neil
Houlsby. 2022. Sparse upcycling: Training mixtureof-experts from dense checkpoints.
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
2022. Fast inference from transformers via speculative decoding. CoRR , abs/2211.17192.
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai