Yaniv Leviathan, Matan Kalman, and Yossi Matias.
2022. Fast inference from transformers via speculative decoding. CoRR , abs/2211.17192.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos
Santos, Çaglar Gülçehre, and Bing Xiang. 2016.
Abstractive text summarization using sequence-tosequence rnns and beyond. In Proceedings of the
20th SIGNLL Conference on Computational Natural
Language Learning, CoNLL 2016, Berlin, Germany,
August 11-12, 2016 , pages 280–290. ACL.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery,
Jacob Devlin, James Bradbury, Anselm Levskaya,
Jonathan Heek, Kefan Xiao, Shivani Agrawal, and
Jeff Dean. 2022. Efﬁciently scaling transformer inference. arXiv preprint arXiv:2211.05102 .
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu. 2020. Exploring the limits
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai