International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9,
2019 . OpenReview.net.
Samuel Williams, Andrew Waterman, and David A.
Patterson. 2009. Rooﬂine: an insightful visual performance model for multicore architectures. Commun. ACM , 52(4):65–76.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang,
Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu,
Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan
Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng
Zhang, Yuxiao Dong, and Jie Tang. 2022. GLM130B: an open bilingual pre-trained model. CoRR ,
abs/2210.02414.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng.
2021. Mediasum: A large-scale media interview
dataset for dialogue summarization. In Proceedings
of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, NAACL-HLT 2021,
Online, June 6-11, 2021 , pages 5927–5934. Association for Computational Linguistics.
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai