and PubMed (Cohan et al., 2018), MediaSum (Zhu
et al., 2021), and Multi-News (Fabbri et al., 2019);
translation dataset WMT 2014 English-to-German;
and question answering dataset TriviaQA (Joshi
et al., 2017). We do not evaluate on popular classiﬁcation benchmarks such as GLUE (Wang et al.,
2019) as autoregressive inference is less applicable
for those tasks.
1https://github.com/google/flaxformer
Model Tinfer Average CNN arXiv PubMed MediaSum MultiNews WMT TriviaQA
s R1 R1 R1 R1 R1 BLEU F1
MHA-Large 0.37 46.0 42.9 44.6 46.2 35.5 46.6 27.7 78.2
MHA-XXL 1.51 47.2 43.8 45.6 47.5 36.4 46.9 28.4 81.9
MQA-XXL 0.24 46.5 43.0 45.0 47.2 36.1 46.5 28.5 81.3
GQA-8-XXL 0.28 47.1 43.5 45.4 47.7 36.3 47.2 28.4 81.6
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai