effect of the number of GQA groups on inference speed. For larger models the memory bandwidth overhead from the KV cache is less constraining (Shazeer, 2019), while the reduction in
key-value size is sharper due to the increased number of heads. As a result, increasing the number
of groups from MQA only results in modest slowdowns initially, with increasing cost as we move
closer to MHA . We selected 8 groups as a favorable middle ground.
4 Related Work
This work is focused on achieving a better tradeoff between decoder quality and inference time
through reducing the memory bandwidth overhead (Williams et al., 2009) from loading keys and
values. Shazeer (2019) ﬁrst proposed reducing this
overhead through multi-query attention. Follow-up
work showed that multi-query attention is especially helpful for long inputs (Pope et al., 2022;
de Jong et al., 2022).
A number of other methods have been proposed
to reduce memory bandwidth overhead from keys
and values, as well as parameters. Flash atten-1 4 8 16 32 6412
GQA groupsTime per sample (s)MHA
GQA
MQA
Figure 6: Time per sample for GQA-XXL as a function of the number of GQA groups with input length
2048 and output length 512. Going from 1 (MQA) to
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai