key and value heads, ‘First’ selects the ﬁrst head and
‘Random’ initializes heads from scratch.
Checkpoint conversion Figure 4 compares the
performance of different methods for checkpoint
conversion. Mean pooling appears to work best,
followed by selecting a single head and then random initialization. Intuitively, results are ordered
by the degree to which information is preserved
from the pre-trained model.
Uptraining steps Figure 5 shows how performance varies with uptraining proportion for T5
XXL with MQA andGQA . First, we note that
GQA already achieves reasonable performance after conversion while MQA requires uptraining to
be useful. Both MQA andGQA gain from 5%
uptraining with diminishing returns from 10%.
0 0:02 0:04 0:06 0:08 0:154555657
Uptraining proportion PerformanceMHA
GQA
MQA
Figure 5: Performance as a function of uptraining proportion for T5 XXL models with MQA and GQA-8.
Number of groups Figure 6 demonstrates the
effect of the number of GQA groups on inference speed. For larger models the memory bandwidth overhead from the KV cache is less constraining (Shazeer, 2019), while the reduction in
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai