Timing We report time per sample per TPUv4
chip, as measured by xprof (Google, 2020). For
timing experiments we use 8 TPUs with the largest
batch size that ﬁts up to 32 per TPU, and parallelization optimized separately for each model.
0 0:5 1 1:54646:547
MHA-LargeMHA-XXL
MQA-XXLGQA-XXL
Time per sample (ms)Performance
Figure 3: Uptrained MQA yields a favorable tradeoff compared to MHA with higher quality and
faster speed than MHA-Large, and GQA achieves
even better performance with similar speed gains
and comparable quality to MHA-XXL. Average performance on all tasks as a function of average inference
time per sample for T5-Large and T5-XXL with multihead attention, and 5% uptrained T5-XXL with MQA
and GQA-8 attention.3.2 Main results
Figure 3 shows average performance over all
datasets as a function of average inference time
forMHA T5-Large and T5-XXL, and uptrained
MQA andGQA -8XXL models with uptraining
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai