Human Language Technologies, NAACL-HLT 2021,
Online, June 6-11, 2021 , pages 5927–5934. Association for Computational Linguistics.
A Training Stability
We ﬁnd that multi-query attention can lead to training instability, in particular combined with long
input tasks. We trained multiple T5-Large models with multi-query attention from scratch. In
each case, pre-training suffered from frequent loss
spikes and the ﬁnal models diverged immediately
on long-input tasks. Uptrained multi-query attention models are more stable but still display high
variance, so for multi-query models on unstable
tasks we report average performance over multiple
runs. Uptrained grouped-query attention models,
however, appear to be stable.
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai