The converted checkpoint is then pre-trained forarXiv:2305.13245v1  [cs.CL]  22 May 2023
Figure 2: Overview of grouped-query method. Multi-head attention has H query, key, and value heads. Multi-query
attention shares single key and value heads across all query heads. Grouped-query attention instead shares single
key and value heads for each group of query heads, interpolating between multi-head and multi-query attention.
a small proportion of its original training steps
on the same pre-training recipe.
2.2 Grouped-query attention
Grouped-query attention divides query heads into
Ggroups , each of which shares a single key head
and value head. GQA -Grefers to grouped-query
withGgroups. GQA -1, with a single group and
therefore single key and value head, is equivalent to
MQA , while GQA -H, with groups equal to number
of heads, is equivalent to MHA . Figure 2 shows a
comparison of grouped-query attention and multihead/multi-query attention. When converting a
multi-head checkpoint to a GQA checkpoint, we
construct each group key and value head by meanpooling all the original heads within that group.
An intermediate number of groups leads to an
Multi-query attention (MQA), which only uses a single key-value head,
drastically speeds up decoder inference. However, MQA can lead to quality
degradation, and moreover it may not be desirable to train a separate model
just for faster inference. We (1) propose a recipe for uptraining existing
multi-head language model checkpoints into models with MQA using 5% of original
pre-training compute, and (2) introduce grouped-query attention (GQA), a
generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number of key-value heads. We show that
uptrained GQA achieves quality close to multi-head attention with comparable
speed to MQA.
Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebrón Sumit Sanghai