5.3.2. Architecture changes to improve sampling speed
Because drawing more samples is important for improving performance, architecture changes that
increase sampling speed would also increase the overall solve rate within a certain compute budget.
Therefore, we made two architecture decisions: using (1) an encoder-decoder architecture with
asymmetric encoder and decoder structures and (2) the multi-query attention setup from Shazeer
(2019) which uses one shared attention head for keys and one for values each block.
To investigate the eﬀects of these decisions, we compared our base 1B parameter model against
the two alternatives that remove each of the changes. We pre-trained and ﬁne-tuned the standard
multi-head attention model in exactly the same way as our base 1B model. The decoder-only model
was trained with the same amount of compute. However, due to the signiﬁcantly longer decoder
sequence length (2304 tokens), with the same amount of training compute it consumes 50% more
alphacode