We also found that improving the architecture and a more varied model objective was important in performance gains.
Finally, we ﬁnd that the data mixture is a critical component of the ﬁnal model. At this scale, even though translation
pairs were a minor part of the mixture, it allowed the model to be on par with production translation services. We thus
ﬁnd that it is more beneﬁcial to invest more compute in training a smaller model compared to modifying a model’s
architecture to be more inference-efﬁcient. In effect, we ﬁnd that it is generally more efﬁcient to train a smaller model
with more tokens, for a ﬁxed inference and training budget.

We believe that further scaling of both model parameters and dataset size and quality as well as improvements in the
architecture and objective will continue to yield gains in language understanding and generation.

27
PaLM 2 Technical Report