We show downstream results of differently-sized models in Table 15 in the Appendix. The results suggest that the
optimal number of parameters for a model with 1× 1022 FLOPs is 10B. However, the training loss is not a perfect proxy
for downstream metrics. For example, the 8.95B model, which shows the lowest loss (Table 1) and is closest to the
optimal model, slightly underperforms the 14.7B model on downstream tasks. This suggests that while scaling laws can
be used to achieve optimal training loss for a given quantity of FLOPs, this does not necessarily transfer to achieving
optimal performance for a given task. Moreover, there are several other considerations besides the optimal training loss,
such as training throughput and serving latency, which affect the decision regarding the optimal model size.

7

Figure 4: IsoFLOP curves from which we extract the optimal parameters at each compute scale, using a quadratic ﬁt.

Figure 5: The scaling law obtained from all 4 compute scales.

8
PaLM 2 Technical Report