-
-

Random
Average human rater

May 2020 GPT-3 5-shot
Mar. 2022 Chinchilla 5-shot
PaLM 5-shot
Apr. 2022
Flan-PaLM 5-shot
Oct. 2022
Flan-PaLM 5-shot: CoT + SC
Average human expert
Jun. 2023 forecast (Hypermind)
Jun. 2024 forecast (Hypermind)
Jun. 2023 forecast (Metaculus)
Jun. 2024 forecast (Metaculus)

-

25.0
34.5
43.9
67.6
69.3
72.2
75.2
89.8
73.2
75.0
82.7
87.6

Table 1: Average 5-shot MMLU scores (%) for 57 tasks with model and human accuracy comparisons
(Hendrycks et al., 2020). Forecasts were made in July 2022 by competitive human forecasters, regarding a sin-
gle model (Steinhardt, 2021); see https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=
JSAI and https://www.metaculus.com/questions/11676/mmlu-sota-in-2023-2025/. CoT + SC: chain-
of-thought prompting with self-consistency (Wang et al., 2022b).

2

Figure 2: Our ﬁnetuning data comprises 473 datasets, 146 task categories, and 1,836 total tasks. Details for
the tasks used in this paper is given in Appendix F.
Scaling Instruction-Finetuned Language Models