formance by a large margin for all model types. For T5 models without instruction ﬁnetuning, we use
LM-adapted models, which were produced by training T5 on 100B additional tokens from C4 on a standard
language modeling objective (Lester et al., 2021). Given the diﬃculty of our evaluation benchmarks and the
fact that T5 is not multilingual, T5 models beneﬁted the most from instruction ﬁnetuning compared with
their non-ﬁnetuned models. These results were quite strong for some benchmarks—for example, Flan-T5-XL
is only 3B parameters and achieves a MMLU score of 52.4%, surpassing GPT-3 175B’s score of 43.9%. As
another highlight, the strongest overall model we achieve in this paper combines instruction ﬁnetuning
with UL2 continued pre-training that was used in the U-PaLM model. This result shows that instruction
ﬁnetuning and UL2 continued pre-training are complementary compute-eﬃcient methods to improve the
Scaling Instruction-Finetuned Language Models