% Finetune
Compute
1.6%
1.4%
1.1%
0.6%
0.2%
0.4%
0.4%
0.2%
0.4%
0.2%

Table 2: Across several models, instruction ﬁnetuning only costs a small amount of compute relative to
pre-training. T5: Raﬀel et al. (2020). PaLM and cont-PaLM (also known as PaLM 62B at 1.3T tokens):
Chowdhery et al. (2022). U-PaLM: Tay et al. (2022b).
Scaling Instruction-Finetuned Language Models