Humpback (this work)
WizardLLM2 [Xu et al., 2023]
Alpaca-GPT4 [Peng et al., 2023]
Vicuna [Chiang et al., 2023]
Open Assistant (OA) [Köpf et al., 2023] Human Annotation
LIMA [Zhou et al., 2023]
Alpaca [Taori et al., 2023]
FLAN v2 [Chung et al., 2022]

Human Annotation, Community QA
Distilled from ChatGPT (March 2023)
Instruction data for NLP tasks

α ↑
6.95
5.69
5.40
4.53
4.43
2.86
1.99
0.22

Table 3: Scaling coefficient α of representive instruction datasets created using differnet methods and
data sources.

6

320064001280025600Data Size45505560657075Win Ratew/o self-curation(xi,yi)(2)(xi,yi)(2)Figure 4: Comparing data efficiency of different instruction tuning datasets. The y-axis is the win
rate against text-davinci-003 when finetuning 7B LLaMa with the given instruction tuning dataset.
Dashed lines depict models that use distillation from more powerful models to construct data, and
methods with solid lines do not.
Self-AlignmentwithInstructionBacktranslation