MAmmoTH2: Scaling Instructions from the Web
<|reference_start|>MAmmoTH2: Scaling Instructions from the Web: Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B's (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.<|reference_end|>
@article{yue2024mammoth2:,
    title={MAmmoTH2: Scaling Instructions from the Web},
    author={Xiang Yue and Tuney Zheng and Ge Zhang and Wenhu Chen},
    journal={arXiv preprint arXiv:2405.03548},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.03548},
    primaryClass={cs.CL}
}