Leveraging Web-Crawled Data for High-Quality Fine-Tuning
<|reference_start|>Leveraging Web-Crawled Data for High-Quality Fine-Tuning: Most large language models are fine-tuned using either expensive human-annotated data or GPT-4 generated data which cannot guarantee performance in certain domains. We argue that although the web-crawled data often has formatting errors causing semantic inaccuracies, it can still serve as a valuable source for high-quality supervised fine-tuning in specific domains without relying on advanced models like GPT-4. To this end, we create a paired training dataset automatically by aligning web-crawled data with a smaller set of high-quality data. By training a language model on this dataset, we can convert web data with irregular formats into high-quality ones. Our experiments show that training with the model-transformed data yields better results, surpassing training with only high-quality data by an average score of 9.4% in Chinese math problems. Additionally, our 7B model outperforms several open-source models larger than 32B and surpasses well-known closed-source models such as GPT-3.5, highlighting the efficacy of our approach.<|reference_end|>
@article{zhou2024leveraging,
    title={Leveraging Web-Crawled Data for High-Quality Fine-Tuning},
    author={Jing Zhou, Chenglin Jiang, Wei Shen, Xiao Zhou, Xiaonan He},
    journal={arXiv preprint arXiv:2408.08003},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.08003},
    primaryClass={cs.CL}
}