COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
<|reference_start|>COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning: Recently, there have been significant advancements in large language models (LLMs), particularly focused on the English language. These advancements have enabled these LLMs to understand and execute complex instructions with unprecedented accuracy and fluency. However, despite these advancements, there remains a noticeable gap in the development of Chinese instruction tuning. The unique linguistic features and cultural depth of the Chinese language pose challenges for instruction tuning tasks. Existing datasets are either derived from English-centric LLMs or are ill-suited for aligning with the interaction patterns of real-world Chinese users. To bridge this gap, we introduce COIG-CQIA, a high-quality Chinese instruction tuning dataset. Our aim is to build a diverse, wide-ranging instruction-tuning dataset to better align model behavior with human interactions. To this end, we collect a high-quality human-written corpus from various sources on the Chinese Internet, including Q&A communities, Wikis, examinations, and existing NLP datasets. This corpus was rigorously filtered and carefully processed to form the COIG-CQIA dataset. Furthermore, we train models of various scales on different subsets of CQIA, following in-depth evaluation and analyses. The findings from our experiments offer valuable insights for selecting and developing Chinese instruction-tuning datasets. We also find that models trained on CQIA-Subset achieve competitive results in human assessment as well as knowledge and security benchmarks. Data are available at https://huggingface.co/datasets/m-a-p/COIG-CQIA<|reference_end|>
@article{bai2024coig-cqia:,
    title={COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning},
    author={Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Junting Zhou,
  Ziqiang Liu, Feiteng Fang, Mingshan Chang, Tianyu Zheng, Xincheng Zhang, Nuo
  Ma, Zekun Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun
  Zhang, Chenghua Lin, Jie Fu, Min Yang, Shiwen Ni, Ge Zhang},
    journal={arXiv preprint arXiv:2403.18058},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.18058},
    primaryClass={cs.CL cs.AI}
}