Improving Data Efficiency via Curating LLM-Driven Rating Systems
Jinlong Pang Jiaheng Wei Ankit Shah Zhaowei Zhu Yaxuan Wang Chen Qian Yang Liu Yujia Bao Wei Wei
Instruction tuning is critical for adapting large language models (LLMs) to downstream tasks, and recent studies have demonstrated that small amounts of human-curated data can outperform larger datasets, challenging traditional data scaling laws. While LLM-based data quality rating systems offer a cost-effective alternative to human annotation, they often suffer from inaccuracies and biases, even in powerful models like GPT-4. In this work, we introduce $DS^2$, a **D**iversity-aware **S**core curation method for **D**ata **S**election. By systematically modeling error patterns through a score transition matrix, $DS^2$ corrects LLM-based scores and promotes diversity in the selected data samples. Our approach shows that a curated subset (just 3.3\% of the original dataset) outperforms full-scale datasets (300k samples) across various machine-alignment benchmarks, and matches or surpasses human-aligned datasets such as LIMA with the same sample size (1k samples). These findings challenge conventional data scaling assumptions, highlighting that redundant, low-quality samples can degrade performance and reaffirming that ``more can be less''.
Data Selection, LLM, Instruction Tuning
We systematically analyze the inherent errors of the rated scores generated by advanced LLMs, which are typically used for data selection, and then provide a score curation method to rectify and ensure the quality of LLM rated scores.