QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM
  Pretraining
Quality and diversity are two critical metrics for the training data of large
language models (LLMs), positively impacting performance. Existing studies
often optimize these metrics separately, typically by first applying quality
filtering and then adjusting data proportions. However, these approaches
overlook the inherent trade-off between quality and diversity, necessitating
their joint consideration. Given a fixed training quota, it is essential to
evaluate both the quality of each data point and its complementary effect on
the overall dataset. In this paper, we introduce a unified data selection
framework called QuaDMix, which automatically optimizes the data distribution
for LLM pretraining while balancing both quality and diversity. Specifically,
we first propose multiple criteria to measure data quality and employ domain
classification to distinguish data points, thereby measuring overall diversity.
QuaDMix then employs a unified parameterized data sampling function that
determines the sampling probability of each data point based on these quality
and diversity related labels. To accelerate the search for the optimal
parameters involved in the QuaDMix framework, we conduct simulated experiments
on smaller models and use LightGBM for parameters searching, inspired by the
RegMix method. Our experiments across diverse models and datasets demonstrate
that QuaDMix achieves an average performance improvement of 7.2% across
multiple benchmarks. These results outperform the independent strategies for
quality and diversity, highlighting the necessity and ability to balance data
quality and diversity.
680b2a95c94724c1465c20dd
Fengze Liu
680b2a95c94724c1465c20de
Weidong Zhou
680b2a95c94724c1465c20df
Binbin Liu
680b2a95c94724c1465c20e0
Zhimiao Yu
680b2a95c94724c1465c20e1
Yifan Zhang
680b2a95c94724c1465c20e2
Haobin Lin
680b2a95c94724c1465c20e3
Yifeng Yu
680b2a95c94724c1465c20e4
Xiaohuan Zhou
claimed_verified
2025-06-26T09:33:42.461Z
668f5875b5b3081d776e4094
/avatars/8c763393f25afbe5fb8b132f775e746a.svg
Xiaohuan Zhou
user
XiaohuanZhou
680b2a95c94724c1465c20e5
Taifeng Wang
680b2a95c94724c1465c20e6
Yong Cao
QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM
  Pretraining
Quality and diversity are two critical metrics for the training data of large
language models (LLMs), positively impacting performance. Existing studies
often optimize these metrics separately, typically by first applying quality
filtering and then adjusting data proportions. However, these approaches
overlook the inherent trade-off between quality and diversity, necessitating
their joint consideration. Given a fixed training quota, it is essential to
evaluate both the quality of each data point and its complementary effect on
the overall dataset. In this paper, we introduce a unified data selection
framework called QuaDMix, which automatically optimizes the data distribution
for LLM pretraining while balancing both quality and diversity. Specifically,
we first propose multiple criteria to measure data quality and employ domain
classification to distinguish data points, thereby measuring overall diversity.
QuaDMix then employs a unified parameterized data sampling function that
determines the sampling probability of each data point based on these quality
and diversity related labels. To accelerate the search for the optimal
parameters involved in the QuaDMix framework, we conduct simulated experiments
on smaller models and use LightGBM for parameters searching, inspired by the
RegMix method. Our experiments across diverse models and datasets demonstrate
that QuaDMix achieves an average performance improvement of 7.2% across
multiple benchmarks. These results outperform the independent strategies for
quality and diversity, highlighting the necessity and ability to balance data
quality and diversity.
A unified data selection framework called QuaDMix optimizes the distribution of training data for large language models by balancing quality and diversity, leading to improved performance.
data selection framework QuaDMix data distribution large language models data quality domain classification data diversity parameterized data sampling function LightGBM RegMix performance improvement