title: QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM
    Pretraining
summary: Quality and diversity are two critical metrics for the training data of large
  language models (LLMs), positively impacting performance. Existing studies
  often optimize these metrics separately, typically by first applying quality
  filtering and then adjusting data proportions. However, these approaches
  overlook the inherent trade-off between quality and diversity, necessitating
  their joint consideration. Given a fixed training quota, it is essential to
  evaluate both the quality of each data point and its complementary effect on
  the overall dataset. In this paper, we introduce a unified data selection
  framework called QuaDMix, which automatically optimizes the data distribution
  for LLM pretraining while balancing both quality and diversity. Specifically,
  we first propose multiple criteria to measure data quality and employ domain
  classification to distinguish data points, thereby measuring overall diversity.
  QuaDMix then employs a unified parameterized data sampling function that
  determines the sampling probability of each data point based on these quality
  and diversity related labels. To accelerate the search for the optimal
  parameters involved in the QuaDMix framework, we conduct simulated experiments
  on smaller models and use LightGBM for parameters searching, inspired by the
  RegMix method. Our experiments across diverse models and datasets demonstrate
  that QuaDMix achieves an average performance improvement of 7.2% across
  multiple benchmarks. These results outperform the independent strategies for
  quality and diversity, highlighting the necessity and ability to balance data
  quality and diversity.
paper_authors._id: 680b2a95c94724c1465c20dd
paper_authors.name: Fengze Liu
paper_authors._id: 680b2a95c94724c1465c20de
paper_authors.name: Weidong Zhou
paper_authors._id: 680b2a95c94724c1465c20df
paper_authors.name: Binbin Liu
paper_authors._id: 680b2a95c94724c1465c20e0
paper_authors.name: Zhimiao Yu
paper_authors._id: 680b2a95c94724c1465c20e1
paper_authors.name: Yifan Zhang
paper_authors._id: 680b2a95c94724c1465c20e2
paper_authors.name: Haobin Lin
paper_authors._id: 680b2a95c94724c1465c20e3
paper_authors.name: Yifeng Yu
paper_authors._id: 680b2a95c94724c1465c20e4
paper_authors.name: Xiaohuan Zhou
paper_authors._id: 680b2a95c94724c1465c20e5
paper_authors.name: Taifeng Wang
paper_authors._id: 680b2a95c94724c1465c20e6
paper_authors.name: Yong Cao
paper_title: QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM
    Pretraining
paper_summary: Quality and diversity are two critical metrics for the training data of large
  language models (LLMs), positively impacting performance. Existing studies
  often optimize these metrics separately, typically by first applying quality
  filtering and then adjusting data proportions. However, these approaches
  overlook the inherent trade-off between quality and diversity, necessitating
  their joint consideration. Given a fixed training quota, it is essential to
  evaluate both the quality of each data point and its complementary effect on
  the overall dataset. In this paper, we introduce a unified data selection
  framework called QuaDMix, which automatically optimizes the data distribution
  for LLM pretraining while balancing both quality and diversity. Specifically,
  we first propose multiple criteria to measure data quality and employ domain
  classification to distinguish data points, thereby measuring overall diversity.
  QuaDMix then employs a unified parameterized data sampling function that
  determines the sampling probability of each data point based on these quality
  and diversity related labels. To accelerate the search for the optimal
  parameters involved in the QuaDMix framework, we conduct simulated experiments
  on smaller models and use LightGBM for parameters searching, inspired by the
  RegMix method. Our experiments across diverse models and datasets demonstrate
  that QuaDMix achieves an average performance improvement of 7.2% across
  multiple benchmarks. These results outperform the independent strategies for
  quality and diversity, highlighting the necessity and ability to balance data
  quality and diversity.
paper_ai_summary: A unified data selection framework called QuaDMix optimizes the distribution of training data for large language models by balancing quality and diversity, leading to improved performance.
paper_ai_keywords: data selection framework QuaDMix data distribution large language models data quality domain classification data diversity parameterized data sampling function LightGBM RegMix performance improvement