title: SampleMix: A Sample-wise Pre-training Data Mixing Strategey by
    Coordinating Data Quality and Diversity
summary: Existing pretraining data mixing methods for large language models (LLMs)
  typically follow a domain-wise methodology, a top-down process that first
  determines domain weights and then performs uniform data sampling across each
  domain. However, these approaches neglect significant inter-domain overlaps and
  commonalities, failing to control the global diversity of the constructed
  training dataset. Further, uniform sampling within domains ignores fine-grained
  sample-specific features, potentially leading to suboptimal data distribution.
  To address these shortcomings, we propose a novel sample-wise data mixture
  approach based on a bottom-up paradigm. This method performs global
  cross-domain sampling by systematically evaluating the quality and diversity of
  each sample, thereby dynamically determining the optimal domain distribution.
  Comprehensive experiments across multiple downstream tasks and perplexity
  assessments demonstrate that SampleMix surpasses existing domain-based methods.
  Meanwhile, SampleMix requires 1.4x to 2.1x training steps to achieves the
  baselines' performance, highlighting the substantial potential of SampleMix to
  optimize pre-training data.
paper_authors._id: 67c67cf5c8d296910ca74711
paper_authors.name: Xiangyu Xi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T12:01:25.632Z
paper_authors._id: 63edb098679c2cc40abc6c2e
paper_authors.avatarUrl: /avatars/288c7229937c2c3f29fda6d17c7df2eb.svg
paper_authors.fullname: Xiangyu
paper_authors.type: user
paper_authors.user: xixy
paper_authors._id: 67c67cf5c8d296910ca74712
paper_authors.name: Deyang Kong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:40:21.910Z
paper_authors._id: 65a0aade5fafc248c2156e95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a0aade5fafc248c2156e95/S9YjJMTuKc-U1cFizqUMA.jpeg
paper_authors.fullname: DeyangKong
paper_authors.type: user
paper_authors.user: DeyangKong
paper_authors._id: 67c67cf5c8d296910ca74713
paper_authors.name: Jian Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:03:35.983Z
paper_authors._id: 6489817b2e7b004f5811b66c
paper_authors.avatarUrl: /avatars/f4a7be60c4557d0f9f6022151a653c7a.svg
paper_authors.fullname: Jam Young
paper_authors.type: user
paper_authors.user: JamYoung
paper_authors._id: 67c67cf5c8d296910ca74714
paper_authors.name: Jiawei Yang
paper_authors._id: 67c67cf5c8d296910ca74715
paper_authors.name: Zhengyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:16:14.018Z
paper_authors._id: 67b7ebf3d00e69f10cfcf551
paper_authors.avatarUrl: /avatars/8adea7ae44c459079113a690ec7da73a.svg
paper_authors.fullname: Chen Zhengyu
paper_authors.type: user
paper_authors.user: WQYC
paper_authors._id: 67c67cf5c8d296910ca74716
paper_authors.name: Wei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:16:23.635Z
paper_authors._id: 62fa0ffe0697d224219a0cb7
paper_authors.avatarUrl: /avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
paper_authors.fullname: Wei Wang
paper_authors.type: user
paper_authors.user: WeiWang
paper_authors._id: 67c67cf5c8d296910ca74717
paper_authors.name: Jingang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:16:00.303Z
paper_authors._id: 647097cbcfd57849518e656b
paper_authors.avatarUrl: /avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
paper_authors.fullname: Jingang Wang
paper_authors.type: user
paper_authors.user: bitwjg
paper_authors._id: 67c67cf5c8d296910ca74718
paper_authors.name: Xunliang Cai
paper_authors._id: 67c67cf5c8d296910ca74719
paper_authors.name: Shikun Zhang
paper_authors._id: 67c67cf5c8d296910ca7471a
paper_authors.name: Wei Ye
paper_title: SampleMix: A Sample-wise Pre-training Data Mixing Strategey by
    Coordinating Data Quality and Diversity
paper_summary: Existing pretraining data mixing methods for large language models (LLMs)
  typically follow a domain-wise methodology, a top-down process that first
  determines domain weights and then performs uniform data sampling across each
  domain. However, these approaches neglect significant inter-domain overlaps and
  commonalities, failing to control the global diversity of the constructed
  training dataset. Further, uniform sampling within domains ignores fine-grained
  sample-specific features, potentially leading to suboptimal data distribution.
  To address these shortcomings, we propose a novel sample-wise data mixture
  approach based on a bottom-up paradigm. This method performs global
  cross-domain sampling by systematically evaluating the quality and diversity of
  each sample, thereby dynamically determining the optimal domain distribution.
  Comprehensive experiments across multiple downstream tasks and perplexity
  assessments demonstrate that SampleMix surpasses existing domain-based methods.
  Meanwhile, SampleMix requires 1.4x to 2.1x training steps to achieves the
  baselines' performance, highlighting the substantial potential of SampleMix to
  optimize pre-training data.
paper_ai_summary: A sample-wise data mixing method dynamically evaluates sample quality and diversity for large language models, surpassing domain-based methods with fewer training steps.
paper_ai_keywords: large language models pretraining data mixing domain-wise methodology uniform data sampling global diversity fine-grained sample-specific features bottom-up paradigm global cross-domain sampling SampleMix