SampleMix: A Sample-wise Pre-training Data Mixing Strategey by
  Coordinating Data Quality and Diversity
Existing pretraining data mixing methods for large language models (LLMs)
typically follow a domain-wise methodology, a top-down process that first
determines domain weights and then performs uniform data sampling across each
domain. However, these approaches neglect significant inter-domain overlaps and
commonalities, failing to control the global diversity of the constructed
training dataset. Further, uniform sampling within domains ignores fine-grained
sample-specific features, potentially leading to suboptimal data distribution.
To address these shortcomings, we propose a novel sample-wise data mixture
approach based on a bottom-up paradigm. This method performs global
cross-domain sampling by systematically evaluating the quality and diversity of
each sample, thereby dynamically determining the optimal domain distribution.
Comprehensive experiments across multiple downstream tasks and perplexity
assessments demonstrate that SampleMix surpasses existing domain-based methods.
Meanwhile, SampleMix requires 1.4x to 2.1x training steps to achieves the
baselines' performance, highlighting the substantial potential of SampleMix to
optimize pre-training data.
67c67cf5c8d296910ca74711
Xiangyu Xi
claimed_verified
2025-03-04T12:01:25.632Z
63edb098679c2cc40abc6c2e
/avatars/288c7229937c2c3f29fda6d17c7df2eb.svg
Xiangyu
user
xixy
67c67cf5c8d296910ca74712
Deyang Kong
claimed_verified
2025-03-04T08:40:21.910Z
65a0aade5fafc248c2156e95
https://cdn-avatars.huggingface.co/v1/production/uploads/65a0aade5fafc248c2156e95/S9YjJMTuKc-U1cFizqUMA.jpeg
DeyangKong
user
DeyangKong
67c67cf5c8d296910ca74713
Jian Yang
claimed_verified
2025-03-10T08:03:35.983Z
6489817b2e7b004f5811b66c
/avatars/f4a7be60c4557d0f9f6022151a653c7a.svg
Jam Young
user
JamYoung
67c67cf5c8d296910ca74714
Jiawei Yang
67c67cf5c8d296910ca74715
Zhengyu Chen
admin_assigned
2025-03-04T11:16:14.018Z
67b7ebf3d00e69f10cfcf551
/avatars/8adea7ae44c459079113a690ec7da73a.svg
Chen Zhengyu
user
WQYC
67c67cf5c8d296910ca74716
Wei Wang
admin_assigned
2025-03-04T11:16:23.635Z
62fa0ffe0697d224219a0cb7
/avatars/f0ef59e1c0cf4ab4fe5cee08d488bd03.svg
Wei Wang
user
WeiWang
67c67cf5c8d296910ca74717
Jingang Wang
admin_assigned
2025-03-04T11:16:00.303Z
647097cbcfd57849518e656b
/avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
Jingang Wang
user
bitwjg
67c67cf5c8d296910ca74718
Xunliang Cai
67c67cf5c8d296910ca74719
Shikun Zhang
67c67cf5c8d296910ca7471a
Wei Ye
SampleMix: A Sample-wise Pre-training Data Mixing Strategey by
  Coordinating Data Quality and Diversity
Existing pretraining data mixing methods for large language models (LLMs)
typically follow a domain-wise methodology, a top-down process that first
determines domain weights and then performs uniform data sampling across each
domain. However, these approaches neglect significant inter-domain overlaps and
commonalities, failing to control the global diversity of the constructed
training dataset. Further, uniform sampling within domains ignores fine-grained
sample-specific features, potentially leading to suboptimal data distribution.
To address these shortcomings, we propose a novel sample-wise data mixture
approach based on a bottom-up paradigm. This method performs global
cross-domain sampling by systematically evaluating the quality and diversity of
each sample, thereby dynamically determining the optimal domain distribution.
Comprehensive experiments across multiple downstream tasks and perplexity
assessments demonstrate that SampleMix surpasses existing domain-based methods.
Meanwhile, SampleMix requires 1.4x to 2.1x training steps to achieves the
baselines' performance, highlighting the substantial potential of SampleMix to
optimize pre-training data.
A sample-wise data mixing method dynamically evaluates sample quality and diversity for large language models, surpassing domain-based methods with fewer training steps.
large language models pretraining data mixing domain-wise methodology uniform data sampling global diversity fine-grained sample-specific features bottom-up paradigm global cross-domain sampling SampleMix