title: RegMix: Data Mixture as Regression for Language Model Pre-training
summary: The data mixture for large language model pre-training significantly impacts
  performance, yet how to determine an effective mixture remains unclear. We
  propose RegMix to automatically identify a high-performing data mixture by
  formulating it as a regression task. RegMix involves training a set of small
  models with diverse data mixtures and fitting a regression model to predict
  their performance given their respective mixtures. With the fitted regression
  model, we simulate the top-ranked mixture and use it to train a large-scale
  model with orders of magnitude more compute. To empirically validate RegMix, we
  train 512 models with 1M parameters for 1B tokens of different mixtures to fit
  the regression model and find the optimal mixture. Using this mixture we train
  a 1B parameter model for 25B tokens (i.e. 1000x larger and 25x longer) which we
  find performs best among 64 candidate 1B parameter models with other mixtures.
  Further, our method demonstrates superior performance compared to human
  selection and achieves results that match or surpass DoReMi, while utilizing
  only 10% of the compute budget. Our experiments also show that (1) Data
  mixtures significantly impact performance with single-task performance
  variations of up to 14.6%; (2) Web corpora rather than data perceived as
  high-quality like Wikipedia have the strongest positive correlation with
  downstream performance; (3) Domains interact in complex ways often
  contradicting common sense, thus automatic approaches like RegMix are needed;
  (4) Data mixture effects transcend scaling laws, and our approach captures the
  complexity by considering all domains together. Our code is available at
  https://github.com/sail-sg/regmix.
paper_authors._id: 668375662ca1c52c2744c833
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:41.749Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 668375662ca1c52c2744c834
paper_authors.name: Xiaosen Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:44.083Z
paper_authors._id: 6274a2315d12b3a734adebc9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6274a2315d12b3a734adebc9/zLQDbszAvWh0F2BjdKock.jpeg
paper_authors.fullname: Xiaosen Zheng
paper_authors.type: user
paper_authors.user: xszheng2020
paper_authors._id: 668375662ca1c52c2744c835
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:26:53.541Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 668375662ca1c52c2744c836
paper_authors.name: Guangtao Zeng
paper_authors._id: 668375662ca1c52c2744c837
paper_authors.name: Longxu Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:27:12.212Z
paper_authors._id: 6214e4ee1e35c843d42d1f88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg
paper_authors.fullname: Longxu Dou
paper_authors.type: user
paper_authors.user: dreamerdeo
paper_authors._id: 668375662ca1c52c2744c838
paper_authors.name: Tianyu Pang
paper_authors._id: 668375662ca1c52c2744c839
paper_authors.name: Jing Jiang
paper_authors._id: 668375662ca1c52c2744c83a
paper_authors.name: Min Lin
paper_title: RegMix: Data Mixture as Regression for Language Model Pre-training
paper_summary: The data mixture for large language model pre-training significantly impacts
  performance, yet how to determine an effective mixture remains unclear. We
  propose RegMix to automatically identify a high-performing data mixture by
  formulating it as a regression task. RegMix involves training a set of small
  models with diverse data mixtures and fitting a regression model to predict
  their performance given their respective mixtures. With the fitted regression
  model, we simulate the top-ranked mixture and use it to train a large-scale
  model with orders of magnitude more compute. To empirically validate RegMix, we
  train 512 models with 1M parameters for 1B tokens of different mixtures to fit
  the regression model and find the optimal mixture. Using this mixture we train
  a 1B parameter model for 25B tokens (i.e. 1000x larger and 25x longer) which we
  find performs best among 64 candidate 1B parameter models with other mixtures.
  Further, our method demonstrates superior performance compared to human
  selection and achieves results that match or surpass DoReMi, while utilizing
  only 10% of the compute budget. Our experiments also show that (1) Data
  mixtures significantly impact performance with single-task performance
  variations of up to 14.6%; (2) Web corpora rather than data perceived as
  high-quality like Wikipedia have the strongest positive correlation with
  downstream performance; (3) Domains interact in complex ways often
  contradicting common sense, thus automatic approaches like RegMix are needed;
  (4) Data mixture effects transcend scaling laws, and our approach captures the
  complexity by considering all domains together. Our code is available at
  https://github.com/sail-sg/regmix.
paper_ai_keywords: large language model pre-training data mixture RegMix regression task small models autonomous adaptation regression model large-scale model compute 512 models 1M parameters 1B tokens top-ranked mixture 1B parameter model 25B tokens Data mixtures single-task performance Web corpora Wikipedia downstream performance Domain interaction scaling laws