title: Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model
    Pretraining
summary: Large language models exhibit exceptional generalization capabilities,
  primarily attributed to the utilization of diversely sourced data. However,
  conventional practices in integrating this diverse data heavily rely on
  heuristic schemes, lacking theoretical guidance. This research tackles these
  limitations by investigating strategies based on low-cost proxies for data
  mixtures, with the aim of streamlining data curation to enhance training
  efficiency. Specifically, we propose a unified scaling law, termed BiMix, which
  accurately models the bivariate scaling behaviors of both data quantity and
  mixing proportions. We conduct systematic experiments and provide empirical
  evidence for the predictive power and fundamental principles of BiMix. Notably,
  our findings reveal that entropy-driven training-free data mixtures can achieve
  comparable or even better performance than more resource-intensive methods. We
  hope that our quantitative insights can shed light on further judicious
  research and development in cost-effective language modeling.
paper_authors._id: 6653ec0ce6ddaa4f893ebc2f
paper_authors.name: Ce Ge
paper_authors._id: 6653ec0ce6ddaa4f893ebc30
paper_authors.name: Zhijian Ma
paper_authors._id: 6653ec0ce6ddaa4f893ebc31
paper_authors.name: Daoyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T09:59:44.181Z
paper_authors._id: 6530ca70c1b8f265a431fc42
paper_authors.avatarUrl: /avatars/7b1ec5b75398c8434ccd43cdce7fee53.svg
paper_authors.fullname: Daoyuan Chen
paper_authors.type: user
paper_authors.user: yxdyc
paper_authors._id: 6653ec0ce6ddaa4f893ebc32
paper_authors.name: Yaliang Li
paper_authors._id: 6653ec0ce6ddaa4f893ebc33
paper_authors.name: Bolin Ding
paper_title: Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model
    Pretraining
paper_summary: Large language models exhibit exceptional generalization capabilities,
  primarily attributed to the utilization of diversely sourced data. However,
  conventional practices in integrating this diverse data heavily rely on
  heuristic schemes, lacking theoretical guidance. This research tackles these
  limitations by investigating strategies based on low-cost proxies for data
  mixtures, with the aim of streamlining data curation to enhance training
  efficiency. Specifically, we propose a unified scaling law, termed BiMix, which
  accurately models the bivariate scaling behaviors of both data quantity and
  mixing proportions. We conduct systematic experiments and provide empirical
  evidence for the predictive power and fundamental principles of BiMix. Notably,
  our findings reveal that entropy-driven training-free data mixtures can achieve
  comparable or even better performance than more resource-intensive methods. We
  hope that our quantitative insights can shed light on further judicious
  research and development in cost-effective language modeling.
paper_ai_summary: Research proposes BiMix scalability law to model bivariate data scaling behaviors in language models, leading to efficient data curation and improved performance.
paper_ai_keywords: large language models generalization capabilities data mixtures scaling law BiMix entropy-driven training-free data mixtures language modeling