Scaling Laws for Optimal Data Mixtures
Large foundation models are typically trained on data from multiple domains,
with the data mixture--the proportion of each domain used--playing a critical
role in model performance. The standard approach to selecting this mixture
relies on trial and error, which becomes impractical for large-scale
pretraining. We propose a systematic method to determine the optimal data
mixture for any target domain using scaling laws. Our approach accurately
predicts the loss of a model of size N trained with D tokens and a specific
domain weight vector h. We validate the universality of these scaling laws by
demonstrating their predictive power in three distinct and large-scale
settings: large language model (LLM), native multimodal model (NMM), and large
vision models (LVM) pretraining. We further show that these scaling laws can
extrapolate to new data mixtures and across scales: their parameters can be
accurately estimated using a few small-scale training runs, and used to
estimate the performance at larger scales and unseen domain weights. The
scaling laws allow to derive the optimal domain weights for any target domain
under a given training budget (N,D), providing a principled alternative to
costly trial-and-error methods.
68774155257d4f04353707d3
Mustafa Shukor
claimed_verified
2025-07-21T08:30:28.723Z
62bdeedd01dc22b4d22a371e
/avatars/3cc0643feb53bf2e895ec12c275d5483.svg
Mustafa Shukor
user
mshukor
68774155257d4f04353707d4
Louis Bethune
68774155257d4f04353707d5
Dan Busbridge
68774155257d4f04353707d6
David Grangier
68774155257d4f04353707d7
Enrico Fini
68774155257d4f04353707d8
Alaaeldin El-Nouby
68774155257d4f04353707d9
Pierre Ablin
Scaling Laws for Optimal Data Mixtures
Large foundation models are typically trained on data from multiple domains,
with the data mixture--the proportion of each domain used--playing a critical
role in model performance. The standard approach to selecting this mixture
relies on trial and error, which becomes impractical for large-scale
pretraining. We propose a systematic method to determine the optimal data
mixture for any target domain using scaling laws. Our approach accurately
predicts the loss of a model of size N trained with D tokens and a specific
domain weight vector h. We validate the universality of these scaling laws by
demonstrating their predictive power in three distinct and large-scale
settings: large language model (LLM), native multimodal model (NMM), and large
vision models (LVM) pretraining. We further show that these scaling laws can
extrapolate to new data mixtures and across scales: their parameters can be
accurately estimated using a few small-scale training runs, and used to
estimate the performance at larger scales and unseen domain weights. The
scaling laws allow to derive the optimal domain weights for any target domain
under a given training budget (N,D), providing a principled alternative to
costly trial-and-error methods.
Scaling laws predict optimal data mixtures for large foundation models across different domains, improving performance and reducing trial-and-error.
scaling laws large language model native multimodal model large vision models domain weights training budget