et al., 2022) and GLaM (Du et al., 2021) tune the domain weights based on a set of downstream
tasks, but requires training potentially thousands of LMs on different domain weights and risks
overﬁtting to the particular set of downstream tasks.
Instead of optimizing domain weights based on a set of downstream tasks, our approach aims to
ﬁnd domain weights which lead to models that perform well on all domains by minimizing the
∗Work done while interning at Google. Correspondence: <Sang Michael Xie: xie@cs.stanford.edu>, <Hieu Pham:
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining