Datasets for training language models (LMs) are typically sampled from a mixture of many do-
mains (Brown et al., 2020, Chowdhery et al., 2022, Du et al., 2021, Gao et al., 2020). For example,
The Pile (Gao et al., 2020), a large publicly available dataset, is composed of 24% web data, 9%
Wikipedia, 4% GitHub, etc.1 The composition of the pretraining data greatly affects the effective-
ness of an LM (Du et al., 2021, Hoffmann et al., 2022, Xie et al., 2023). However, it is unclear how
much of each domain to include to produce a model that performs well for a wide variety of down-
stream tasks.
Existing works determine domain weights (the sampling probabilities for each domain) by using
intuition or a set of downstream tasks. For example, The Pile uses heuristically-chosen domain
weights, which could be suboptimal. On the other hand, existing LMs such as PaLM (Chowdhery
et al., 2022) and GLaM (Du et al., 2021) tune the domain weights based on a set of downstream
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining