DoReMi can reduce perplexity across all domains without a tradeoff. Figure 4 shows the per-
domain perplexity of the 8B models on The Pile. DoReMi signiﬁcantly reduces the perplexity over
the baseline across all domains, despite allocating lower weight to some domains. How can this
occur? Intuitively, the domains with the lowest and highest entropy can be downweighted without
impacting the perplexity much. The lowest entropy domains statistically require few samples to
learn. The highest entropy domains have token distributions that are close to common uniform
priors — for example, models at random initialization tend to output a uniform next token distri-
bution. Thus, we need less samples to ﬁt these domains. Positive transfer from allocating more
samples to medium entropy domains can then improve perplexity on all domains. In Appendix D,
we provide a simple example where reweighting domains can improve perplexity on all domains
and DoReMi ﬁnds such domain weights in simulations.

7
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining