Perplexity results for ablations. Table 7 shows the perplexities for ablations on the DRO objective.
We change the DRO objective and use these to tune domain weights on 280M reference/proxy
models. These tuned domain weights are then used to train a main 280M model. Hardest refers
to optimizing the domain-level log-perplexity without baselining with a reference model. Easiest
refers to optimizing for the domains with lowest log-perplexity under the reference model. Both

18

Table 6: Summary of per-domain perplexities for 280M, 510M, 760M, and 1B models trained on
The Pile, where the reference/proxy models are the same size. DoReMi improves the worst-case
and average perplexity of the baseline domain weights in all cases. On average, DoReMi improves
perplexity on 18 out of 22 domains.

Baseline (280M)
DoReMi (280M->280M)
Proxy (280M)
Baseline (510M)
DoReMi (510M->510M)
Proxy (510M)
Baseline (760M)
DoReMi (760M->760M)
Proxy (760M)
Baseline (1B)
DoReMi (1B->1B)
Proxy (1B)
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining