Inspecting the DoReMi domain weights. Tables 1 and 2 present the DoReMi domain weights for
The Pile and the GLaM dataset. When running DoReMi on a 280M proxy model (DoReMi (280M)),
most weight is put on the diverse Pile-CC web text domain. Note that Wikipedia is downweighted
in comparison to the baseline, but DoReMi still improves the downstream accuracy on tasks de-
rived from Wikipedia (e.g., TriviaQA, Appendix Table 5). On the GLaM dataset, the DoReMi
weights have the same general pattern as the downstream-tuned domain weights. DoReMi is able
to recover a similar set of domain weights by starting from uniform initial domain weights, without
any use of downstream data.

4 Ablations and Analysis Across Scales
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining