Trajectory of domain weights. Figure 8 shows the exponential moving average (smoothing pa-
rameter 0.99) of domain weights during a run of DoReMi. In both cases, there are domains with
very high weight initially and decrease in weight very quickly (within 50k steps). Since we com-
pute the ﬁnal domain weights by integrating these curves over steps and normalizing, this sug-
gests that if we have a smaller compute budget, these domains could become more important —
this highlights the dependence of the mixture weights on the compute budget. At the same time,
the domain weights tend to quickly stabilize after 50k steps, suggesting that the optimal domain
weights should be similar for larger compute budgets. We may also be able to take advantage
of this stability after 50k steps to run DoReMi for a smaller number of steps and extrapolate the
domain weights to save compute.

19

(a) 280M

(b) 1B

20
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining