(cid:80)T

Step 3: Train large model with new domain weights. The tuned domain weights ¯α deﬁne a new
training distribution P ¯α. We train a main model (larger than the reference/proxy models) on data
from this new distribution using a standard training procedure.
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining