2.6x fewer training steps. On the GLaM dataset, DoReMi, which has no knowledge of downstream tasks, even
matches the performance of using domain weights tuned on downstream tasks.
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining