General data selection methods. Moore-Lewis selection (Axelrod, 2017, Feng et al., 2022, Moore
and Lewis, 2010) selects examples with high cross-entropy difference (similar to excess log-
perplexity) between language models trained on target and raw data. Coleman et al. (2020) select
examples based on the uncertainty of a small proxy model for active learning. Mindermann et al.
(2022) select examples in an online fashion by taking the top k examples in a minibatch according
to excess loss. Many other works on data selection are in vision (Kaushal et al., 2019, Killamsetty
et al., 2021a,b,c, Mirzasoleiman et al., 2020, Paul et al., 2021, Sener and Savarese, 2018, Sorscher et al.,
2022, Wang et al., 2020, Wei et al., 2015). Overall, these methods do not address data selection for
pretraining, where the downstream data distribution may be very different from the pretraining
distribution. DoReMi aims to address the pretraining/downstream distribution shift with DRO.
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining