Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy
Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efﬁcient data selection for deep
learning. In International Conference on Learning Representations (ICLR), 2020.

13

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu,
M. Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma,
Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson,
K. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen,
and Claire Cui. GLaM: Efﬁcient scaling of language models with mixture-of-experts. arXiv, 2021.

John Duchi, Tatsunori Hashimoto, and Hongseok Namkoong.

Distributionally robust
losses against mixture covariate shifts. https://cs.stanford.edu/~thashim/assets/
publications/condrisk.pdf, 2019.
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining