guage modeling. In Empirical Methods in Natural Language Processing (EMNLP), 2019.

Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi,
Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset:
Word prediction requiring a broad discourse context. In Association for Computational Linguistics
(ACL), 2016.

David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel
Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network
training. CoRR, abs/2104.10350, 2021. URL https://arxiv.org/abs/2104.10350.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Find-
ing important examples early in training. In Association for the Advancement of Artiﬁcial Intelligence
(AAAI), 2021.
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining