Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer. arXiv preprint arXiv:1910.10683, 2019.

15

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable ques-

tions for SQuAD. In Association for Computational Linguistics (ACL), 2018.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust
neural networks for group shifts: On the importance of regularization for worst-case generaliza-
tion. In International Conference on Learning Representations (ICLR), 2020.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set

approach. In International Conference on Learning Representations (ICLR), 2018.

Noam Shazeer and Mitchell Stern. 2018.
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining