1 Introduction
There is a wide spectrum of pre-trained model options for NLP researchers and practitioners these days
(Devlin et al., 2018; Brown et al., 2020; Raﬀel et al., 2019; Radford et al., 2019; Liu et al., 2019; Yang et al.,
2019; Thoppilan et al., 2022; Fedus et al., 2021; Du et al., 2021; Chowdhery et al., 2022). When faced with the
question of what model should one use, the answer is often it depends, followed by on what task?
Answering this can be overwhelming, comprising of a number of ﬁne-grained follow-up questions like,
‘encoder-only or encoder-decoder?’, ‘span corruption or language model?’. Pressing further, the answer always
seems to depend on the target downstream task. This paper questions and rethinks this thought process,
speciﬁcally answering the questions of why should the choice of the pre-trained LM depend on the downstream task?
and how can we pre-train models that work universally well across many tasks?.
UL2- Unifying Language Learning Paradigms