Figure 1: In both decoder-only and encoder-decoder setups, UL2 strikes a signiﬁcantly improved balance in
performance between ﬁne-tuned discriminative tasks and prompt-based 1-shot open-ended text generation
than previous methods. Note: Dec and EncDec are compute matched but EncDec models have double the
parameters.
The appeal of a universal model is clear, i.e., as this not only allows concentrated eﬀort in improving and scaling
a single model, instead of diversifying resources across N models. Moreover, under resource constrained
settings where only a few models can be served (e.g., on device), it would be preferable to have a single
pretrained model that can perform well on many types of tasks.
At the core of UL2 is a the newly proposed Mixture-of-Denoisers (MoD), a pre-training objective that enables
strong performance across tasks. MoD is a mixture of several well-established denoising objectives along with

4
UL2- Unifying Language Learning Paradigms