2020)) suggested further improvements to the pre-training process.
At the same time, two-stack encoder-decoder architectures such as T5 (Raﬀel et al., 2019) gained popularity
due to their improved performance on classiﬁcation and sequence-to-sequence (“seq2seq”) tasks. However,
so far, these models have shown limited performance on open-text generation and prompt-based inference
(i.e., in-context learning), which motivates the use of decoder-only models that are trained with diﬀerent
objectives (e.g., GPT-3 (Brown et al., 2020), GLaM (Du et al., 2021), LaMDa (Thoppilan et al., 2022) and
PaLM (Chowdhery et al., 2022)). In this work, we aim to bridge the performance gap between the two by
means of a general training paradigm that suits both architectures.
UL2- Unifying Language Learning Paradigms