2.2 Pre-training Objectives for Large Language Models
While recent research demonstrates the potential of large supervised multi-task pre-training (Aribandi et al.,
2021; Sanh et al., 2021; Wang et al., 2022a), most pre-training objectives rely on the vast availability of
unsupervised data and use self-training techniques. As mentioned above, diﬀerent architectures typically
leverage diﬀerent objectives. Decoder-only models are typically trained with causal language model objectives
to mimic auto-regressive generation (Radford et al., 2019). Raﬀel et al. (2019) explored many objectives
for encoder-decoder models and found span corruption to be eﬀective. (Wang et al., 2022a) conducts a
systematic study of diﬀerent architectures combined with three diﬀerent pretraining objectives (causal
LM, preﬁxLM and span corruption) and analyzed their impact on zero-shot generalization. Related to
UL2- Unifying Language Learning Paradigms