5

2 Background: Pre-trained Language Models
In this section, we discuss background surrounding pretrained language models, pretraining objectives and
other uniﬁed pretraining proposals.
UL2- Unifying Language Learning Paradigms