3
2
0
2

 

b
e
F
8
2

 

 
 
]
L
C
.
s
c
[
 
 

3
v
1
3
1
5
0

.

5
0
2
2
:
v
i
X
r
a

∗Yi and Mostafa are co-leads of this project and are denoted with ∗. (cid:93) denotes technical research contributors. (cid:91) denotes data &
infrastructure contributions. (cid:52) denotes advising contributions. Don, denoted with (cid:3) is the last author. Full contributions of all authors
at the end of paper. Correspondence to yitay@google.com or dehghani@google.com.

1

Contents

1 Introduction

2 Background: Pre-trained Language Models

2.1 Pre-trained Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Pre-training Objectives for Large Language Models
. . . . . . . . . . . . . . . . . . . . . . . .
2.3 Uniﬁed Pre-training Proposals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 Unifying Language Learning Paradigms (UL2)

3.1 Pre-training .

.

.

.

.

.

.

.

.

.
UL2- Unifying Language Learning Paradigms