Params
167M
167M
167M
167M
167M
335M
335M
335M
335M
335M

Supervised Finetuning

In-context One-shot

SG
62.24
62.44
67.67
63.36
65.50
69.30
72.00
72.50
71.10
73.10

XS
28.18
28.21
29.14
29.02
28.90
31.95
31.05
31.69
31.00
31.86

SGD
55.44
55.55
55.48
55.71
55.80
55.70
55.80
55.70
55.83
56.10

TOT
59.40
59.52
60.47
60.00
60.39
60.91
61.25
60.94
61.03
61.50

SG
39.22
42.54
38.53
40.78
42.30
38.18
38.51
39.74
39.86
41.30

XS
1.16
1.08
1.16
3.03
8.01
6.50
7.49
5.13
6.70
11.51

SGD TOT
0.20
1.40
6.40
3.70
1.60
2.20
1.27
0.10
6.30
5.80
3.90
7.11
2.10
1.43
8.70
7.30
6.50
4.10
6.50
6.63

LM
-2.35
-2.54
-3.62
-2.38
-2.34
-2.42
-7.23
-2.40
-2.65
-2.55

attention. To not further conﬂate architectural modiﬁcations with pretraining contributions, the backbone of
the model remains similar to a T5-like model. This is also in light of results such as (Narang et al., 2021).
UL2- Unifying Language Learning Paradigms