in the span corruption objective, when the corrupted span, i.e., target, is equal to the entire sequence, the
problem becomes eﬀectively1 a language modeling problem. With this in mind, using span corruption, by
setting the span length to be large, we can eﬀectively mimic the LM objective in local regions.
We deﬁne a notation that covers all of the diﬀerent denoising tasks that we use in this paper. The inputs and
targets of the denoising tasks are generated by a SpanCorrupt function that is parameterized by three values
(µ, r, n), where µ is the mean span length, r is the corruption rate, and n which is number of corrupted spans.
Note that n may be a function of the input length, L, and the span length µ, e.g. L/µ, but in some cases, we
use a ﬁxed value of n. Given an input text, SpanCorrupt introduces corruptions to the spans of lengths that
are drawn from a (normal or uniform) distribution with mean of µ. After corruption, the input text is then
UL2- Unifying Language Learning Paradigms