mix ratio. We use the same hyper-parameters for SC for the SC component of this objective.

• UniLM (ULM) - This is the objective proposed in Dong et al. (2019). Similar to the original UniLM, we
mix causal language modeling, Preﬁx LM (sequence-to-sequence LM) and bidirectional i.i.d denoising.
Instead of training UniLM in cloze-style or BERT-style, we opt to generate the masked tokens. This
allows this objective to be applicable to both decoder-only and encoder-decoder architectures and
remove the need for task-speciﬁc linear heads for ﬁne-tuning.

For all objectives, we explore both single-stack and encoder-decoder architectures. All architectures are
inputs-to-targets either implemented in encoder-decoder or decoder-only model structures since we consider

11
UL2- Unifying Language Learning Paradigms