readme: ---
  language: 
  - en
  datasets:
  - c4
  
  license: apache-2.0
  ---
  
  # Introduction
  
  UL2 is a unified framework for pretraining models that are universally effective across datasets and setups. UL2 uses Mixture-of-Denoisers (MoD), apre-training objective that combines diverse pre-training paradigms together. UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes.
  
  ![model image](https://raw.githubusercontent.com/google-research/google-research/master/ul2/figs/ul2.png)
  
  **Abstract**
  
  Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 and/or GPT-like models across multiple diverse setups. Finally, by scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised NLP tasks ranging from language generation (with automated and human evaluation), language understanding, text classification, question answering, commonsense reasoning, long text reasoning, structured knowledge grounding and information retrieval. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. 
  
  For more information, please take a look at the original paper.
  
  Paper: [Unifying Language Learning Paradigms](https://arxiv.org/abs/2205.05131v1)
  
  Authors: *Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald Metzler* 
  
  # Training
  
  The checkpoint was iteratively pre-trained on C4 and fine-tuned on a variety of datasets
  
  ## PreTraining
  
  The model is pretrained on the C4 corpus. For pretraining, the model is trained on a total of 1 trillion tokens on C4 (2 million steps)
  with a batch size of 1024. The sequence length is set to 512/512 for inputs and targets. 
  Dropout is set to 0 during pretraining. Pre-training took slightly more than one month for about 1 trillion
  tokens. The model has 32 encoder layers and 32 decoder layers, `dmodel` of 4096 and `df` of 16384. 
  The dimension of each head is 256 for a total of 16 heads. Our model uses a model parallelism of 8. 
  The same same sentencepiece tokenizer as T5 of vocab size 32000 is used (click [here](https://huggingface.co/docs/transformers/v4.20.0/en/model_doc/t5#transformers.T5Tokenizer) for more information about the T5 tokenizer).
  
  UL-20B can be interpreted as a model that is quite similar to T5 but trained with a different objective and slightly different scaling knobs. 
  UL-20B was trained using the [Jax](https://github.com/google/jax) and [T5X](https://github.com/google-research/t5x) infrastructure.
  
  The training objective during pretraining is a mixture of different denoising strategies that are explained in the following:
  
  ## Mixture of Denoisers
  
  To quote the paper:
  > We conjecture that a strong universal model has to be exposed to solving diverse set of problems
  > during pre-training. Given that pre-training is done using self-supervision, we argue that such diversity
  > should be injected to the objective of the model, otherwise the model might suffer from lack a certain
  > ability, like long-coherent text generation.
  > Motivated by this, as well as current class of objective functions, we define three main paradigms that
  > are used during pre-training:
  
  - **R-Denoiser**: The regular denoising is the standard span corruption introduced in [T5](https://huggingface.co/docs/transformers/v4.20.0/en/model_doc/t5)
   that uses a range of 2 to 5 tokens as the span length, which masks about 15% of
  input tokens. These spans are short and potentially useful to acquire knowledge instead of
  learning to generate fluent text.
  
  - **S-Denoiser**: A specific case of denoising where we observe a strict sequential order when
  framing the inputs-to-targets task, i.e., prefix language modeling. To do so, we simply
  partition the input sequence into two sub-sequences of tokens as context and target such that
  the targe