readme: 
  ---
  language:
  - nl 
  - en
  - multilingual
  license: apache-2.0
  tags:
  - dutch
  - english
  - t5
  - t5x
  - ul2
  - seq2seq
  datasets:
  - yhavinga/mc4_nl_cleaned
  - yhavinga/nedd_wiki_news
  inference: false
  ---
  
  # ul2-small-dutch-english for Dutch and English
  
  Pretrained T5 model on Dutch and English using a UL2 (Mixture-of-Denoisers) objective.
  The T5 model was introduced in
  [this paper](https://arxiv.org/abs/1910.10683)
  and first released at [this page](https://github.com/google-research/text-to-text-transfer-transformer).
  The UL2 objective was introduced in
  [this paper](https://arxiv.org/abs/2205.05131)
  and first released at [this page](https://github.com/google-research/google-research/tree/master/ul2).
  
  **Note:** The Hugging Face inference widget is deactivated because this model needs a text-to-text fine-tuning on
  a specific downstream task to be useful in practice.
  
  ## Model description
  
  T5 is an encoder-decoder model and treats all NLP problems in a text-to-text format.
  `ul2-small-dutch-english` T5 is a transformers model pretrained on a very large corpus of
  Dutch and English data in a self-supervised fashion.
  This means it was pretrained on the raw texts only, with no humans labelling them in any way
  (which is why it can use lots of publicly available data) with an automatic process to generate
  inputs and outputs from those texts.
  
  
  This model used the [T5 v1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511) improvements compared to the original T5 model during the pretraining:
  - GEGLU activation in the feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202)
  - Dropout was turned off during pre-training. Dropout should be re-enabled during fine-tuning
  - Pre-trained on self-supervised objective only without mixing in the downstream tasks
  - No parameter sharing between embedding and classifier layer
  
  
  
  ### UL2 pretraining objective
  
  This model was pretrained with the UL2's Mixture-of-Denoisers (MoD) objective, that combines diverse pre-training
  paradigms together. UL2 frames different objective functions for training language models as denoising tasks, where
  the model has to recover missing sub-sequences of a given input. During pre-training it uses a novel mixture-of-denoisers
  that samples from a varied set of such objectives, each with different configurations. UL2 is trained using a mixture of
  three denoising tasks:
  
  1. R-denoising (or regular span corruption), which emulates the standard T5 span corruption objective;
  2. X-denoising (or extreme span corruption); and
  3. S-denoising (or sequential PrefixLM).
  
  During pre-training, we sample from the available denoising tasks based on user-specified ratios.
  UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training
  denoising task. During the pre-training, a paradigm token is inserted to the input
  (`[NLU]` for R-denoising, `[NLG]` for X-denoising, or `[S2S]` for S-denoising) indicating the denoising task at hand.
  Then, during fine-tuning the same input token should be inserted to get the best performance for different downstream
  fine-tuning tasks.
  
  ## Intended uses & limitations
  
  This model was only pretrained in a self-supervised way excluding any supervised training.
  Therefore, this model has to be fine-tuned before it is usable on a downstream task,
  like text classification, unlike the Google's original T5 model.
  
  **Note:** You most likely need to fine-tune these T5/UL2 models without mixed precision
  so fine-tune them with full fp32 precision. Fine-tuning with Flax in bf16 - `model.to_bf16()` - is possible
  if you set the mask correctly to exclude layernorm and embedding layers. Also note that the T5x pre-training
  and fine-tuning configs set `z_loss` to 1e-4, which is used to keep the loss scale from underflowing.
  You can also find more fine-tuning tips from [here](https://discuss.huggingface.co/t/t5-finetuning-tips), for example.
  
  **Note**: For fine-tuning, most likely you can get better results if you insert a prefix token
  of `[NLU]`, `[NLG]`, or `[S2S]` to your input texts.
  For general language understanding fine-tuning tasks, you could use the `[NLU]` token.
  For GPT-style causal language generation, you could use the `[S2S]` token.
  The token `[NLG]` of the X-denoising pretrain task is somewhat mix between the language understanding and causal language
  generation so the token `[NLG]` could maybe be used for language generation fine-tuning too.
  
  ### How to use
  
  Here is how to use this model in PyTorch:
  
  ```python
  from transformers import T5Tokenizer, T5ForConditionalGeneration
  
  tokenizer = T5Tokenizer.from_pretrained("yhavinga/ul2-small-dutch-english", use_fast=False)
  model = T5ForConditionalGeneration.from_pretrained("yhavinga/ul2-small-dutch-english")
  ```
  
  and in Flax:
  
  ```python
  from transformers import T5Tokenizer, FlaxT5ForConditionalGeneration
  
  tokenizer = T5Tokenizer.from_pretrained("yhavinga/ul2-small-dutch