readme: 
  ---
  language:
  - nl 
  - en
  - multilingual
  license: apache-2.0
  tags:
  - dutch
  - english
  - t5
  - t5x
  - ul2
  - seq2seq
  - translation
  datasets:
  - yhavinga/mc4_nl_cleaned
  - yhavinga/nedd_wiki_news
  pipeline_tag: translation
  widget:
    - text: >-
        Redistricting and West Virginia’s shrinking population forced the state’s
        Republican Legislature to pit Mr. McKinley, a six-term Republican with a
        pragmatic bent, against Mr. Mooney, who has served four terms marked more
        by conservative rhetoric than legislative achievements.
    - text: >-
        It is a painful and tragic spectacle that rises before me: I have drawn
        back the curtain from the rottenness of man. This word, in my mouth, is at
        least free from one suspicion: that it involves a moral accusation against
        humanity.
    - text: >-
        Young Wehling was hunched in his chair, his head in his hand. He was so
        rumpled, so still and colorless as to be virtually invisible. His
        camouflage was perfect, since the waiting room had a disorderly and
        demoralized air, too. Chairs and ashtrays had been moved away from the
        walls. The floor was paved with spattered dropcloths.
  ---
  
  # ul2-base-nl36-en-nl for English to Dutch translation
  
  Fine-tuned T5 model on English to Dutch translation that was pretrained on Dutch using a UL2 (Mixture-of-Denoisers) objective.
  The T5 model was introduced in
  [this paper](https://arxiv.org/abs/1910.10683)
  and first released at [this page](https://github.com/google-research/text-to-text-transfer-transformer).
  The UL2 objective was introduced in
  [this paper](https://arxiv.org/abs/2205.05131)
  and first released at [this page](https://github.com/google-research/google-research/tree/master/ul2).
  
  
  
  ## Model description
  
  T5 is an encoder-decoder model and treats all NLP problems in a text-to-text format.
  
  `ul2-base-nl36-en-nl` T5 is a transformers model fine-tuned on parallel sentence and paragraph pairs
  sampled from books.
  
  This model used the [T5 v1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511) improvements compared to the original T5 model during the pretraining:
  - GEGLU activation in the feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202)
  - Dropout was turned off during pre-training. Dropout should be re-enabled during fine-tuning
  - Pre-trained on self-supervised objective only without mixing in the downstream tasks
  - No parameter sharing between embedding and classifier layer
  
  The "efficient" T5 architecture findings presented in [this paper](https://arxiv.org/abs/2109.10686) were also applied,
  which suggests that a Deep-Narrow model architecture is favorable for downstream performance compared to other model
  architectures of similar parameter count. Specifically, the model depth is defined as the number of transformer blocks
  that are stacked sequentially.
  This model uses the [t5-efficient-base-nl36](https://huggingface.co/google/t5-efficient-base-nl36) architecture's
  layer depth, which means both the encoder and the decoder have 36 transformer layers compared to the original T5 "base"
  model's architecture of 12 transformer layers.
  
  ### UL2 pretraining objective
  
  This model was pretrained with the UL2's Mixture-of-Denoisers (MoD) objective, that combines diverse pre-training
  paradigms together. UL2 frames different objective functions for training language models as denoising tasks, where
  the model has to recover missing sub-sequences of a given input. During pre-training it uses a novel mixture-of-denoisers
  that samples from a varied set of such objectives, each with different configurations. UL2 is trained using a mixture of
  three denoising tasks:
  
  1. R-denoising (or regular span corruption), which emulates the standard T5 span corruption objective;
  2. X-denoising (or extreme span corruption); and
  3. S-denoising (or sequential PrefixLM).
  
  During pre-training, we sample from the available denoising tasks based on user-specified ratios.
  UL2 introduces a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training
  denoising task. During the pre-training, a paradigm token is inserted to the input
  (`[NLU]` for R-denoising, `[NLG]` for X-denoising, or `[S2S]` for S-denoising) indicating the denoising task at hand.
  Then, during fine-tuning the same input token should be inserted to get the best performance for different downstream
  fine-tuning tasks.
  
  ## Intended uses & limitations
  
  This model was fine-tuned on parallel sentence and paragraph pairs and can be used
  for machine translation.
  
  ### How to use
  
  Here is how to use this model in PyTorch:
  
  ```python
  model_name = "yhavinga/ul2-base-nl36-en-nl"
  from transformers import AutoTokenizer
  from transformers import AutoModelForSeq2SeqLM
  from transformers import pipeline
  import torch
  device_num = 0 if torch.cuda.is_available() else -1
  device = "cpu" if device_num < 0 else f"cuda:{device_num}"
  
  tokenizer = AutoTo