readme: ---
  language:
  - nl
  - en
  datasets:
  - yhavinga/mc4_nl_cleaned
  tags:
  - t5
  - seq2seq
  
  inference: false
  license: apache-2.0
  ---
  
  # t5-v1_1-base-dutch-english-cased
  
  A [T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) sequence to sequence model
  pre-trained from scratch on [cleaned Dutch 🇳🇱🇧🇪 mC4 and cleaned English 🇬🇧 C4](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned).
  
  
  
  
  This **t5-v1.1** model has **247M** parameters.
  It was pre-trained with masked language modeling (denoise token span corruption) objective on the dataset
  `mc4_nl_cleaned` config `small_en_nl` for **10** epoch(s) and a duration of **11d18h**,
  with a sequence length of **512**, batch size **128** and **2839630** total steps (**186B** tokens).
  Pre-training evaluation loss and accuracy are **1,11** and **0,75**.
  Refer to the evaluation section below for a comparison of the pre-trained models on summarization and translation.
  * Pre-trained T5 models need to be finetuned before they can be used for downstream tasks, therefore the inference widget on the right has been turned off.
  * For a demo of the Dutch CNN summarization models, head over to the Hugging Face Spaces for
  the **[Netherformer 📰](https://huggingface.co/spaces/flax-community/netherformer)** example application!
  
  Please refer to the original T5 papers and Scale Efficiently papers for more information about the T5 architecture
  and configs, though it must be noted that this model (t5-v1_1-base-dutch-english-cased) is unrelated to these projects and not an 'official' checkpoint.
  * **[Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf)** by *Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu*.
  * **[Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers](https://arxiv.org/abs/2109.10686)** by *Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler*.
  
  
  ## Tokenizer
  
  The model uses a cased SentencePiece tokenizer configured with the `Nmt, NFKC, Replace multi-space to single-space` normalizers
  and has 32003 tokens.
  It was trained on Dutch and English  with scripts from the Huggingface Transformers [Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling).
  See [./raw/main/tokenizer.json](tokenizer.json) for details.
   
  ## Dataset(s)
  
  All models listed below are pre-trained on 
  [cleaned Dutch mC4](https://huggingface.co/datasets/yhavinga/mc4_nl_cleaned),
  which is the original mC4, except
  
    * Documents that contained words from a selection of the Dutch and English [List of Dirty Naught Obscene and Otherwise Bad Words](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words) are removed
    * Sentences with less than 3 words are removed
    * Sentences with a word of more than 1000 characters are removed
    * Documents with less than 5 sentences are removed
    * Documents with "javascript", "lorum ipsum", "terms of use", "privacy policy", "cookie policy", "uses cookies",
      "use of cookies", "use cookies", "elementen ontbreken", "deze printversie" are removed.
   
  The Dutch and English models are pre-trained on a 50/50% mix of Dutch mC4 and English C4.
  
  The translation models are fine-tuned on [CCMatrix](https://huggingface.co/datasets/yhavinga/ccmatrix).
  
  ## Dutch T5 Models
  
  Three types of [Dutch T5 models have been trained (blog)](https://huggingface.co/spaces/yhavinga/pre-training-dutch-t5-models).
  `t5-base-dutch` is the only model with an original T5 config.
  The other model types t5-v1.1 and t5-eff have `gated-relu` instead of `relu` as activation function,
  and trained with a drop-out of `0.0` unless training would diverge (`t5-v1.1-large-dutch-cased`).
  The T5-eff models are models that differ in their number of layers. The table will list
  the several dimensions of these models. Not all t5-eff models are efficient, the best example being the inefficient
  `t5-xl-4L-dutch-english-cased`. 
  
  |                   | [t5-base-dutch](https://huggingface.co/yhavinga/t5-base-dutch)   | [t5-v1.1-base-dutch-uncased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-uncased)   | [t5-v1.1-base-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-base-dutch-cased)   | [t5-v1.1-large-dutch-cased](https://huggingface.co/yhavinga/t5-v1.1-large-dutch-cased)   | [t5-v1_1-base-dutch-english-cased](https://huggingface.co/yhavinga/t5-v1_1-base-dutch-english-cased)   | [t5-v1_1-base-dutch-english-cased-1024](https://huggingface.co/yhavinga/t5-v1_1-base-dutch-english-cased-1024)   | [t5-small-24L-dutch-english](https://huggingface.co/yhavinga/t5-small-24L-dutch-english)   | [t5-xl-4L-dutch-english-cased](https://huggingface.co/yhavinga/t5-xl-4L-dutch-english-cased)   | [t5-base-36L-dutch-english-cased](https://huggingface.co/yhavinga/t5-base-36L-dutch-english-cased)   | [t