readme: ---
  language:
  - fi
  license: apache-2.0
  tags:
  - finnish
  - t5
  - t5x
  - seq2seq
  datasets:
  - Finnish-NLP/mc4_fi_cleaned
  - wikipedia
  inference: false
  
  ---
  
  # T5-small-nl16 for Finnish
  
  Pretrained T5 model on Finnish language using a span-based masked language modeling (MLM) objective. T5 was introduced in
  [this paper](https://arxiv.org/abs/1910.10683)
  and first released at [this page](https://github.com/google-research/text-to-text-transfer-transformer).
  
  **Note:** The Hugging Face inference widget is deactivated because this model needs a text-to-text fine-tuning on a specific downstream task to be useful in practice. As an example of a fine-tuned Finnish T5 model, you can check [Finnish-NLP/t5-small-nl24-casing-punctuation-correction](https://huggingface.co/Finnish-NLP/t5-small-nl24-casing-punctuation-correction) which has been fine-tuned to correct missing casing and punctuation for Finnish text.
  
  ## Model description
  
  T5 is an encoder-decoder model and treats all NLP problems in a text-to-text format.
  
  Finnish T5 is a transformers model pretrained on a very large corpus of Finnish data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and outputs from those texts.
  
  More precisely, it was pretrained with the span-based masked language modeling (MLM) objective. Spans of the input sequence are masked by so-called sentinel tokens (a.k.a unique mask tokens) and the output sequence is formed as a concatenation of the same sentinel tokens and the real masked tokens. This way, the model learns an inner representation of the Finnish language.
  
  This model used the [T5 v1.1](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#t511) improvements compared to the original T5 model during the pretraining:
  - GEGLU activation in feed-forward hidden layer, rather than ReLU - see [here](https://arxiv.org/abs/2002.05202)
  - Dropout was turned off in pretraining (quality win). Dropout should be re-enabled during fine-tuning
  - Pretrained on span-based masked language modeling (MLM) objective only without mixing in the downstream tasks
  - No parameter sharing between embedding and classifier layer
  
  This model also used the "efficient" T5 architecture findings presented in [this paper](https://arxiv.org/abs/2109.10686). In a nutshell, the paper indicates that a Deep-Narrow model architecture is favorable for downstream performance compared to other model architectures of similar parameter count. To be more precise, model depth is defined as the number of transformer blocks that are stacked sequentially.
  
  This model uses the [t5-efficient-small-nl16](https://huggingface.co/google/t5-efficient-small-nl16) architecture's layer depth which means both the encoder and the decoder have 16 transformer layers compared to the original T5 "small" model's architecture of 6 transformer layers.
  
  In total, this model has 184 million parameters.
  
  ## Intended uses & limitations
  
  This model was only pretrained in a self-supervised way excluding any supervised training. Therefore, this model has to be fine-tuned before it is usable on a downstream task, like text classification, unlike the Google's original T5 model. **Note:** You most likely need to fine-tune these T5 models without mixed precision so fine-tune them with full fp32 precision. You can also find more fine-tuning tips from [here](https://discuss.huggingface.co/t/t5-finetuning-tips), for example.
  
  ### How to use
  
  Here is how to use this model in PyTorch:
  
  ```python
  from transformers import T5Tokenizer, T5ForConditionalGeneration
  
  tokenizer = T5Tokenizer.from_pretrained("Finnish-NLP/t5-small-nl16-finnish")
  model = T5ForConditionalGeneration.from_pretrained("Finnish-NLP/t5-small-nl16-finnish")
  ```
  
  and in TensorFlow:
  
  ```python
  from transformers import T5Tokenizer, TFT5ForConditionalGeneration
  
  tokenizer = T5Tokenizer.from_pretrained("Finnish-NLP/t5-small-nl16-finnish")
  model = T5ForConditionalGeneration.from_pretrained("Finnish-NLP/t5-small-nl16-finnish", from_pt=True)
  ```
  
  ### Limitations and bias
  
  The training data used for this model contains a lot of unfiltered content from the internet, which is far from neutral. Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.
  
  ## Training data
  
  This Finnish T5 model was pretrained on the combination of six datasets:
  - [mc4_fi_cleaned](https://huggingface.co/datasets/Finnish-NLP/mc4_fi_cleaned), the dataset mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 dataset and further cleaned it with our own text data cleaning codes (check the dataset repo).
  - [wikipedia](https://huggingface.co/datasets/wikipedia) We used the Finnish subset of the wikipedia (August 2021) dataset
  - [Yle Finnish News Arc