readme: ---
  language:
  - it
  datasets:
  - gsarti/clean_mc4_it
  tags:
   - seq2seq
   - lm-head
  license: apache-2.0
  inference: false
  thumbnail: https://gsarti.com/publication/it5/featured.png
  ---
  
  # Italian T5 Large 🇮🇹
  
  The [IT5](https://huggingface.co/models?search=it5) model family represents the first effort in pretraining large-scale sequence-to-sequence transformer models for the Italian language, following the approach adopted by the original [T5 model](https://github.com/google-research/text-to-text-transfer-transformer). 
  
  This model is released as part of the project ["IT5: Large-Scale Text-to-Text Pretraining for Italian Language Understanding and Generation"](https://arxiv.org/abs/2203.03759) (to be released), by [Gabriele Sarti](https://gsarti.com/) and [Malvina Nissim](https://malvinanissim.github.io/) with the support of [Huggingface](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) and with TPU usage sponsored by Google's [TPU Research Cloud](https://sites.research.google/trc/). All the training was conducted on a single TPU3v8-VM machine on Google Cloud. Refer to the Tensorboard tab of the repository for an overview of the training process.
  
  *The inference widget is deactivated because the model needs a task-specific seq2seq fine-tuning on a downstream task to be useful in practice. The models in the  [`it5`](https://huggingface.co/it5) organization provide some examples of this model fine-tuned on various downstream task.*
  
  ## Model variants
  
  This repository contains the checkpoints for the `base` version of the model. The model was trained for one epoch (1.05M steps) on the [Thoroughly Cleaned Italian mC4 Corpus](https://huggingface.co/datasets/gsarti/clean_mc4_it) (~41B words, ~275GB) using 🤗 Datasets and the `google/t5-v1_1-large` improved configuration. The training procedure is made available [on Github](https://github.com/gsarti/t5-flax-gcp).
  
  The following table summarizes the parameters for all available models
  
  |                       |`it5-small`            |`it5-base`            |`it5-large` (this one) |`it5-base-oscar`                  |
  |-----------------------|-----------------------|----------------------|-----------------------|----------------------------------|
  |`dataset`              |`gsarti/clean_mc4_it`  |`gsarti/clean_mc4_it` |`gsarti/clean_mc4_it`  |`oscar/unshuffled_deduplicated_it`|
  |`architecture`         |`google/t5-v1_1-small` |`google/t5-v1_1-base` |`google/t5-v1_1-large` |`t5-base`                         |
  |`learning rate`        | 5e-3                  | 5e-3                 | 5e-3                  | 1e-2                             |
  |`steps`                | 1'050'000             | 1'050'000            | 2'100'000             | 258'000                          |
  |`training time`        | 36 hours              | 101 hours            | 370 hours             | 98 hours                         |
  |`ff projection`        |`gated-gelu`           |`gated-gelu`          |`gated-gelu`           |`relu`                            |
  |`tie embeds`           |`false`                |`false`               |`false`                |`true`                            |
  |`optimizer`            | adafactor             | adafactor            | adafactor             | adafactor                        |
  |`max seq. length`      | 512                   | 512                  | 512                   | 512                              |
  |`per-device batch size`| 16                    | 16                   | 8                     | 16                               |
  |`tot. batch size`      | 128                   | 128                  | 64                    | 128                              |
  |`weigth decay`         | 1e-3                  | 1e-3                 | 1e-2                  | 1e-3                             |
  |`validation split size`| 15K examples          | 15K examples         | 15K examples          | 15K examples                     |
  
  The high training time of `it5-base-oscar` was due to [a bug](https://github.com/huggingface/transformers/pull/13012) in the training script.
  
  For a list of individual model parameters, refer to the `config.json` file in the respective repositories.
  
  ## Using the models
  
  ```python
  from transformers import AutoTokenzier, AutoModelForSeq2SeqLM
  
  tokenizer = AutoTokenizer.from_pretrained("gsarti/it5-large")
  model = AutoModelForSeq2SeqLM.from_pretrained("gsarti/it5-large")
  ```
  
  *Note: You will need to fine-tune the model on your downstream seq2seq task to use it. See an example [here](https://huggingface.co/gsarti/it5-base-nli).*
  
  Flax and Tensorflow versions of the model are also available:
  
  ```python
  from transformers import FlaxT5ForConditionalGeneration, TFT5ForConditionalGeneration
  
  model_flax = FlaxT5ForConditionalGeneration.from_pretrained("gsarti/it5-large")
  model_tf = TFT5ForConditionalGeneration.from_pretrained("gsarti/it5-large")
  ```
  
  ## Limitations
  
  Due to the nature of the web-scraped corp