 Model variants  This repository contains the checkpoints for the `base` version of the model. The model was trained for one epoch (1.05M steps) on the [Thoroughly Cleaned Italian mC4 Corpus](https://huggingface.co/datasets/gsarti/clean_mc4_it) (~41B words, ~275GB) using 🤗 Datasets and the `google/t5-v1_1-large` improved configuration. The training procedure is made available [on Github](https://github.com/gsarti/t5-flax-gcp).  The following table summarizes the parameters for all available models  |                       |`it5-small`            |`it5-base`            |`it5-large` (this one) |`it5-base-oscar`                  | |-----------------------|-----------------------|----------------------|-----------------------|----------------------------------| |`dataset`              |`gsarti/clean_mc4_it`  |`gsarti/clean_mc4_it` |`gsarti/clean_mc4_it`  |`oscar/unshuffled_deduplicated_it`| |`architecture`         |`google/t5-v1_1-small` |`google/t5-v1_1-base` |`google/t5-v1_1-large` |`t5-base`                         | |`learning rate`        | 5e-3                  | 5e-3                 | 5e-3                  | 1e-2                             | |`steps`                | 1'050'000             | 1'050'000            | 2'100'000             | 258'000                          | |`training time`        | 36 hours              | 101 hours            | 370 hours             | 98 hours                         | |`ff projection`        |`gated-gelu`           |`gated-gelu`          |`gated-gelu`           |`relu`                            | |`tie embeds`           |`false`                |`false`               |`false`                |`true`                            | |`optimizer`            | adafactor             | adafactor            | adafactor             | adafactor                        | |`max seq. length`      | 512                   | 512                  | 512                   | 512                              | |`per-device batch size`| 16                    | 16                   | 8                     | 16                               | |`tot. batch size`      | 128                   | 128                  | 64                    | 128                              | |`weigth decay`         | 1e-3                  | 1e-3                 | 1e-2                  | 1e-3                             | |`validation split size`| 15K examples          | 15K examples         | 15K examples          | 15K examples                     |  The high training time of `it5-base-oscar` was due to [a bug](https://github.com/huggingface/transformers/pull/13012) in the training script.  For a list of individual model parameters, refer to the `config.json` file in the respective repositories.  