pytorch t5 seq2seq id transformers text2text-generation
.gitattributes README.md config.json pytorch_model.bin spiece.model
 ---
inference: false
language: id
tags:
- t5
- pytorch
---

# IndoT5 Small Model

IndoT5 is a T5 model pretrained on various corpus. It's available in three sizes (small, base and large). IndoT5 only pretrained on denoising task, fine-tuning is still needed.

## Pretraining details

IndoT5 is pretrained on :
- OSCAR Corpus
- id_newspaper2018
- Leipzig Newscrawl
- OPUS (OpenSubtitle, Tanzil, etc)

Using script provided by unicamp-dl on [PTT5 Repository](https://github.com/unicamp-dl/PTT5). Trained 500k steps.

## All Pre-trained Models

| Model                          | Arch. | Training data                     |
|--------------------------------|--------------------------------|-------|
| `Wikidepia/IndoT5-small` | Small  | OSCAR, News, OPUS            |
| `Wikidepia/IndoT5-base` | Base  | OSCAR, News, OPUS           |
| `Wikidepia/IndoT5-large` | Large  | OSCAR, News, OPUS           |
| `Wikidepia/IndoT5-small-cc100` | Small  | CC100            |

## Acknowledgement

Big thanks to TFRC (TensorFlow Research Cloud) for sponsoring TPU.