GPT2 for Tamil
We aim to create a GPT2 model for tamil language.

Model
A randomly initialized GPT2 model

Datasets
OSCAR dataset (https://oscar-corpus.com/ 3) has about 9 GB of training data and Indic-NLP (IndicCorp | AI4Bharat IndicNLP 5) has 500 million token dataset.

Available training scripts
A causal language modeling script for Flax is available [here ]It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Tamil language.