GPT2 for Thai
The goal is to create a strong language generation model for Thai 
Since initial code and data are pretty much written by @patrickvonplaten and other huggingface members, it should not be so hard to get the first sense.

Model
Randomly initialized GPT2 model

Datasets
We can use OSCAR which is available through datasets

Datasets
A causal language modeling script for Flax is available here  1. It can be used pretty much without any required code changes.
If there is time left, I’d love to try some private crawling and integrate it datasets.

Expected Outcome
Understandable Thai text generation model

Challenges
Lack of data → The OSCAR dataset might be too small (it has < 20GB of data for Thai). It would be better to collect and accumulate to bigger dataset.

Links

Colab Notebook to train CLM 4
OSCAR
flax code  1


Team Members

Sakares Saengkaew (@sakares)
Wilson Wongso (@w11wo 1)