GPT2 for Spanish
Currently, there is no GPT2 model that was trained from scratch for Spanish on the hub: Hugging Face – The AI community building the future. 3 . For this project, the goal is to create a strong language generation model for Spanish.

Model
A randomly initialized GPT2 model

Datasets
One can make use OSCAR 4 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 3.

Available training scripts
A causal language modeling script for Flax is available here 8. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Spanish language.

(Optional) Challenges
The dataset on OSCAR is quite large → around 150 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged to datasets in a couple of days. See PR here 2 and the docs here 4.

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 7

Other reads that might be interesting include:

GPT-2 no habla español: Artificial Intelligence, Anglocentrism, and the non-human side of DH | | Vanderbilt University 4
https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171 6