RoBERTa for Portuguese
Currently, there is only a very limited amount of RoBERTa or RoBERTa Large model that was trained from scratch for Portuguese on the hub: Hugging Face – The AI community building the future. 4. For this project, the goal is to create a RoBERTa model for just the Portuguese language.

Model
A randomly initialized RoBERTa model

Available training scripts
A masked language modeling script for Flax is available here 4. It can be used pretty much without any required code changes.

Dataset
We can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.

Challenges

Data pre-processing
Sample preparation for LM


(Optional) Desired project outcome
The desired project output is a strong RoBERTa model in Portuguese.