1. RobIt
The idea is to use the Italian portion of mC4 (which is about  590 GB) and OSCAR dataset(which is about 137 GB) to pre-train a RoBERTa-base model.

2. Language
The model will be trained in Italian

3. Model used
ROBERTa-base

4. Datasets
OSCAR-Italian, Italian Portion of multilingual C4 dataset

5.Training Scripts:
There are already Flax scripts to pre-train RoBERTa that we can easily use:

  
      

      github.com
  

  
    huggingface/transformers

  master/examples/flax/language-modeling

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

  

  
    
    
  

  

Google Colaboratory

6. Challenges:

It is too much data. We need a way to reduce the amount of data to finish on time so we will use random sampling.
Achieving SOTA.


7. Desired Possible Outcome:

A Italian monolingual well performing model. We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.


8. Reads

https://arxiv.org/pdf/1907.11692.pdf
We found a model trained on RoBERTa  1 but we plan on increasing the dataset with a different tokenisation.