Scandinavian RoBERTa
Rather than training separate models for Swedish, Norwegian, Danish and Icelandic [1][2][3], we could probably produce a better model by pretraining a Scandinavian model and then finetuning to one of the four, considering how similar the four languages are (in its written form that is, cough cough Danish).
We can train a RoBERTa-large model on the combined mC4 dataset 5, containing 386 GB uncompressed text (179 GB Swedish, 107 GB Danish and 100 GB Norwegian). Furthermore, there are gigaword datasets in both Swedish, Danish and Icelandic that we could use. As suggested in [1], we could start training the model with a sequence length of 128, then 256 and lastly 512.

2. Language
The model will be trained in Swedish, Danish, Norwegian and Icelandic.

3. Model
RoBERTa-large

4. Datasets

Swedish:

mC4 (179 GB)
Gigaword [4] (~9 GB compressed)


Danish

mC4 (107 GB)
Gigaword [5] (~2 GB compressed)


Norwegian

mC4 (100 GB)


Icelandic

mC4 (9 GB)
Gigaword [6] (~14 GB compressed)




5. Training scripts
There are already Flax scripts to pre-train RoBERTa that we can easily use:
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling 14

6. Challenges

Will the data be enough to be able to train a good model?
Will all the languages be well-represented?


7. Desired project outcome
A Scandinavian language model which performs well on the usual benchmarks, on each of the four languages.

8. Reads
[1] https://discuss.huggingface.co/t/pretrain-roberta-large-from-scratch-in-swedish
[2] https://discuss.huggingface.co/t/pretrain-gpt2-from-scratch-in-swedish
[3] https://discuss.huggingface.co/t/pretrain-roberta-from-scratch-in-norwegian
[4] The Swedish Culturomics Gigaword Corpus | Språkbanken Text 3
[5] https://gigaword.dk/ 2
[6] Icelandic Gigaword Corpus