FlaxBigBird for Dutch language
Currently, there are only a very limited amount of long-range sequence models for other languages than English on the hub. The goal of this project is to create a strong Dutch FlaxBigBird model.

Model
A randomly initialized  FlaxBigBird model.

Datasets
CC-100 2
mC4 2

Available training scripts
Scripts to pre-train Flax models are here: transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub 5
Scripts for fine-tuning and evaluation for FlaxBigBird just got released here: transformers/examples/research_projects/jax-projects/big_bird at master · huggingface/transformers · GitHub 2 Thanks to @vasudevgupta

(Optional) Desired project outcome
The desired project output is a strong Dutch FlaxBigBird model in Dutch. For downstream tasks that require long sequences (e.g. long text classification).

(Optional) Challenges