tensorboard roberta masked-lm transformers fill-mask
.gitattributes README.md config.json run_mlm.sh run_mlm_flax.py tokenizer.json train_tokenizer.py runs/config.json runs/logs/events.out.tfevents.1619649831.t1v-n-71556209-w-0.63096.3.v2 runs/logs/events.out.tfevents.1619650245.t1v-n-71556209-w-0.65176.3.v2 runs/logs/events.out.tfevents.1619658715.t1v-n-71556209-w-0.66938.3.v2 runs/logs/events.out.tfevents.1619789329.t1v-n-71556209-w-0.90330.3.v2 runs/logs/events.out.tfevents.1619963181.t1v-n-71556209-w-0.47643.3.v2 runs/logs/events.out.tfevents.1619964922.t1v-n-71556209-w-0.54388.3.v2 runs/logs/events.out.tfevents.1619992500.t1v-n-71556209-w-0.79622.3.v2 runs/logs/events.out.tfevents.1619992737.t1v-n-71556209-w-0.81383.3.v2 runs/logs/events.out.tfevents.1619993670.t1v-n-71556209-w-0.83638.3.v2 runs/logs/events.out.tfevents.1619994044.t1v-n-71556209-w-0.85400.3.v2 runs/logs/events.out.tfevents.1619994255.t1v-n-71556209-w-0.87341.3.v2 runs/logs/events.out.tfevents.1619994499.t1v-n-71556209-w-0.89395.3.v2 runs/logs/events.out.tfevents.1619994922.t1v-n-71556209-w-0.91380.3.v2
## Roberta-Large

This repo trains [roberta-large](https://huggingface.co/roberta-large) from scratch on the [Norwegian training subset of Oscar](https://oscar-corpus.com/) containing roughly 4.7 GB of data.

A ByteLevelBPETokenizer as shown in [this]( ) blog post was trained on the whole [Norwegian training subset of Oscar](https://oscar-corpus.com/).

Training is done on a TPUv3-8 in Flax. The training script as well as the script to create a tokenizer are attached below.

### Run 1

```
--weight_decay="0.01"
--max_seq_length="128"
--train_batch_size="1048"
--eval_batch_size="1048"
--learning_rate="1e-3"
--warmup_steps="2000"
--pad_to_max_length
--num_train_epochs="12"
--adam_beta1="0.9"
--adam_beta2="0.98"
```

Trained for 12 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 1 day and 8 hours. Final loss was 3.695.

**Acc**:

![Acc](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_acc_1.svg)

**Loss**:

![Loss](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_loss_1.svg)

### Run 2

```
--weight_decay="0.01"
--max_seq_length="128"
--train_batch_size="1048"
--eval_batch_size="1048"
--learning_rate="5e-3"
--warmup_steps="2000"
--pad_to_max_length
--num_train_epochs="7"
--adam_beta1="0.9"
--adam_beta2="0.98"
```

Trained for 7 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 18 hours. Final loss was 2.216 and accuracy 0.58.


**Acc**:

![Acc](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_acc_2.svg)

**Loss**:

![Loss](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_loss_2.svg)
