metadata
license: bsd
language:
- sk
base_model:
- FacebookAI/roberta-base
Slovak Roberta Base
A monolingual Slovak language model.
Model was trained on a collection of Slovak web pages from various sources.
Training parameters
We used 4 x A100 40GB GPU for 14 hours.
- Effective batch size: 192
- Sequence length 512
- Training Steps 120 000.
- warmup_steps 1000
- optimizer adamw
- Per device batch size 48
- mixed_precision bf16
- weight decay 0.01
- gradient clipping 1.0
- learning_rate 1e-5
- scheduler cosine