README.md

---
license: bsd
language:
- sk
base_model:
- FacebookAI/roberta-base
---

# Slovak Roberta Base

A monolingual Slovak language model.

Model was trained on a collection of Slovak web pages from various sources.

## Training parameters

We used 4 x A100 40GB GPU for 14 hours.

- Effective batch size: 192
- Sequence length 512
- Training Steps 120 000.
- warmup_steps 1000 
- optimizer adamw 
- Per device batch size 48
- mixed_precision bf16
- weight decay 0.01
- gradient clipping 1.0 
- learning_rate 1e-5
- scheduler cosine