readme: ---
  language:
  - lu
  tags:
  - text
  - MLM
  license: mit
  ---
  
  ## BERT Medium for Luxembourgish
  
  Created from a dataset with 1M Luxembourgish sentences from Wikipedia. Corpus has approx. 16M words.
  
  The MLM objective was trained. The BERT model has parameters `L=8` and `H=512`. Vocabulary has 70K word pieces.
  
  Final loss scores, after 3 epochs:
  
  - Final train loss: 4.230
  - Final train perplexity: 68.726
  - Final validation loss: 4.074
  - Final validation perplexity: 58.765
  