readme: ---
  language:
  - dv
  license: apache-2.0
  ---
  
  # BERT base for Dhivehi
  
  Pretrained model on Dhivehi language using masked language modeling (MLM).
  
  ## Tokenizer
  
  The *WordPiece* tokenizer uses several components:
  
  * **Normalization**: lowercase and then NFKD unicode normalization.
  * **Pretokenization**: splits by whitespace and punctuation.
  * **Postprocessing**: single sentences are output in format `[CLS] sentence A [SEP]` and pair sentences in format `[CLS] sentence A [SEP] sentence B [SEP]`.
  
  ## Training
  
  Training was performed over 16M+ Dhivehi sentences/paragraphs put together by [@ashraq](https://huggingface.co/ashraq). An Adam optimizer with weighted decay was used with following parameters:
  
  * Learning rate: 1e-5
  * Weight decay: 0.1
  * Warmup steps: 10% of data
  