readme: ---
  language: dv
  ---
  
  # byt5-dv
  
  Pretrained from scratch on Dhivei (language of the Maldives)
  with ByT5, Google's new byte-level tokenizer strategy.
  
  Corpus: dv.wikipedia.org as of March 2020 (TFDS)
  
  Notebook - Pretraining on Wikipedia: https://colab.research.google.com/drive/19Afq7CI6cOi1DaTpnQhBbEbnBzLSFHbH
  
  ## Demo
  
  Notebook - Finetuning on Maldivian news classification task: https://colab.research.google.com/drive/11u5SafR4bKICmArgDl6KQ9vqfYtDpyWp
  
  Current performance:
  
  - mBERT: 52%
  - **byt5-dv**: 81%
  - dv-wave (ELECTRA): 89%
  - dv-muril: 90.7%
  - dv-labse: 91.3-91.5%
  
  Source of dataset: https://github.com/Sofwath/DhivehiDatasets
  
  ## Work in progress - todos
  
  The Wikipedia corpus is too small for this language. In the future I would add
  OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those
  as one TFDS dataset.
  
  This is based on ByT5-small ... we should try a larger model
  
  This needs more time for pretraining