readme: ---
  language: dv
  ---
  
  # byt5-base-dv
  
  Pretrained from scratch on Dhivei (language of the Maldives)
  with ByT5, Google's new byte-level tokenizer strategy.
  
  **Use byt5-dv for now; this is less accurate**
  
  Corpus: Sofwath's Dhivehi corpus https://github.com/Sofwath/DhivehiDatasets
  
  Pretraining Notebook: 
  https://colab.research.google.com/drive/1ERIZ1PyHn-yN_jo7dTQeODn22vrt-d1d?usp=sharing
  
  ## Fine-tuning Demo
  
  On Dhivehi news classification task
  
  https://colab.research.google.com/drive/11u5SafR4bKICmArgDl6KQ9vqfYtDpyWp?usp=sharing
  
  ## Issues
  
  There was an issue with the vocabulary size, final layer, and/or accuracy on fine-tuning.
  