readme: ---
  language: ms
  tags:
  - melayu-bert
  license: mit
  datasets:
  - oscar
  widget:
  - text: "Saya [MASK] makan nasi hari ini."
  ---
  
  ## Melayu BERT
  
  Melayu BERT is a masked language model based on [BERT](https://arxiv.org/abs/1810.04805). It was trained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset, specifically the `unshuffled_original_ms` subset. The model used was [English BERT model](https://huggingface.co/bert-base-uncased) and fine-tuned on the Malaysian dataset. The model achieved a perplexity of 9.46 on a 20% validation dataset. Many of the techniques used are based on a Hugging Face tutorial [notebook](https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) written by [Sylvain Gugger](https://github.com/sgugger), and [fine-tuning tutorial notebook](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb) written by [Pierre Guillou](https://huggingface.co/pierreguillou). The model is available both for PyTorch and TensorFlow use.
  
  ## Model
  
  The model was trained on 3 epochs with a learning rate of 2e-3 and achieved a training loss per steps as shown below.
  
  |  Step  |Training loss|
  |--------|-------------|
  |500	 |  5.051300   |
  |1000	 |  3.701700   |
  |1500	 |  3.288600   |
  |2000	 |  3.024000   |
  |2500	 |  2.833500   |
  |3000	 |  2.741600   |
  |3500	 |  2.637900   |
  |4000	 |  2.547900   |
  |4500	 |  2.451500   |
  |5000	 |  2.409600   |
  |5500	 |  2.388300   |
  |6000	 |  2.351600   |
  
  ## How to Use
  ### As Masked Language Model
  ```python
  from transformers import pipeline
  pretrained_name = "StevenLimcorn/MelayuBERT"
  fill_mask = pipeline(
      "fill-mask",
      model=pretrained_name,
      tokenizer=pretrained_name
  )
  fill_mask("Saya [MASK] makan nasi hari ini.")
  ```
  
  ### Import Tokenizer and Model
  ```python
  from transformers import AutoTokenizer, AutoModelForMaskedLM
    
  tokenizer = AutoTokenizer.from_pretrained("StevenLimcorn/MelayuBERT")
  
  model = AutoModelForMaskedLM.from_pretrained("StevenLimcorn/MelayuBERT")
  ```
  ## Author
  Melayu BERT was trained by [Steven Limcorn](https://github.com/stevenlimcorn) and [Wilson Wongso](https://hf.co/w11wo).