NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural
<|reference_start|>NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural: Indonesia's linguistic landscape is remarkably diverse, encompassing over 700 languages and dialects, making it one of the world's most linguistically rich nations. This diversity, coupled with the widespread practice of code-switching and the presence of low-resource regional languages, presents unique challenges for modern pre-trained language models. In response to these challenges, we developed NusaBERT, building upon IndoBERT by incorporating vocabulary expansion and leveraging a diverse multilingual corpus that includes regional languages and dialects. Through rigorous evaluation across a range of benchmarks, NusaBERT demonstrates state-of-the-art performance in tasks involving multiple languages of Indonesia, paving the way for future natural language understanding research for under-represented languages.<|reference_end|>
@article{wongso2024nusabert:,
    title={NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural},
    author={Wilson Wongso, David Samuel Setiawan, Steven Limcorn, Ananto
  Joyoadikusumo},
    journal={arXiv preprint arXiv:2403.01817},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.01817},
    primaryClass={cs.CL}
}