EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models
<|reference_start|>EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models: In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks and PolyWrite, an open-ended generation benchmark developed in this study. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability.<|reference_end|>
@article{ji2024emma-500:,
    title={EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language
  Models},
    author={Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin,
  Pinzhen Chen, Dayy'an O'Brien, Hengyu Luo, Hinrich Sch"utze, J"org
  Tiedemann, Barry Haddow},
    journal={arXiv preprint arXiv:2409.17892},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17892},
    primaryClass={cs.CL}
}