ALLaM: Large Language Models for Arabic and English
<|reference_start|>ALLaM: Large Language Models for Arabic and English: We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.<|reference_end|>
@article{bari2024allam:,
    title={ALLaM: Large Language Models for Arabic and English},
    author={M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi,
  Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie,
  Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef
  Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed
  Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed
  Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb
  Alowisheq, Haidar Khan},
    journal={arXiv preprint arXiv:2407.15390},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15390},
    primaryClass={cs.CL cs.AI}
}