Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan
  Arabic Dialect
We introduce Atlas-Chat, the first-ever collection of large language models
specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also
known as Darija, we construct our instruction dataset by consolidating existing
Darija language resources, creating novel datasets both manually and
synthetically, and translating English instructions with stringent quality
control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit
superior ability in following Darija instructions and performing standard NLP
tasks. Notably, our models outperform both state-of-the-art and
Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13%
performance boost over a larger 13B model on DarijaMMLU, in our newly
introduced evaluation suite for Darija covering both discriminative and
generative tasks. Furthermore, we perform an experimental analysis of various
fine-tuning strategies and base model choices to determine optimal
configurations. All our resources are publicly accessible, and we believe our
work offers comprehensive design methodologies of instruction-tuning for
low-resource language variants, which are often neglected in favor of data-rich
languages by contemporary LLMs.
66f650dfb35008620e600b6f
Guokan Shang
claimed_verified
2024-09-27T13:47:58.578Z
6087e598e2b7cc3a117b0dc5
https://cdn-avatars.huggingface.co/v1/production/uploads/6087e598e2b7cc3a117b0dc5/Ctz_W-uo1gOQRBHXalD1P.png
Guokan Shang
user
guokan-shang
66f650dfb35008620e600b70
Hadi Abdine
claimed_verified
2024-09-27T06:57:58.895Z
66448b4fecac3bc79b26304f
https://cdn-avatars.huggingface.co/v1/production/uploads/66448b4fecac3bc79b26304f/aPH3UFbc20CL2Bz2yn7nH.jpeg
Hadi Abdine
user
habdine
66f650dfb35008620e600b71
Yousef Khoubrane
claimed_verified
2024-09-30T08:07:07.298Z
6617fd349d3673a2621d6ea5
https://cdn-avatars.huggingface.co/v1/production/uploads/6617fd349d3673a2621d6ea5/-PXtZY3y2zWbaRx9qFc39.jpeg
Yousef Khoubrane
user
khoubrane-yousef
66f650dfb35008620e600b72
Amr Mohamed
claimed_verified
2024-09-30T12:08:31.573Z
655efd24afee0e00788bb589
https://cdn-avatars.huggingface.co/v1/production/uploads/655efd24afee0e00788bb589/22guLxIWNybbJR3jI-c4w.jpeg
Amr Mohamed
user
amr-mohamed
66f650dfb35008620e600b73
Yassine Abbahaddou
66f650dfb35008620e600b74
Sofiane Ennadir
66f650dfb35008620e600b75
Imane Momayiz
admin_assigned
2024-10-02T08:44:58.056Z
64f78a17ceabf1e6fc352fbc
https://cdn-avatars.huggingface.co/v1/production/uploads/64f78a17ceabf1e6fc352fbc/KFEGWmhC7PGNwbVzqoEJB.jpeg
Imane Momayiz
user
imomayiz
66f650dfb35008620e600b76
Xuguang Ren
66f650dfb35008620e600b77
Eric Moulines
66f650dfb35008620e600b78
Preslav Nakov
admin_assigned
2024-10-02T08:43:45.303Z
647f7eb25e1bc4753746bf9f
/avatars/cc9c6210fdc822d8a106937e747dff41.svg
Preslav Nakov
user
preslavnakov
66f650dfb35008620e600b79
Michalis Vazirgiannis
66f650dfb35008620e600b7a
Eric Xing
admin_assigned
2024-10-02T08:44:14.611Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan
  Arabic Dialect
We introduce Atlas-Chat, the first-ever collection of large language models
specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also
known as Darija, we construct our instruction dataset by consolidating existing
Darija language resources, creating novel datasets both manually and
synthetically, and translating English instructions with stringent quality
control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit
superior ability in following Darija instructions and performing standard NLP
tasks. Notably, our models outperform both state-of-the-art and
Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13%
performance boost over a larger 13B model on DarijaMMLU, in our newly
introduced evaluation suite for Darija covering both discriminative and
generative tasks. Furthermore, we perform an experimental analysis of various
fine-tuning strategies and base model choices to determine optimal
configurations. All our resources are publicly accessible, and we believe our
work offers comprehensive design methodologies of instruction-tuning for
low-resource language variants, which are often neglected in favor of data-rich
languages by contemporary LLMs.
Atlas-Chat, a collection of large language models fine-tuned for Moroccan Arabic (Darija), outperforms state-of-the-art LLMs on Darija-specific tasks.
large language models dialectal Arabic Moroccan Arabic Darija instruction dataset fine-tuned NLP tasks LLaMa Jais AceGPT DarijaMMLU fine-tuning strategies base model choices instruction-tuning low-resource language variants