From Guidelines to Practice: A New Paradigm for Arabic Language Model
  Evaluation
This paper addresses critical gaps in Arabic language model evaluation by
establishing comprehensive theoretical guidelines and introducing a novel
evaluation framework. We first analyze existing Arabic evaluation datasets,
identifying significant issues in linguistic accuracy, cultural alignment, and
methodological rigor. To address these limitations in LLMs, we present the
Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490
challenging questions spanning ten major domains (42 sub-domains, see Figure 1.
Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet,
Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant
variations in model performance across different domains, with particular
challenges in areas requiring deep cultural understanding and specialized
knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%,
showing relative strength in mathematical theory in Arabic, Arabic language,
and islamic domains. This work provides both theoretical foundations and
practical insights for improving Arabic language model evaluation, emphasizing
the importance of cultural competence alongside technical capabilities.
683ec5047ec12b4ee9a21215
Serry Sibaee
admin_assigned
2025-06-03T12:41:17.495Z
63fc7fe6d44f50f559587f93
/avatars/0066af2fb1399a842c2ecca9e95b65dd.svg
Serry Sibaee
user
SerrySibaee
683ec5047ec12b4ee9a21216
Omer Nacar
admin_assigned
2025-06-03T12:41:40.815Z
628f7a71dd993507cfcbe587
https://cdn-avatars.huggingface.co/v1/production/uploads/628f7a71dd993507cfcbe587/3frAcMgfBCx-OJsrdAhnb.png
Omartificial Intelligence Space
user
Omartificial-Intelligence-Space
683ec5047ec12b4ee9a21217
Adel Ammar
admin_assigned
2025-06-03T12:41:49.306Z
647c60cd36e109abce3b3b15
/avatars/d702ec97cc2cde5ca7b5d16feb5f45c9.svg
Adel Ammar
user
ammaradel
683ec5047ec12b4ee9a21218
Yasser Al-Habashi
683ec5047ec12b4ee9a21219
Abdulrahman Al-Batati
683ec5047ec12b4ee9a2121a
Wadii Boulila
admin_assigned
2025-06-03T12:42:08.679Z
67bde4936c7c7de3a68608d1
/avatars/e08afba50d8aae8f7698ccba4c1e7b4a.svg
Wadii Boulila
user
wboulila
From Guidelines to Practice: A New Paradigm for Arabic Language Model
  Evaluation
This paper addresses critical gaps in Arabic language model evaluation by
establishing comprehensive theoretical guidelines and introducing a novel
evaluation framework. We first analyze existing Arabic evaluation datasets,
identifying significant issues in linguistic accuracy, cultural alignment, and
methodological rigor. To address these limitations in LLMs, we present the
Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490
challenging questions spanning ten major domains (42 sub-domains, see Figure 1.
Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet,
Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant
variations in model performance across different domains, with particular
challenges in areas requiring deep cultural understanding and specialized
knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%,
showing relative strength in mathematical theory in Arabic, Arabic language,
and islamic domains. This work provides both theoretical foundations and
practical insights for improving Arabic language model evaluation, emphasizing
the importance of cultural competence alongside technical capabilities.
A new evaluation framework and dataset, ADMD, are introduced to assess Arabic language models, highlighting variations in performance and emphasizing the importance of cultural competence.
evaluation framework Arabic Depth Mini Dataset (ADMD) GPT-4 Claude 3.5 Sonnet Gemini Flash 1.5 CommandR 100B Qwen-Max cultural understanding specialized knowledge cultural competence