title: From Guidelines to Practice: A New Paradigm for Arabic Language Model
    Evaluation
summary: This paper addresses critical gaps in Arabic language model evaluation by
  establishing comprehensive theoretical guidelines and introducing a novel
  evaluation framework. We first analyze existing Arabic evaluation datasets,
  identifying significant issues in linguistic accuracy, cultural alignment, and
  methodological rigor. To address these limitations in LLMs, we present the
  Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490
  challenging questions spanning ten major domains (42 sub-domains, see Figure 1.
  Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet,
  Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant
  variations in model performance across different domains, with particular
  challenges in areas requiring deep cultural understanding and specialized
  knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%,
  showing relative strength in mathematical theory in Arabic, Arabic language,
  and islamic domains. This work provides both theoretical foundations and
  practical insights for improving Arabic language model evaluation, emphasizing
  the importance of cultural competence alongside technical capabilities.
paper_authors._id: 683ec5047ec12b4ee9a21215
paper_authors.name: Serry Sibaee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:41:17.495Z
paper_authors._id: 63fc7fe6d44f50f559587f93
paper_authors.avatarUrl: /avatars/0066af2fb1399a842c2ecca9e95b65dd.svg
paper_authors.fullname: Serry Sibaee
paper_authors.type: user
paper_authors.user: SerrySibaee
paper_authors._id: 683ec5047ec12b4ee9a21216
paper_authors.name: Omer Nacar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:41:40.815Z
paper_authors._id: 628f7a71dd993507cfcbe587
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f7a71dd993507cfcbe587/3frAcMgfBCx-OJsrdAhnb.png
paper_authors.fullname: Omartificial Intelligence Space
paper_authors.type: user
paper_authors.user: Omartificial-Intelligence-Space
paper_authors._id: 683ec5047ec12b4ee9a21217
paper_authors.name: Adel Ammar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:41:49.306Z
paper_authors._id: 647c60cd36e109abce3b3b15
paper_authors.avatarUrl: /avatars/d702ec97cc2cde5ca7b5d16feb5f45c9.svg
paper_authors.fullname: Adel Ammar
paper_authors.type: user
paper_authors.user: ammaradel
paper_authors._id: 683ec5047ec12b4ee9a21218
paper_authors.name: Yasser Al-Habashi
paper_authors._id: 683ec5047ec12b4ee9a21219
paper_authors.name: Abdulrahman Al-Batati
paper_authors._id: 683ec5047ec12b4ee9a2121a
paper_authors.name: Wadii Boulila
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:42:08.679Z
paper_authors._id: 67bde4936c7c7de3a68608d1
paper_authors.avatarUrl: /avatars/e08afba50d8aae8f7698ccba4c1e7b4a.svg
paper_authors.fullname: Wadii Boulila
paper_authors.type: user
paper_authors.user: wboulila
paper_title: From Guidelines to Practice: A New Paradigm for Arabic Language Model
    Evaluation
paper_summary: This paper addresses critical gaps in Arabic language model evaluation by
  establishing comprehensive theoretical guidelines and introducing a novel
  evaluation framework. We first analyze existing Arabic evaluation datasets,
  identifying significant issues in linguistic accuracy, cultural alignment, and
  methodological rigor. To address these limitations in LLMs, we present the
  Arabic Depth Mini Dataset (ADMD), a carefully curated collection of 490
  challenging questions spanning ten major domains (42 sub-domains, see Figure 1.
  Using ADMD, we evaluate five leading language models: GPT-4, Claude 3.5 Sonnet,
  Gemini Flash 1.5, CommandR 100B, and Qwen-Max. Our results reveal significant
  variations in model performance across different domains, with particular
  challenges in areas requiring deep cultural understanding and specialized
  knowledge. Claude 3.5 Sonnet demonstrated the highest overall accuracy at 30\%,
  showing relative strength in mathematical theory in Arabic, Arabic language,
  and islamic domains. This work provides both theoretical foundations and
  practical insights for improving Arabic language model evaluation, emphasizing
  the importance of cultural competence alongside technical capabilities.
paper_ai_summary: A new evaluation framework and dataset, ADMD, are introduced to assess Arabic language models, highlighting variations in performance and emphasizing the importance of cultural competence.
paper_ai_keywords: evaluation framework Arabic Depth Mini Dataset (ADMD) GPT-4 Claude 3.5 Sonnet Gemini Flash 1.5 CommandR 100B Qwen-Max cultural understanding specialized knowledge cultural competence