title: CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
summary: Recent years have witnessed a significant interest in developing large
  multimodal models (LMMs) capable of performing various visual reasoning and
  understanding tasks. This has led to the introduction of multiple LMM
  benchmarks to evaluate LMMs on different tasks. However, most existing LMM
  evaluation benchmarks are predominantly English-centric. In this work, we
  develop a comprehensive LMM evaluation benchmark for the Arabic language to
  represent a large population of over 400 million speakers. The proposed
  benchmark, named CAMEL-Bench, comprises eight diverse domains and 38
  sub-domains including, multi-image understanding, complex visual perception,
  handwritten document understanding, video understanding, medical imaging, plant
  diseases, and remote sensing-based land use understanding to evaluate broad
  scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions
  that are filtered from a larger pool of samples, where the quality is manually
  verified by native speakers to ensure reliable model assessment. We conduct
  evaluations of both closed-source, including GPT-4 series, and open-source
  LMMs. Our analysis reveals the need for substantial improvement, especially
  among the best open-source models, with even the closed-source GPT-4o achieving
  an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.
paper_authors._id: 671b3cc85f7cde5ac31f34cf
paper_authors.name: Sara Ghaboura
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T16:06:28.185Z
paper_authors._id: 66d559103c5bc37ee0dfa61b
paper_authors.avatarUrl: /avatars/8310fc0b01e6d8873aec37ba9ef27c5b.svg
paper_authors.fullname: SaraG
paper_authors.type: user
paper_authors.user: SLMLAH
paper_authors._id: 671b3cc85f7cde5ac31f34d0
paper_authors.name: Ahmed Heakl
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:28:34.074Z
paper_authors._id: 656864e12d73834278a8dea7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg
paper_authors.fullname: Ahmed Heakl
paper_authors.type: user
paper_authors.user: ahmedheakl
paper_authors._id: 671b3cc85f7cde5ac31f34d1
paper_authors.name: Omkar Thawakar
paper_authors._id: 671b3cc85f7cde5ac31f34d2
paper_authors.name: Ali Alharthi
paper_authors._id: 671b3cc85f7cde5ac31f34d3
paper_authors.name: Ines Riahi
paper_authors._id: 671b3cc85f7cde5ac31f34d4
paper_authors.name: Abduljalil Saif
paper_authors._id: 671b3cc85f7cde5ac31f34d5
paper_authors.name: Jorma Laaksonen
paper_authors._id: 671b3cc85f7cde5ac31f34d6
paper_authors.name: Fahad S. Khan
paper_authors._id: 671b3cc85f7cde5ac31f34d7
paper_authors.name: Salman Khan
paper_authors._id: 671b3cc85f7cde5ac31f34d8
paper_authors.name: Rao M. Anwer
paper_title: CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
paper_summary: Recent years have witnessed a significant interest in developing large
  multimodal models (LMMs) capable of performing various visual reasoning and
  understanding tasks. This has led to the introduction of multiple LMM
  benchmarks to evaluate LMMs on different tasks. However, most existing LMM
  evaluation benchmarks are predominantly English-centric. In this work, we
  develop a comprehensive LMM evaluation benchmark for the Arabic language to
  represent a large population of over 400 million speakers. The proposed
  benchmark, named CAMEL-Bench, comprises eight diverse domains and 38
  sub-domains including, multi-image understanding, complex visual perception,
  handwritten document understanding, video understanding, medical imaging, plant
  diseases, and remote sensing-based land use understanding to evaluate broad
  scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions
  that are filtered from a larger pool of samples, where the quality is manually
  verified by native speakers to ensure reliable model assessment. We conduct
  evaluations of both closed-source, including GPT-4 series, and open-source
  LMMs. Our analysis reveals the need for substantial improvement, especially
  among the best open-source models, with even the closed-source GPT-4o achieving
  an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.
paper_ai_keywords: large multimodal models (LMMs) CAMEL-Bench multi-image understanding complex visual perception handwritten document understanding video understanding medical imaging plant diseases remote sensing-based land use understanding scenario generalizability native speakers closed-source open-source LMMs GPT-4 overall score