CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
Recent years have witnessed a significant interest in developing large
multimodal models (LMMs) capable of performing various visual reasoning and
understanding tasks. This has led to the introduction of multiple LMM
benchmarks to evaluate LMMs on different tasks. However, most existing LMM
evaluation benchmarks are predominantly English-centric. In this work, we
develop a comprehensive LMM evaluation benchmark for the Arabic language to
represent a large population of over 400 million speakers. The proposed
benchmark, named CAMEL-Bench, comprises eight diverse domains and 38
sub-domains including, multi-image understanding, complex visual perception,
handwritten document understanding, video understanding, medical imaging, plant
diseases, and remote sensing-based land use understanding to evaluate broad
scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions
that are filtered from a larger pool of samples, where the quality is manually
verified by native speakers to ensure reliable model assessment. We conduct
evaluations of both closed-source, including GPT-4 series, and open-source
LMMs. Our analysis reveals the need for substantial improvement, especially
among the best open-source models, with even the closed-source GPT-4o achieving
an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.
671b3cc85f7cde5ac31f34cf
Sara Ghaboura
claimed_verified
2025-02-06T16:06:28.185Z
66d559103c5bc37ee0dfa61b
/avatars/8310fc0b01e6d8873aec37ba9ef27c5b.svg
SaraG
user
SLMLAH
671b3cc85f7cde5ac31f34d0
Ahmed Heakl
claimed_verified
2024-10-25T09:28:34.074Z
656864e12d73834278a8dea7
https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg
Ahmed Heakl
user
ahmedheakl
671b3cc85f7cde5ac31f34d1
Omkar Thawakar
671b3cc85f7cde5ac31f34d2
Ali Alharthi
671b3cc85f7cde5ac31f34d3
Ines Riahi
671b3cc85f7cde5ac31f34d4
Abduljalil Saif
671b3cc85f7cde5ac31f34d5
Jorma Laaksonen
671b3cc85f7cde5ac31f34d6
Fahad S. Khan
671b3cc85f7cde5ac31f34d7
Salman Khan
671b3cc85f7cde5ac31f34d8
Rao M. Anwer
CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
Recent years have witnessed a significant interest in developing large
multimodal models (LMMs) capable of performing various visual reasoning and
understanding tasks. This has led to the introduction of multiple LMM
benchmarks to evaluate LMMs on different tasks. However, most existing LMM
evaluation benchmarks are predominantly English-centric. In this work, we
develop a comprehensive LMM evaluation benchmark for the Arabic language to
represent a large population of over 400 million speakers. The proposed
benchmark, named CAMEL-Bench, comprises eight diverse domains and 38
sub-domains including, multi-image understanding, complex visual perception,
handwritten document understanding, video understanding, medical imaging, plant
diseases, and remote sensing-based land use understanding to evaluate broad
scenario generalizability. Our CAMEL-Bench comprises around 29,036 questions
that are filtered from a larger pool of samples, where the quality is manually
verified by native speakers to ensure reliable model assessment. We conduct
evaluations of both closed-source, including GPT-4 series, and open-source
LMMs. Our analysis reveals the need for substantial improvement, especially
among the best open-source models, with even the closed-source GPT-4o achieving
an overall score of 62%. Our benchmark and evaluation scripts are open-sourced.
Researchers developed CAMEL-Bench, a comprehensive Arabic multimodal evaluation benchmark, to assess visual reasoning and understanding models across diverse domains and scenarios.
large multimodal models LMM benchmarks multi-image understanding complex visual perception handwritten document understanding video understanding medical imaging plant diseases remote sensing generalizability model assessment closed-source models open-source models GPT-4 series