KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and
  Document Understanding
With the growing adoption of Retrieval-Augmented Generation (RAG) in document
processing, robust text recognition has become increasingly critical for
knowledge extraction. While OCR (Optical Character Recognition) for English and
other languages benefits from large datasets and well-established benchmarks,
Arabic OCR faces unique challenges due to its cursive script, right-to-left
text flow, and complex typographic and calligraphic features. We present
KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in
current evaluation systems. Our benchmark comprises 8,809 samples across 9
major domains and 36 sub-domains, encompassing diverse document types including
handwritten text, structured tables, and specialized coverage of 21 chart types
for business intelligence. Our findings show that modern vision-language models
(such as GPT-4, Gemini, and Qwen) outperform traditional OCR approaches (like
EasyOCR, PaddleOCR, and Surya) by an average of 60% in Character Error Rate
(CER). Furthermore, we highlight significant limitations of current Arabic OCR
models, particularly in PDF-to-Markdown conversion, where the best model
Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges in
accurately recognizing Arabic text, including issues with complex fonts,
numeral recognition errors, word elongation, and table structure detection.
This work establishes a rigorous evaluation framework that can drive
improvements in Arabic document analysis methods and bridge the performance gap
with English OCR technologies.
67bc4ced7727595ca5b108f1
Ahmed Heakl
claimed_verified
2025-02-24T10:58:23.973Z
656864e12d73834278a8dea7
https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg
Ahmed Heakl
user
ahmedheakl
67bc4ced7727595ca5b108f2
Abdullah Sohail
claimed_verified
2025-02-24T13:05:04.733Z
672e4574b60c3a27d783a1ac
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/aut4W4hJcOT8jvQnlWs-y.png
Muhammad Abdullah
user
mabdullahsohail
67bc4ced7727595ca5b108f3
Mukul Ranjan
claimed_verified
2025-02-24T11:54:56.690Z
65262a396b41932089fd7bae
https://cdn-avatars.huggingface.co/v1/production/uploads/65262a396b41932089fd7bae/6YIEoAfJojuTW1UOKlwZT.png
Mukul Ranjan
user
mukul54
67bc4ced7727595ca5b108f4
Rania Hossam
67bc4ced7727595ca5b108f5
Ghazi Ahmed
67bc4ced7727595ca5b108f6
Mohamed El-Geish
claimed_verified
2025-04-22T10:16:22.469Z
5e796e1230dc073f817a2b92
https://cdn-avatars.huggingface.co/v1/production/uploads/5e796e1230dc073f817a2b92/3i3MVu2XezcCpEOOB-OQH.jpeg
Mohamed El-Geish
user
elgeish
67bc4ced7727595ca5b108f7
Omar Maher
67bc4ced7727595ca5b108f8
Zhiqiang Shen
67bc4ced7727595ca5b108f9
Fahad Khan
67bc4ced7727595ca5b108fa
Salman Khan
KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and
  Document Understanding
With the growing adoption of Retrieval-Augmented Generation (RAG) in document
processing, robust text recognition has become increasingly critical for
knowledge extraction. While OCR (Optical Character Recognition) for English and
other languages benefits from large datasets and well-established benchmarks,
Arabic OCR faces unique challenges due to its cursive script, right-to-left
text flow, and complex typographic and calligraphic features. We present
KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in
current evaluation systems. Our benchmark comprises 8,809 samples across 9
major domains and 36 sub-domains, encompassing diverse document types including
handwritten text, structured tables, and specialized coverage of 21 chart types
for business intelligence. Our findings show that modern vision-language models
(such as GPT-4, Gemini, and Qwen) outperform traditional OCR approaches (like
EasyOCR, PaddleOCR, and Surya) by an average of 60% in Character Error Rate
(CER). Furthermore, we highlight significant limitations of current Arabic OCR
models, particularly in PDF-to-Markdown conversion, where the best model
Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges in
accurately recognizing Arabic text, including issues with complex fonts,
numeral recognition errors, word elongation, and table structure detection.
This work establishes a rigorous evaluation framework that can drive
improvements in Arabic document analysis methods and bridge the performance gap
with English OCR technologies.
KITAB-Bench, a comprehensive Arabic OCR benchmark, highlights the strengths and limitations of modern vision-language models in recognizing Arabic text, especially in complex scenarios like PDF-to-Markdown conversion.
Retrieval-Augmented Generation RAG text recognition KITAB-Bench OCR Arabic OCR cursive script right-to-left text character error rate CER GPT-4 Gemini Qwen EasyOCR PaddleOCR Surya vision-language models PDF-to-Markdown complex fonts numeral recognition word elongation table structure detection