OCR Hinders RAG: Evaluating the Cascading Impact of OCR on
  Retrieval-Augmented Generation
Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by
integrating external knowledge to reduce hallucinations and incorporate
up-to-date information without retraining. As an essential part of RAG,
external knowledge bases are commonly built by extracting structured data from
unstructured PDF documents using Optical Character Recognition (OCR). However,
given the imperfect prediction of OCR and the inherent non-uniform
representation of structured data, knowledge bases inevitably contain various
OCR noises. In this paper, we introduce OHRBench, the first benchmark for
understanding the cascading impact of OCR on RAG systems. OHRBench includes 350
carefully selected unstructured PDF documents from six real-world RAG
application domains, along with Q&As derived from multimodal elements in
documents, challenging existing OCR solutions used for RAG To better understand
OCR's impact on RAG systems, we identify two primary types of OCR noise:
Semantic Noise and Formatting Noise and apply perturbation to generate a set of
structured data with varying degrees of each OCR noise. Using OHRBench, we
first conduct a comprehensive evaluation of current OCR solutions and reveal
that none is competent for constructing high-quality knowledge bases for RAG
systems. We then systematically evaluate the impact of these two noise types
and demonstrate the vulnerability of RAG systems. Furthermore, we discuss the
potential of employing Vision-Language Models (VLMs) without OCR in RAG
systems. Code: https://github.com/opendatalab/OHR-Bench
674fc1a0c3d093ab09e84812
Junyuan Zhang
claimed_verified
2024-12-04T09:45:35.334Z
65411de281a8731a1c28293d
/avatars/18607dcd65303e1f688fae8015f43314.svg
junyuan
user
Carkham
674fc1a0c3d093ab09e84813
Qintong Zhang
admin_assigned
2024-12-04T12:03:34.385Z
66bd5c190dc132aeea87f3fb
/avatars/3c27b0efda647349d4621e8c82f61f9e.svg
ZQT
user
qintong21
674fc1a0c3d093ab09e84814
Bin Wang
claimed_verified
2024-12-05T09:28:19.974Z
63ae9ff5557befe297a76f90
https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
Bin Wang
user
wanderkid
674fc1a0c3d093ab09e84815
Linke Ouyang
admin_assigned
2024-12-04T12:03:47.897Z
66753c556f2ac48ee625d7d1
/avatars/8f7c252675fd8a096794d12971903722.svg
Linke Ouyang
user
ouyanglinke
674fc1a0c3d093ab09e84816
Zichen Wen
admin_assigned
2024-12-04T12:03:53.798Z
653b8c3e97a4d71d950e2f20
/avatars/b68880022e14556d0be58c69615db3be.svg
Zichen Wen
user
zichenwen
674fc1a0c3d093ab09e84817
Ying Li
admin_assigned
2024-12-04T12:04:04.033Z
669dcc055bd3f749a31a80b9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/G5khFVYMzouYpykpKWA-B.png
Ying Li
user
yingli-Claire
674fc1a0c3d093ab09e84818
Ka-Ho Chow
admin_assigned
2024-12-04T12:04:09.477Z
6511f6612f0aa026dd57791b
/avatars/80a4e79d9c48d6d7367300cd3d2824e2.svg
Ka Ho Chow
user
khchow
674fc1a0c3d093ab09e84819
Conghui He
admin_assigned
2024-12-04T12:04:14.536Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
674fc1a0c3d093ab09e8481a
Wentao Zhang
OCR Hinders RAG: Evaluating the Cascading Impact of OCR on
  Retrieval-Augmented Generation
Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by
integrating external knowledge to reduce hallucinations and incorporate
up-to-date information without retraining. As an essential part of RAG,
external knowledge bases are commonly built by extracting structured data from
unstructured PDF documents using Optical Character Recognition (OCR). However,
given the imperfect prediction of OCR and the inherent non-uniform
representation of structured data, knowledge bases inevitably contain various
OCR noises. In this paper, we introduce OHRBench, the first benchmark for
understanding the cascading impact of OCR on RAG systems. OHRBench includes 350
carefully selected unstructured PDF documents from six real-world RAG
application domains, along with Q&As derived from multimodal elements in
documents, challenging existing OCR solutions used for RAG To better understand
OCR's impact on RAG systems, we identify two primary types of OCR noise:
Semantic Noise and Formatting Noise and apply perturbation to generate a set of
structured data with varying degrees of each OCR noise. Using OHRBench, we
first conduct a comprehensive evaluation of current OCR solutions and reveal
that none is competent for constructing high-quality knowledge bases for RAG
systems. We then systematically evaluate the impact of these two noise types
and demonstrate the vulnerability of RAG systems. Furthermore, we discuss the
potential of employing Vision-Language Models (VLMs) without OCR in RAG
systems. Code: https://github.com/opendatalab/OHR-Bench
Introducing OHRBench to evaluate OCR's impact on Retrieval-augmented Generation (RAG) systems and explore the use of Vision-Language Models (VLMs) as an alternative to OCR.
Retrieval-augmented Generation (RAG) Large Language Models (LLMs) Optical Character Recognition (OCR) OHRBench knowledge bases PDF documents semantic noise formatting noise Vision-Language Models (VLMs)