title: OCR Hinders RAG: Evaluating the Cascading Impact of OCR on
    Retrieval-Augmented Generation
summary: Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by
  integrating external knowledge to reduce hallucinations and incorporate
  up-to-date information without retraining. As an essential part of RAG,
  external knowledge bases are commonly built by extracting structured data from
  unstructured PDF documents using Optical Character Recognition (OCR). However,
  given the imperfect prediction of OCR and the inherent non-uniform
  representation of structured data, knowledge bases inevitably contain various
  OCR noises. In this paper, we introduce OHRBench, the first benchmark for
  understanding the cascading impact of OCR on RAG systems. OHRBench includes 350
  carefully selected unstructured PDF documents from six real-world RAG
  application domains, along with Q&As derived from multimodal elements in
  documents, challenging existing OCR solutions used for RAG To better understand
  OCR's impact on RAG systems, we identify two primary types of OCR noise:
  Semantic Noise and Formatting Noise and apply perturbation to generate a set of
  structured data with varying degrees of each OCR noise. Using OHRBench, we
  first conduct a comprehensive evaluation of current OCR solutions and reveal
  that none is competent for constructing high-quality knowledge bases for RAG
  systems. We then systematically evaluate the impact of these two noise types
  and demonstrate the vulnerability of RAG systems. Furthermore, we discuss the
  potential of employing Vision-Language Models (VLMs) without OCR in RAG
  systems. Code: https://github.com/opendatalab/OHR-Bench
paper_authors._id: 674fc1a0c3d093ab09e84812
paper_authors.name: Junyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:45:35.334Z
paper_authors._id: 65411de281a8731a1c28293d
paper_authors.avatarUrl: /avatars/18607dcd65303e1f688fae8015f43314.svg
paper_authors.fullname: junyuan
paper_authors.type: user
paper_authors.user: Carkham
paper_authors._id: 674fc1a0c3d093ab09e84813
paper_authors.name: Qintong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:03:34.385Z
paper_authors._id: 66bd5c190dc132aeea87f3fb
paper_authors.avatarUrl: /avatars/3c27b0efda647349d4621e8c82f61f9e.svg
paper_authors.fullname: ZQT
paper_authors.type: user
paper_authors.user: qintong21
paper_authors._id: 674fc1a0c3d093ab09e84814
paper_authors.name: Bin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:19.974Z
paper_authors._id: 63ae9ff5557befe297a76f90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
paper_authors.fullname: Bin Wang
paper_authors.type: user
paper_authors.user: wanderkid
paper_authors._id: 674fc1a0c3d093ab09e84815
paper_authors.name: Linke Ouyang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:03:47.897Z
paper_authors._id: 66753c556f2ac48ee625d7d1
paper_authors.avatarUrl: /avatars/8f7c252675fd8a096794d12971903722.svg
paper_authors.fullname: Linke Ouyang
paper_authors.type: user
paper_authors.user: ouyanglinke
paper_authors._id: 674fc1a0c3d093ab09e84816
paper_authors.name: Zichen Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:03:53.798Z
paper_authors._id: 653b8c3e97a4d71d950e2f20
paper_authors.avatarUrl: /avatars/b68880022e14556d0be58c69615db3be.svg
paper_authors.fullname: Zichen Wen
paper_authors.type: user
paper_authors.user: zichenwen
paper_authors._id: 674fc1a0c3d093ab09e84817
paper_authors.name: Ying Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:04:04.033Z
paper_authors._id: 669dcc055bd3f749a31a80b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/G5khFVYMzouYpykpKWA-B.png
paper_authors.fullname: Ying Li
paper_authors.type: user
paper_authors.user: yingli-Claire
paper_authors._id: 674fc1a0c3d093ab09e84818
paper_authors.name: Ka-Ho Chow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:04:09.477Z
paper_authors._id: 6511f6612f0aa026dd57791b
paper_authors.avatarUrl: /avatars/80a4e79d9c48d6d7367300cd3d2824e2.svg
paper_authors.fullname: Ka Ho Chow
paper_authors.type: user
paper_authors.user: khchow
paper_authors._id: 674fc1a0c3d093ab09e84819
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:04:14.536Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 674fc1a0c3d093ab09e8481a
paper_authors.name: Wentao Zhang
paper_title: OCR Hinders RAG: Evaluating the Cascading Impact of OCR on
    Retrieval-Augmented Generation
paper_summary: Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by
  integrating external knowledge to reduce hallucinations and incorporate
  up-to-date information without retraining. As an essential part of RAG,
  external knowledge bases are commonly built by extracting structured data from
  unstructured PDF documents using Optical Character Recognition (OCR). However,
  given the imperfect prediction of OCR and the inherent non-uniform
  representation of structured data, knowledge bases inevitably contain various
  OCR noises. In this paper, we introduce OHRBench, the first benchmark for
  understanding the cascading impact of OCR on RAG systems. OHRBench includes 350
  carefully selected unstructured PDF documents from six real-world RAG
  application domains, along with Q&As derived from multimodal elements in
  documents, challenging existing OCR solutions used for RAG To better understand
  OCR's impact on RAG systems, we identify two primary types of OCR noise:
  Semantic Noise and Formatting Noise and apply perturbation to generate a set of
  structured data with varying degrees of each OCR noise. Using OHRBench, we
  first conduct a comprehensive evaluation of current OCR solutions and reveal
  that none is competent for constructing high-quality knowledge bases for RAG
  systems. We then systematically evaluate the impact of these two noise types
  and demonstrate the vulnerability of RAG systems. Furthermore, we discuss the
  potential of employing Vision-Language Models (VLMs) without OCR in RAG
  systems. Code: https://github.com/opendatalab/OHR-Bench
paper_ai_keywords: Retrieval-augmented Generation (RAG) Large Language Models (LLMs) Optical Character Recognition (OCR) Semantic Noise Formatting Noise Vision-Language Models (VLMs)