title: MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents
summary: Multi-modal document retrieval is designed to identify and retrieve various
  forms of multi-modal content, such as figures, tables, charts, and layout
  information from extensive documents. Despite its significance, there is a
  notable lack of a robust benchmark to effectively evaluate the performance of
  systems in multi-modal document retrieval. To address this gap, this work
  introduces a new benchmark, named as MMDocIR, encompassing two distinct tasks:
  page-level and layout-level retrieval. The former focuses on localizing the
  most relevant pages within a long document, while the latter targets the
  detection of specific layouts, offering a more fine-grained granularity than
  whole-page analysis. A layout can refer to a variety of elements such as
  textual paragraphs, equations, figures, tables, or charts. The MMDocIR
  benchmark comprises a rich dataset featuring expertly annotated labels for
  1,685 questions and bootstrapped labels for 173,843 questions, making it a
  pivotal resource for advancing multi-modal document retrieval for both training
  and evaluation. Through rigorous experiments, we reveal that (i) visual
  retrievers significantly outperform their text counterparts, (ii) MMDocIR train
  set can effectively benefit the training process of multi-modal document
  retrieval and (iii) text retrievers leveraging on VLM-text perform much better
  than those using OCR-text. These findings underscores the potential advantages
  of integrating visual elements for multi-modal document retrieval.
paper_authors._id: 67889537383254ec3f017a1d
paper_authors.name: Kuicai Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T09:41:42.249Z
paper_authors._id: 66337cb5bd8ef15a47e72ce0
paper_authors.avatarUrl: /avatars/cc49056fcdc6bdabfe72a0d3de5c196d.svg
paper_authors.fullname: DONG KUICAI
paper_authors.type: user
paper_authors.user: daviddongdong
paper_authors._id: 67889537383254ec3f017a1e
paper_authors.name: Yujing Chang
paper_authors._id: 67889537383254ec3f017a1f
paper_authors.name: Xin Deik Goh
paper_authors._id: 67889537383254ec3f017a20
paper_authors.name: Dexun Li
paper_authors._id: 67889537383254ec3f017a21
paper_authors.name: Ruiming Tang
paper_authors._id: 67889537383254ec3f017a22
paper_authors.name: Yong Liu
paper_title: MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents
paper_summary: Multi-modal document retrieval is designed to identify and retrieve various
  forms of multi-modal content, such as figures, tables, charts, and layout
  information from extensive documents. Despite its significance, there is a
  notable lack of a robust benchmark to effectively evaluate the performance of
  systems in multi-modal document retrieval. To address this gap, this work
  introduces a new benchmark, named as MMDocIR, encompassing two distinct tasks:
  page-level and layout-level retrieval. The former focuses on localizing the
  most relevant pages within a long document, while the latter targets the
  detection of specific layouts, offering a more fine-grained granularity than
  whole-page analysis. A layout can refer to a variety of elements such as
  textual paragraphs, equations, figures, tables, or charts. The MMDocIR
  benchmark comprises a rich dataset featuring expertly annotated labels for
  1,685 questions and bootstrapped labels for 173,843 questions, making it a
  pivotal resource for advancing multi-modal document retrieval for both training
  and evaluation. Through rigorous experiments, we reveal that (i) visual
  retrievers significantly outperform their text counterparts, (ii) MMDocIR train
  set can effectively benefit the training process of multi-modal document
  retrieval and (iii) text retrievers leveraging on VLM-text perform much better
  than those using OCR-text. These findings underscores the potential advantages
  of integrating visual elements for multi-modal document retrieval.
paper_ai_keywords: multi-modal document retrieval figures tables charts layout information benchmark page-level retrieval layout-level retrieval textual paragraphs equations visual retrievers text retrievers VLM-text OCR-text