title: M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page
    Multi-document Understanding
summary: Document visual question answering (DocVQA) pipelines that answer questions
  from documents have broad applications. Existing methods focus on handling
  single-page documents with multi-modal language models (MLMs), or rely on
  text-based retrieval-augmented generation (RAG) that uses text extraction tools
  such as optical character recognition (OCR). However, there are difficulties in
  applying these methods in real-world scenarios: (a) questions often require
  information across different pages or documents, where MLMs cannot handle many
  long documents; (b) documents often have important information in visual
  elements such as figures, but text extraction tools ignore them. We introduce
  M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various
  document contexts (closed-domain and open-domain), question hops (single-hop
  and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG
  finds relevant documents and answers questions using a multi-modal retriever
  and an MLM, so that it can efficiently handle single or many documents while
  preserving visual information. Since previous DocVQA datasets ask questions in
  the context of a specific document, we also present M3DocVQA, a new benchmark
  for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages.
  In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results
  show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance
  than many strong baselines, including state-of-the-art performance in
  MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and
  retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully
  handle various scenarios, such as when relevant information exists across
  multiple pages and when answer evidence only exists in images.
paper_authors._id: 672e3c07b2f2dc21e1aa49bb
paper_authors.name: Jaemin Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T08:34:28.939Z
paper_authors._id: 5ffe32d8942cf3533d364449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
paper_authors.fullname: Jaemin Cho
paper_authors.type: user
paper_authors.user: j-min
paper_authors._id: 672e3c07b2f2dc21e1aa49bc
paper_authors.name: Debanjan Mahata
paper_authors._id: 672e3c07b2f2dc21e1aa49bd
paper_authors.name: Ozan Irsoy
paper_authors._id: 672e3c07b2f2dc21e1aa49be
paper_authors.name: Yujie He
paper_authors._id: 672e3c07b2f2dc21e1aa49bf
paper_authors.name: Mohit Bansal
paper_title: M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page
    Multi-document Understanding
paper_summary: Document visual question answering (DocVQA) pipelines that answer questions
  from documents have broad applications. Existing methods focus on handling
  single-page documents with multi-modal language models (MLMs), or rely on
  text-based retrieval-augmented generation (RAG) that uses text extraction tools
  such as optical character recognition (OCR). However, there are difficulties in
  applying these methods in real-world scenarios: (a) questions often require
  information across different pages or documents, where MLMs cannot handle many
  long documents; (b) documents often have important information in visual
  elements such as figures, but text extraction tools ignore them. We introduce
  M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various
  document contexts (closed-domain and open-domain), question hops (single-hop
  and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG
  finds relevant documents and answers questions using a multi-modal retriever
  and an MLM, so that it can efficiently handle single or many documents while
  preserving visual information. Since previous DocVQA datasets ask questions in
  the context of a specific document, we also present M3DocVQA, a new benchmark
  for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages.
  In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results
  show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance
  than many strong baselines, including state-of-the-art performance in
  MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and
  retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully
  handle various scenarios, such as when relevant information exists across
  multiple pages and when answer evidence only exists in images.
paper_ai_keywords: multi-modal RAG framework multi-modal retriever ColPali Qwen2-VL 7B open-domain DocVQA visual information indexing empirical results state-of-the-art performance