title: VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal
    Retrieval-Augmented Generation
summary: Understanding information from a collection of multiple documents,
  particularly those with visually rich elements, is important for
  document-grounded question answering. This paper introduces VisDoMBench, the
  first comprehensive benchmark designed to evaluate QA systems in multi-document
  settings with rich multimodal content, including tables, charts, and
  presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval
  Augmented Generation (RAG) approach that simultaneously utilizes visual and
  textual RAG, combining robust visual retrieval capabilities with sophisticated
  linguistic reasoning. VisDoMRAG employs a multi-step reasoning process
  encompassing evidence curation and chain-of-thought reasoning for concurrent
  textual and visual RAG pipelines. A key novelty of VisDoMRAG is its
  consistency-constrained modality fusion mechanism, which aligns the reasoning
  processes across modalities at inference time to produce a coherent final
  answer. This leads to enhanced accuracy in scenarios where critical information
  is distributed across modalities and improved answer verifiability through
  implicit context attribution. Through extensive experiments involving
  open-source and proprietary large language models, we benchmark
  state-of-the-art document QA methods on VisDoMBench. Extensive results show
  that VisDoMRAG outperforms unimodal and long-context LLM baselines for
  end-to-end multimodal document QA by 12-20%.
paper_authors._id: 6762d715220f276ceedda74d
paper_authors.name: Manan Suri
paper_authors._id: 6762d715220f276ceedda74e
paper_authors.name: Puneet Mathur
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-18T14:07:19.812Z
paper_authors._id: 65c16444d4c3b8dff2f0d78d
paper_authors.avatarUrl: /avatars/4ed764c1657bd260d2a12ba61c111062.svg
paper_authors.fullname: Puneet Mathur
paper_authors.type: user
paper_authors.user: puneetm
paper_authors._id: 6762d715220f276ceedda74f
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T14:32:28.183Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 6762d715220f276ceedda750
paper_authors.name: Kanika Goswami
paper_authors._id: 6762d715220f276ceedda751
paper_authors.name: Ryan A. Rossi
paper_authors._id: 6762d715220f276ceedda752
paper_authors.name: Dinesh Manocha
paper_title: VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal
    Retrieval-Augmented Generation
paper_summary: Understanding information from a collection of multiple documents,
  particularly those with visually rich elements, is important for
  document-grounded question answering. This paper introduces VisDoMBench, the
  first comprehensive benchmark designed to evaluate QA systems in multi-document
  settings with rich multimodal content, including tables, charts, and
  presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval
  Augmented Generation (RAG) approach that simultaneously utilizes visual and
  textual RAG, combining robust visual retrieval capabilities with sophisticated
  linguistic reasoning. VisDoMRAG employs a multi-step reasoning process
  encompassing evidence curation and chain-of-thought reasoning for concurrent
  textual and visual RAG pipelines. A key novelty of VisDoMRAG is its
  consistency-constrained modality fusion mechanism, which aligns the reasoning
  processes across modalities at inference time to produce a coherent final
  answer. This leads to enhanced accuracy in scenarios where critical information
  is distributed across modalities and improved answer verifiability through
  implicit context attribution. Through extensive experiments involving
  open-source and proprietary large language models, we benchmark
  state-of-the-art document QA methods on VisDoMBench. Extensive results show
  that VisDoMRAG outperforms unimodal and long-context LLM baselines for
  end-to-end multimodal document QA by 12-20%.
paper_ai_keywords: VisDoMBench VisDoMRAG multimodal Retrieval Augmented Generation (RAG) visual retrieval sophisticated linguistic reasoning evidence curation chain-of-thought reasoning consistency-constrained modality fusion mechanism