VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal
  Retrieval-Augmented Generation
Understanding information from a collection of multiple documents,
particularly those with visually rich elements, is important for
document-grounded question answering. This paper introduces VisDoMBench, the
first comprehensive benchmark designed to evaluate QA systems in multi-document
settings with rich multimodal content, including tables, charts, and
presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval
Augmented Generation (RAG) approach that simultaneously utilizes visual and
textual RAG, combining robust visual retrieval capabilities with sophisticated
linguistic reasoning. VisDoMRAG employs a multi-step reasoning process
encompassing evidence curation and chain-of-thought reasoning for concurrent
textual and visual RAG pipelines. A key novelty of VisDoMRAG is its
consistency-constrained modality fusion mechanism, which aligns the reasoning
processes across modalities at inference time to produce a coherent final
answer. This leads to enhanced accuracy in scenarios where critical information
is distributed across modalities and improved answer verifiability through
implicit context attribution. Through extensive experiments involving
open-source and proprietary large language models, we benchmark
state-of-the-art document QA methods on VisDoMBench. Extensive results show
that VisDoMRAG outperforms unimodal and long-context LLM baselines for
end-to-end multimodal document QA by 12-20%.
6762d715220f276ceedda74d
Manan Suri
6762d715220f276ceedda74e
Puneet Mathur
extracted_pending
2024-12-18T14:07:19.812Z
65c16444d4c3b8dff2f0d78d
/avatars/4ed764c1657bd260d2a12ba61c111062.svg
Puneet Mathur
user
puneetm
6762d715220f276ceedda74f
Franck Dernoncourt
claimed_verified
2024-12-18T14:32:28.183Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
6762d715220f276ceedda750
Kanika Goswami
6762d715220f276ceedda751
Ryan A. Rossi
6762d715220f276ceedda752
Dinesh Manocha
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal
  Retrieval-Augmented Generation
Understanding information from a collection of multiple documents,
particularly those with visually rich elements, is important for
document-grounded question answering. This paper introduces VisDoMBench, the
first comprehensive benchmark designed to evaluate QA systems in multi-document
settings with rich multimodal content, including tables, charts, and
presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval
Augmented Generation (RAG) approach that simultaneously utilizes visual and
textual RAG, combining robust visual retrieval capabilities with sophisticated
linguistic reasoning. VisDoMRAG employs a multi-step reasoning process
encompassing evidence curation and chain-of-thought reasoning for concurrent
textual and visual RAG pipelines. A key novelty of VisDoMRAG is its
consistency-constrained modality fusion mechanism, which aligns the reasoning
processes across modalities at inference time to produce a coherent final
answer. This leads to enhanced accuracy in scenarios where critical information
is distributed across modalities and improved answer verifiability through
implicit context attribution. Through extensive experiments involving
open-source and proprietary large language models, we benchmark
state-of-the-art document QA methods on VisDoMBench. Extensive results show
that VisDoMRAG outperforms unimodal and long-context LLM baselines for
end-to-end multimodal document QA by 12-20%.
VisDoMRAG, a multimodal Retrieval Augmented Generation model combining text and visual RAG, enhances document-grounded question answering in multimodal rich content settings by aligning reasoning processes across modalities.
multimodal benchmark VisDoMBench VisDoMRAG multimodal Retrieval Augmented Generation evidence curation chain-of-thought reasoning modality fusion consistency-constrained multimodal document QA