MDocAgent: A Multi-Modal Multi-Agent Framework for Document
  Understanding
Document Question Answering (DocQA) is a very common task. Existing methods
using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and
Retrieval Augmented Generation (RAG) often prioritize information from a single
modal, failing to effectively integrate textual and visual cues. These
approaches struggle with complex multi-modal reasoning, limiting their
performance on real-world documents. We present MDocAgent (A Multi-Modal
Multi-Agent Framework for Document Understanding), a novel RAG and multi-agent
framework that leverages both text and image. Our system employs five
specialized agents: a general agent, a critical agent, a text agent, an image
agent and a summarizing agent. These agents engage in multi-modal context
retrieval, combining their individual insights to achieve a more comprehensive
understanding of the document's content. This collaborative approach enables
the system to synthesize information from both textual and visual components,
leading to improved accuracy in question answering. Preliminary experiments on
five benchmarks like MMLongBench, LongDocURL demonstrate the effectiveness of
our MDocAgent, achieve an average improvement of 12.1% compared to current
state-of-the-art method. This work contributes to the development of more
robust and comprehensive DocQA systems capable of handling the complexities of
real-world documents containing rich textual and visual information. Our data
and code are available at https://github.com/aiming-lab/MDocAgent.
67e20852c0c932395394dbb0
Siwei Han
67e20852c0c932395394dbb1
Peng Xia
claimed_verified
2025-03-26T20:49:23.939Z
643e9ee6f6bb3c31a26e7bc4
/avatars/acfaa7d6a23dada24c86b954c3be116a.svg
Peng Xia
user
richardxp888
67e20852c0c932395394dbb2
Ruiyi Zhang
67e20852c0c932395394dbb3
Tong Sun
67e20852c0c932395394dbb4
Yun Li
67e20852c0c932395394dbb5
Hongtu Zhu
67e20852c0c932395394dbb6
Huaxiu Yao
MDocAgent: A Multi-Modal Multi-Agent Framework for Document
  Understanding
Document Question Answering (DocQA) is a very common task. Existing methods
using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and
Retrieval Augmented Generation (RAG) often prioritize information from a single
modal, failing to effectively integrate textual and visual cues. These
approaches struggle with complex multi-modal reasoning, limiting their
performance on real-world documents. We present MDocAgent (A Multi-Modal
Multi-Agent Framework for Document Understanding), a novel RAG and multi-agent
framework that leverages both text and image. Our system employs five
specialized agents: a general agent, a critical agent, a text agent, an image
agent and a summarizing agent. These agents engage in multi-modal context
retrieval, combining their individual insights to achieve a more comprehensive
understanding of the document's content. This collaborative approach enables
the system to synthesize information from both textual and visual components,
leading to improved accuracy in question answering. Preliminary experiments on
five benchmarks like MMLongBench, LongDocURL demonstrate the effectiveness of
our MDocAgent, achieve an average improvement of 12.1% compared to current
state-of-the-art method. This work contributes to the development of more
robust and comprehensive DocQA systems capable of handling the complexities of
real-world documents containing rich textual and visual information. Our data
and code are available at https://github.com/aiming-lab/MDocAgent.
MDocAgent, a multi-modal multi-agent framework, enhances document understanding by integrating both textual and visual cues, improving question answering accuracy through collaborative agent insights.
Large Language Models Large Vision Language Models Retrieval Augmented Generation multi-modal reasoning context retrieval general agent critical agent text agent image agent summarizing agent