title: MDocAgent: A Multi-Modal Multi-Agent Framework for Document
    Understanding
summary: Document Question Answering (DocQA) is a very common task. Existing methods
  using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and
  Retrieval Augmented Generation (RAG) often prioritize information from a single
  modal, failing to effectively integrate textual and visual cues. These
  approaches struggle with complex multi-modal reasoning, limiting their
  performance on real-world documents. We present MDocAgent (A Multi-Modal
  Multi-Agent Framework for Document Understanding), a novel RAG and multi-agent
  framework that leverages both text and image. Our system employs five
  specialized agents: a general agent, a critical agent, a text agent, an image
  agent and a summarizing agent. These agents engage in multi-modal context
  retrieval, combining their individual insights to achieve a more comprehensive
  understanding of the document's content. This collaborative approach enables
  the system to synthesize information from both textual and visual components,
  leading to improved accuracy in question answering. Preliminary experiments on
  five benchmarks like MMLongBench, LongDocURL demonstrate the effectiveness of
  our MDocAgent, achieve an average improvement of 12.1% compared to current
  state-of-the-art method. This work contributes to the development of more
  robust and comprehensive DocQA systems capable of handling the complexities of
  real-world documents containing rich textual and visual information. Our data
  and code are available at https://github.com/aiming-lab/MDocAgent.
paper_authors._id: 67e20852c0c932395394dbb0
paper_authors.name: Siwei Han
paper_authors._id: 67e20852c0c932395394dbb1
paper_authors.name: Peng Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:49:23.939Z
paper_authors._id: 643e9ee6f6bb3c31a26e7bc4
paper_authors.avatarUrl: /avatars/acfaa7d6a23dada24c86b954c3be116a.svg
paper_authors.fullname: Peng Xia
paper_authors.type: user
paper_authors.user: richardxp888
paper_authors._id: 67e20852c0c932395394dbb2
paper_authors.name: Ruiyi Zhang
paper_authors._id: 67e20852c0c932395394dbb3
paper_authors.name: Tong Sun
paper_authors._id: 67e20852c0c932395394dbb4
paper_authors.name: Yun Li
paper_authors._id: 67e20852c0c932395394dbb5
paper_authors.name: Hongtu Zhu
paper_authors._id: 67e20852c0c932395394dbb6
paper_authors.name: Huaxiu Yao
paper_title: MDocAgent: A Multi-Modal Multi-Agent Framework for Document
    Understanding
paper_summary: Document Question Answering (DocQA) is a very common task. Existing methods
  using Large Language Models (LLMs) or Large Vision Language Models (LVLMs) and
  Retrieval Augmented Generation (RAG) often prioritize information from a single
  modal, failing to effectively integrate textual and visual cues. These
  approaches struggle with complex multi-modal reasoning, limiting their
  performance on real-world documents. We present MDocAgent (A Multi-Modal
  Multi-Agent Framework for Document Understanding), a novel RAG and multi-agent
  framework that leverages both text and image. Our system employs five
  specialized agents: a general agent, a critical agent, a text agent, an image
  agent and a summarizing agent. These agents engage in multi-modal context
  retrieval, combining their individual insights to achieve a more comprehensive
  understanding of the document's content. This collaborative approach enables
  the system to synthesize information from both textual and visual components,
  leading to improved accuracy in question answering. Preliminary experiments on
  five benchmarks like MMLongBench, LongDocURL demonstrate the effectiveness of
  our MDocAgent, achieve an average improvement of 12.1% compared to current
  state-of-the-art method. This work contributes to the development of more
  robust and comprehensive DocQA systems capable of handling the complexities of
  real-world documents containing rich textual and visual information. Our data
  and code are available at https://github.com/aiming-lab/MDocAgent.
paper_ai_summary: MDocAgent, a multi-modal multi-agent framework, enhances document understanding by integrating both textual and visual cues, improving question answering accuracy through collaborative agent insights.
paper_ai_keywords: Large Language Models Large Vision Language Models Retrieval Augmented Generation multi-modal reasoning context retrieval general agent critical agent text agent image agent summarizing agent