MedAgent-Pro: Towards Multi-modal Evidence-based Medical Diagnosis via
  Reasoning Agentic Workflow
Developing reliable AI systems to assist human clinicians in multi-modal
medical diagnosis has long been a key objective for researchers. Recently,
Multi-modal Large Language Models (MLLMs) have gained significant attention and
achieved success across various domains. With strong reasoning capabilities and
the ability to perform diverse tasks based on user instructions, they hold
great potential for enhancing medical diagnosis. However, directly applying
MLLMs to the medical domain still presents challenges. They lack detailed
perception of visual inputs, limiting their ability to perform quantitative
image analysis, which is crucial for medical diagnostics. Additionally, MLLMs
often exhibit hallucinations and inconsistencies in reasoning, whereas clinical
diagnoses must adhere strictly to established criteria. To address these
challenges, we propose MedAgent-Pro, an evidence-based reasoning agentic system
designed to achieve reliable, explainable, and precise medical diagnoses. This
is accomplished through a hierarchical workflow: at the task level,
knowledge-based reasoning generate reliable diagnostic plans for specific
diseases following retrieved clinical criteria. While at the case level,
multiple tool agents process multi-modal inputs, analyze different indicators
according to the plan, and provide a final diagnosis based on both quantitative
and qualitative evidence. Comprehensive experiments on both 2D and 3D medical
diagnosis tasks demonstrate the superiority and effectiveness of MedAgent-Pro,
while case studies further highlight its reliability and interpretability. The
code is available at https://github.com/jinlab-imvr/MedAgent-Pro.
67ea0ede7b856e8fa8ff50d0
Ziyue Wang
admin_assigned
2025-03-31T09:59:13.929Z
65be4d7d5e342a230dc19a54
/avatars/04ba16980da94954d811032f0091212f.svg
Ziyue Wang
user
ZiyueWang
67ea0ede7b856e8fa8ff50d1
Junde Wu
extracted_pending
2025-03-31T03:41:20.293Z
6317257fc92fd6fee317ff7c
/avatars/2f460a2f28562c987becb2acad8d93e7.svg
Junde Wu
user
morson
67ea0ede7b856e8fa8ff50d2
Chang Han Low
67ea0ede7b856e8fa8ff50d3
Yueming Jin
MedAgent-Pro: Towards Multi-modal Evidence-based Medical Diagnosis via
  Reasoning Agentic Workflow
Developing reliable AI systems to assist human clinicians in multi-modal
medical diagnosis has long been a key objective for researchers. Recently,
Multi-modal Large Language Models (MLLMs) have gained significant attention and
achieved success across various domains. With strong reasoning capabilities and
the ability to perform diverse tasks based on user instructions, they hold
great potential for enhancing medical diagnosis. However, directly applying
MLLMs to the medical domain still presents challenges. They lack detailed
perception of visual inputs, limiting their ability to perform quantitative
image analysis, which is crucial for medical diagnostics. Additionally, MLLMs
often exhibit hallucinations and inconsistencies in reasoning, whereas clinical
diagnoses must adhere strictly to established criteria. To address these
challenges, we propose MedAgent-Pro, an evidence-based reasoning agentic system
designed to achieve reliable, explainable, and precise medical diagnoses. This
is accomplished through a hierarchical workflow: at the task level,
knowledge-based reasoning generate reliable diagnostic plans for specific
diseases following retrieved clinical criteria. While at the case level,
multiple tool agents process multi-modal inputs, analyze different indicators
according to the plan, and provide a final diagnosis based on both quantitative
and qualitative evidence. Comprehensive experiments on both 2D and 3D medical
diagnosis tasks demonstrate the superiority and effectiveness of MedAgent-Pro,
while case studies further highlight its reliability and interpretability. The
code is available at https://github.com/jinlab-imvr/MedAgent-Pro.
MedAgent-Pro addresses challenges in multi-modal medical diagnosis by combining knowledge-based reasoning with tool agents for precise and explainable diagnoses.
Multi-modal Large Language Models MLMLs evidence-based reasoning enigmatic reasoning clinical criteria hierarchical workflow knowledge-based reasoning tool agents multi-modal inputs quantitative analysis qualitative evidence