title: MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language
    Models
summary: Artificial Intelligence (AI) has demonstrated significant potential in
  healthcare, particularly in disease diagnosis and treatment planning. Recent
  progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new
  possibilities for interactive diagnostic tools. However, these models often
  suffer from factual hallucination, which can lead to incorrect diagnoses.
  Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to
  address these issues. However, the amount of high-quality data and distribution
  shifts between training data and deployment data limit the application of
  fine-tuning methods. Although RAG is lightweight and effective, existing
  RAG-based approaches are not sufficiently general to different medical domains
  and can potentially cause misalignment issues, both between modalities and
  between the model and the ground truth. In this paper, we propose a versatile
  multimodal RAG system, MMed-RAG, designed to enhance the factuality of
  Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an
  adaptive retrieved contexts selection method, and a provable RAG-based
  preference fine-tuning strategy. These innovations make the RAG process
  sufficiently general and reliable, significantly improving alignment when
  introducing retrieved contexts. Experimental results across five medical
  datasets (involving radiology, ophthalmology, pathology) on medical VQA and
  report generation demonstrate that MMed-RAG can achieve an average improvement
  of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available
  in https://github.com/richard-peng-xia/MMed-RAG.
paper_authors._id: 6711c6c03a39fce88e963f74
paper_authors.name: Peng Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:31.084Z
paper_authors._id: 643e9ee6f6bb3c31a26e7bc4
paper_authors.avatarUrl: /avatars/acfaa7d6a23dada24c86b954c3be116a.svg
paper_authors.fullname: Peng Xia
paper_authors.type: user
paper_authors.user: richardxp888
paper_authors._id: 6711c6c03a39fce88e963f75
paper_authors.name: Kangyu Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:39:50.268Z
paper_authors._id: 661af01a8ad066992403cc6f
paper_authors.avatarUrl: /avatars/b4680010940a84c23521aa4b043ccab4.svg
paper_authors.fullname: Kangyu Zhu
paper_authors.type: user
paper_authors.user: zky11235
paper_authors._id: 6711c6c03a39fce88e963f76
paper_authors.name: Haoran Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:48:24.989Z
paper_authors._id: 64e86d6e7ef2c72cbcdb27b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e86d6e7ef2c72cbcdb27b7/RLuJJ1xuO665LMWneVDZz.jpeg
paper_authors.fullname: Haoran Li
paper_authors.type: user
paper_authors.user: brian13
paper_authors._id: 6711c6c03a39fce88e963f77
paper_authors.name: Tianze Wang
paper_authors._id: 6711c6c03a39fce88e963f78
paper_authors.name: Weijia Shi
paper_authors._id: 6711c6c03a39fce88e963f79
paper_authors.name: Sheng Wang
paper_authors._id: 6711c6c03a39fce88e963f7a
paper_authors.name: Linjun Zhang
paper_authors._id: 6711c6c03a39fce88e963f7b
paper_authors.name: James Zou
paper_authors._id: 6711c6c03a39fce88e963f7c
paper_authors.name: Huaxiu Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:41:17.646Z
paper_authors._id: 65f667a455009c4ad9e6ac4c
paper_authors.avatarUrl: /avatars/ed10c2cf2ba3fde6d7da93f076961607.svg
paper_authors.fullname: Yao
paper_authors.type: user
paper_authors.user: Huaxiu
paper_title: MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language
    Models
paper_summary: Artificial Intelligence (AI) has demonstrated significant potential in
  healthcare, particularly in disease diagnosis and treatment planning. Recent
  progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new
  possibilities for interactive diagnostic tools. However, these models often
  suffer from factual hallucination, which can lead to incorrect diagnoses.
  Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to
  address these issues. However, the amount of high-quality data and distribution
  shifts between training data and deployment data limit the application of
  fine-tuning methods. Although RAG is lightweight and effective, existing
  RAG-based approaches are not sufficiently general to different medical domains
  and can potentially cause misalignment issues, both between modalities and
  between the model and the ground truth. In this paper, we propose a versatile
  multimodal RAG system, MMed-RAG, designed to enhance the factuality of
  Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an
  adaptive retrieved contexts selection method, and a provable RAG-based
  preference fine-tuning strategy. These innovations make the RAG process
  sufficiently general and reliable, significantly improving alignment when
  introducing retrieved contexts. Experimental results across five medical
  datasets (involving radiology, ophthalmology, pathology) on medical VQA and
  report generation demonstrate that MMed-RAG can achieve an average improvement
  of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available
  in https://github.com/richard-peng-xia/MMed-RAG.
paper_ai_summary: A versatile multimodal RAG system, MMed-RAG, enhances the factuality of Medical Large Vision-Language Models across different medical domains by introducing domain-aware retrieval, adaptive context selection, and preference fine-tuning.
paper_ai_keywords: Medical Large Vision-Language Models (Med-LVLMs) retrieval-augmented generation (RAG) medical VQA report generation domain-aware retrieval adaptive retrieved contexts selection preference fine-tuning strategy