MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language
  Models
Artificial Intelligence (AI) has demonstrated significant potential in
healthcare, particularly in disease diagnosis and treatment planning. Recent
progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new
possibilities for interactive diagnostic tools. However, these models often
suffer from factual hallucination, which can lead to incorrect diagnoses.
Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to
address these issues. However, the amount of high-quality data and distribution
shifts between training data and deployment data limit the application of
fine-tuning methods. Although RAG is lightweight and effective, existing
RAG-based approaches are not sufficiently general to different medical domains
and can potentially cause misalignment issues, both between modalities and
between the model and the ground truth. In this paper, we propose a versatile
multimodal RAG system, MMed-RAG, designed to enhance the factuality of
Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an
adaptive retrieved contexts selection method, and a provable RAG-based
preference fine-tuning strategy. These innovations make the RAG process
sufficiently general and reliable, significantly improving alignment when
introducing retrieved contexts. Experimental results across five medical
datasets (involving radiology, ophthalmology, pathology) on medical VQA and
report generation demonstrate that MMed-RAG can achieve an average improvement
of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available
in https://github.com/richard-peng-xia/MMed-RAG.
6711c6c03a39fce88e963f74
Peng Xia
claimed_verified
2024-10-18T08:12:31.084Z
643e9ee6f6bb3c31a26e7bc4
/avatars/acfaa7d6a23dada24c86b954c3be116a.svg
Peng Xia
user
richardxp888
6711c6c03a39fce88e963f75
Kangyu Zhu
admin_assigned
2024-10-18T12:39:50.268Z
661af01a8ad066992403cc6f
/avatars/b4680010940a84c23521aa4b043ccab4.svg
Kangyu Zhu
user
zky11235
6711c6c03a39fce88e963f76
Haoran Li
claimed_verified
2024-10-21T07:48:24.989Z
64e86d6e7ef2c72cbcdb27b7
https://cdn-avatars.huggingface.co/v1/production/uploads/64e86d6e7ef2c72cbcdb27b7/RLuJJ1xuO665LMWneVDZz.jpeg
Haoran Li
user
brian13
6711c6c03a39fce88e963f77
Tianze Wang
6711c6c03a39fce88e963f78
Weijia Shi
6711c6c03a39fce88e963f79
Sheng Wang
6711c6c03a39fce88e963f7a
Linjun Zhang
6711c6c03a39fce88e963f7b
James Zou
6711c6c03a39fce88e963f7c
Huaxiu Yao
admin_assigned
2024-10-18T12:41:17.646Z
65f667a455009c4ad9e6ac4c
/avatars/ed10c2cf2ba3fde6d7da93f076961607.svg
Yao
user
Huaxiu
MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language
  Models
Artificial Intelligence (AI) has demonstrated significant potential in
healthcare, particularly in disease diagnosis and treatment planning. Recent
progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new
possibilities for interactive diagnostic tools. However, these models often
suffer from factual hallucination, which can lead to incorrect diagnoses.
Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to
address these issues. However, the amount of high-quality data and distribution
shifts between training data and deployment data limit the application of
fine-tuning methods. Although RAG is lightweight and effective, existing
RAG-based approaches are not sufficiently general to different medical domains
and can potentially cause misalignment issues, both between modalities and
between the model and the ground truth. In this paper, we propose a versatile
multimodal RAG system, MMed-RAG, designed to enhance the factuality of
Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an
adaptive retrieved contexts selection method, and a provable RAG-based
preference fine-tuning strategy. These innovations make the RAG process
sufficiently general and reliable, significantly improving alignment when
introducing retrieved contexts. Experimental results across five medical
datasets (involving radiology, ophthalmology, pathology) on medical VQA and
report generation demonstrate that MMed-RAG can achieve an average improvement
of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available
in https://github.com/richard-peng-xia/MMed-RAG.
A versatile multimodal RAG system, MMed-RAG, enhances the factuality of Medical Large Vision-Language Models across different medical domains by introducing domain-aware retrieval, adaptive context selection, and preference fine-tuning.
Medical Large Vision-Language Models (Med-LVLMs) retrieval-augmented generation (RAG) medical VQA report generation domain-aware retrieval adaptive retrieved contexts selection preference fine-tuning strategy