Ask in Any Modality: A Comprehensive Survey on Multimodal
  Retrieval-Augmented Generation
Large Language Models (LLMs) struggle with hallucinations and outdated
knowledge due to their reliance on static training data. Retrieval-Augmented
Generation (RAG) mitigates these issues by integrating external dynamic
information enhancing factual and updated grounding. Recent advances in
multimodal learning have led to the development of Multimodal RAG,
incorporating multiple modalities such as text, images, audio, and video to
enhance the generated outputs. However, cross-modal alignment and reasoning
introduce unique challenges to Multimodal RAG, distinguishing it from
traditional unimodal RAG. This survey offers a structured and comprehensive
analysis of Multimodal RAG systems, covering datasets, metrics, benchmarks,
evaluation, methodologies, and innovations in retrieval, fusion, augmentation,
and generation. We precisely review training strategies, robustness
enhancements, and loss functions, while also exploring the diverse Multimodal
RAG scenarios. Furthermore, we discuss open challenges and future research
directions to support advancements in this evolving field. This survey lays the
foundation for developing more capable and reliable AI systems that effectively
leverage multimodal dynamic external knowledge bases. Resources are available
at https://github.com/llm-lab-org/Multimodal-RAG-Survey.
67b303f18bd6e9a5cad8bc4d
Mohammad Mahdi Abootorabi
extracted_confirmed
2025-02-17T09:40:27.588Z
64ba58d377dd483716aba098
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba58d377dd483716aba098/6VASAUkFpDC-PR01yUJWj.png
Mahdi Abootorabi
user
aboots
67b303f18bd6e9a5cad8bc4e
Amirhosein Zobeiri
claimed_verified
2025-02-20T10:49:51.840Z
659d22a87ca51193252ec403
https://cdn-avatars.huggingface.co/v1/production/uploads/659d22a87ca51193252ec403/4r6RO3enW8q6gv_eWFZhl.jpeg
Amirhosein Zobeiri
user
ZobeiriA
67b303f18bd6e9a5cad8bc4f
Mahdi Dehghani
claimed_verified
2025-02-20T09:37:45.858Z
67b61b5692649eb787279693
/avatars/1d3b1f4d48ec8064a25ae5f27ea58576.svg
Mahdi Dehghani
user
Mahdi-dh
67b303f18bd6e9a5cad8bc50
Mohammadali Mohammadkhani
claimed_verified
2025-02-19T09:04:55.291Z
64a43776f489f7ced54a4c4b
https://cdn-avatars.huggingface.co/v1/production/uploads/64a43776f489f7ced54a4c4b/zO36YPuyaSoTbug99PxYs.jpeg
Mohammadali Mohammadkhani
user
moali-mkh-2000
67b303f18bd6e9a5cad8bc51
Bardia Mohammadi
claimed_verified
2025-02-19T14:37:52.589Z
6531334db5672703b2efcf6f
/avatars/598d6b4922d5f7d434ed1cf9ffc94f6d.svg
Bardia Mohammadi
user
bardia79mhd
67b303f18bd6e9a5cad8bc52
Omid Ghahroodi
claimed_verified
2025-02-19T09:33:28.667Z
6250819a5bf543dbd2607081
/avatars/dfade22a578b8349ab544acda5e8bcad.svg
Omid Ghahroodi
user
omidgh
67b303f18bd6e9a5cad8bc53
Mahdieh Soleymani Baghshah
extracted_pending
2025-02-17T09:40:02.049Z
661a88f0bcd78151e521bc60
/avatars/bedab01ce7909ecde7a60f891770c18c.svg
Mahdieh Soleymani Baghshah
user
Soleymani
67b303f18bd6e9a5cad8bc54
Ehsaneddin Asgari
Ask in Any Modality: A Comprehensive Survey on Multimodal
  Retrieval-Augmented Generation
Large Language Models (LLMs) struggle with hallucinations and outdated
knowledge due to their reliance on static training data. Retrieval-Augmented
Generation (RAG) mitigates these issues by integrating external dynamic
information enhancing factual and updated grounding. Recent advances in
multimodal learning have led to the development of Multimodal RAG,
incorporating multiple modalities such as text, images, audio, and video to
enhance the generated outputs. However, cross-modal alignment and reasoning
introduce unique challenges to Multimodal RAG, distinguishing it from
traditional unimodal RAG. This survey offers a structured and comprehensive
analysis of Multimodal RAG systems, covering datasets, metrics, benchmarks,
evaluation, methodologies, and innovations in retrieval, fusion, augmentation,
and generation. We precisely review training strategies, robustness
enhancements, and loss functions, while also exploring the diverse Multimodal
RAG scenarios. Furthermore, we discuss open challenges and future research
directions to support advancements in this evolving field. This survey lays the
foundation for developing more capable and reliable AI systems that effectively
leverage multimodal dynamic external knowledge bases. Resources are available
at https://github.com/llm-lab-org/Multimodal-RAG-Survey.
Survey of Multimodal Retrieval-Augmented Generation systems, discussing advancements, challenges, and future directions in integrating multiple modalities for enhanced AI capabilities.
Large Language Models hallucinations outdated knowledge Retrieval-Augmented Generation Multimodal RAG cross-modal alignment reasoning datasets metrics benchmarks evaluation methodologies retrieval fusion augmentation generation training strategies robustness loss functions open challenges future research directions