title: The Curse of Multi-Modalities: Evaluating Hallucinations of Large
    Multimodal Models across Language, Visual, and Audio
summary: Recent advancements in large multimodal models (LMMs) have significantly
  enhanced performance across diverse tasks, with ongoing efforts to further
  integrate additional modalities such as video and audio. However, most existing
  LMMs remain vulnerable to hallucinations, the discrepancy between the factual
  multimodal input and the generated textual output, which has limited their
  applicability in various real-world scenarios. This paper presents the first
  systematic investigation of hallucinations in LMMs involving the three most
  common modalities: language, visual, and audio. Our study reveals two key
  contributors to hallucinations: overreliance on unimodal priors and spurious
  inter-modality correlations. To address these challenges, we introduce the
  benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates
  hallucinations in LMMs, providing a detailed analysis of their underlying
  issues. Our findings highlight key vulnerabilities, including imbalances in
  modality integration and biases from training data, underscoring the need for
  balanced cross-modal learning and enhanced hallucination mitigation strategies.
  Based on our observations and findings, we suggest potential research
  directions that could enhance the reliability of LMMs.
paper_authors._id: 6710899cb95dec8a173c7140
paper_authors.name: Sicong Leng
paper_authors._id: 6710899cb95dec8a173c7141
paper_authors.name: Yun Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:02:00.241Z
paper_authors._id: 65386066bf790526ef358701
paper_authors.avatarUrl: /avatars/f02b48f1b05c353e5ab371de45b9cf4c.svg
paper_authors.fullname: Xing Yun
paper_authors.type: user
paper_authors.user: xing0047
paper_authors._id: 6710899cb95dec8a173c7142
paper_authors.name: Zesen Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:30.926Z
paper_authors._id: 65b2529285b6c21448a10d65
paper_authors.avatarUrl: /avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
paper_authors.fullname: Zesen Cheng
paper_authors.type: user
paper_authors.user: ClownRat
paper_authors._id: 6710899cb95dec8a173c7143
paper_authors.name: Yang Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:24:03.059Z
paper_authors._id: 670cc0b60c2a160841c8a343
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/21wJ_Eg4jBvTfaGR4Y7CY.png
paper_authors.fullname: ZHOU Yang
paper_authors.type: user
paper_authors.user: youngzhou12
paper_authors._id: 6710899cb95dec8a173c7144
paper_authors.name: Hang Zhang
paper_authors._id: 6710899cb95dec8a173c7145
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:27:27.753Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 6710899cb95dec8a173c7146
paper_authors.name: Deli Zhao
paper_authors._id: 6710899cb95dec8a173c7147
paper_authors.name: Shijian Lu
paper_authors._id: 6710899cb95dec8a173c7148
paper_authors.name: Chunyan Miao
paper_authors._id: 6710899cb95dec8a173c7149
paper_authors.name: Lidong Bing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:15.975Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: The Curse of Multi-Modalities: Evaluating Hallucinations of Large
    Multimodal Models across Language, Visual, and Audio
paper_summary: Recent advancements in large multimodal models (LMMs) have significantly
  enhanced performance across diverse tasks, with ongoing efforts to further
  integrate additional modalities such as video and audio. However, most existing
  LMMs remain vulnerable to hallucinations, the discrepancy between the factual
  multimodal input and the generated textual output, which has limited their
  applicability in various real-world scenarios. This paper presents the first
  systematic investigation of hallucinations in LMMs involving the three most
  common modalities: language, visual, and audio. Our study reveals two key
  contributors to hallucinations: overreliance on unimodal priors and spurious
  inter-modality correlations. To address these challenges, we introduce the
  benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates
  hallucinations in LMMs, providing a detailed analysis of their underlying
  issues. Our findings highlight key vulnerabilities, including imbalances in
  modality integration and biases from training data, underscoring the need for
  balanced cross-modal learning and enhanced hallucination mitigation strategies.
  Based on our observations and findings, we suggest potential research
  directions that could enhance the reliability of LMMs.
paper_ai_summary: Investigation into hallucinations in large multimodal models highlights vulnerabilities in modality integration and suggests strategies for improved reliability.
paper_ai_keywords: large multimodal models LMMs hallucinations unimodal priors spurious inter-modality correlations benchmark Curse of Multi-Modalities CMM modality integration cross-modal learning hallucination mitigation