title: MLLMs are Deeply Affected by Modality Bias
summary: Recent advances in Multimodal Large Language Models (MLLMs) have shown
  promising results in integrating diverse modalities such as texts and images.
  MLLMs are heavily influenced by modality bias, often relying on language while
  under-utilizing other modalities like visual inputs. This position paper argues
  that MLLMs are deeply affected by modality bias. Firstly, we diagnose the
  current state of modality bias, highlighting its manifestations across various
  tasks. Secondly, we propose a systematic research road-map related to modality
  bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and
  offer actionable suggestions for future research to mitigate it. To
  substantiate these findings, we conduct experiments that demonstrate the
  influence of each factor: 1. Data Characteristics: Language data is compact and
  abstract, while visual data is redundant and complex, creating an inherent
  imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The
  dominance of pretrained language models in MLLMs leads to overreliance on
  language and neglect of visual information. 3. Training Objectives: Current
  objectives often fail to promote balanced cross-modal alignment, resulting in
  shortcut learning biased toward language. These findings highlight the need for
  balanced training strategies and model architectures to better integrate
  multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle
  these challenges and drive innovation in MLLM research. Our work provides a
  fresh perspective on modality bias in MLLMs and offers insights for developing
  more robust and generalizable multimodal systems-advancing progress toward
  Artificial General Intelligence.
paper_authors._id: 6836db002cbd03bbddcc8696
paper_authors.name: Xu Zheng
paper_authors._id: 6836db002cbd03bbddcc8697
paper_authors.name: Chenfei Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T10:11:13.295Z
paper_authors._id: 6806464ed918f6d2fee2bc8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6806464ed918f6d2fee2bc8b/rgpG2oO0m6PT0KltCF_Wf.jpeg
paper_authors.fullname: Chenfei Liao
paper_authors.type: user
paper_authors.user: Chenfei-Liao
paper_authors._id: 6836db002cbd03bbddcc8698
paper_authors.name: Yuqian Fu
paper_authors._id: 6836db002cbd03bbddcc8699
paper_authors.name: Kaiyu Lei
paper_authors._id: 6836db002cbd03bbddcc869a
paper_authors.name: Yuanhuiyi Lyu
paper_authors._id: 6836db002cbd03bbddcc869b
paper_authors.name: Lutao Jiang
paper_authors._id: 6836db002cbd03bbddcc869c
paper_authors.name: Bin Ren
paper_authors._id: 6836db002cbd03bbddcc869d
paper_authors.name: Jialei Chen
paper_authors._id: 6836db002cbd03bbddcc869e
paper_authors.name: Jiawen Wang
paper_authors._id: 6836db002cbd03bbddcc869f
paper_authors.name: Chengxin Li
paper_authors._id: 6836db002cbd03bbddcc86a0
paper_authors.name: Linfeng Zhang
paper_authors._id: 6836db002cbd03bbddcc86a1
paper_authors.name: Danda Pani Paudel
paper_authors._id: 6836db002cbd03bbddcc86a2
paper_authors.name: Xuanjing Huang
paper_authors._id: 6836db002cbd03bbddcc86a3
paper_authors.name: Yu-Gang Jiang
paper_authors._id: 6836db002cbd03bbddcc86a4
paper_authors.name: Nicu Sebe
paper_authors._id: 6836db002cbd03bbddcc86a5
paper_authors.name: Dacheng Tao
paper_authors._id: 6836db002cbd03bbddcc86a6
paper_authors.name: Luc Van Gool
paper_authors._id: 6836db002cbd03bbddcc86a7
paper_authors.name: Xuming Hu
paper_title: MLLMs are Deeply Affected by Modality Bias
paper_summary: Recent advances in Multimodal Large Language Models (MLLMs) have shown
  promising results in integrating diverse modalities such as texts and images.
  MLLMs are heavily influenced by modality bias, often relying on language while
  under-utilizing other modalities like visual inputs. This position paper argues
  that MLLMs are deeply affected by modality bias. Firstly, we diagnose the
  current state of modality bias, highlighting its manifestations across various
  tasks. Secondly, we propose a systematic research road-map related to modality
  bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and
  offer actionable suggestions for future research to mitigate it. To
  substantiate these findings, we conduct experiments that demonstrate the
  influence of each factor: 1. Data Characteristics: Language data is compact and
  abstract, while visual data is redundant and complex, creating an inherent
  imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The
  dominance of pretrained language models in MLLMs leads to overreliance on
  language and neglect of visual information. 3. Training Objectives: Current
  objectives often fail to promote balanced cross-modal alignment, resulting in
  shortcut learning biased toward language. These findings highlight the need for
  balanced training strategies and model architectures to better integrate
  multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle
  these challenges and drive innovation in MLLM research. Our work provides a
  fresh perspective on modality bias in MLLMs and offers insights for developing
  more robust and generalizable multimodal systems-advancing progress toward
  Artificial General Intelligence.
paper_ai_summary: MLLMs exhibit modality bias, favoring language over other modalities like visual inputs, which impedes balanced multimodal integration and necessitates research into balanced strategies and architectures.
paper_ai_keywords: Multimodal Large Language Models MLLMs modality bias language data visual data pretrained language models cross-modal alignment shortcut learning balanced training strategies