MLLMs are Deeply Affected by Modality Bias
Recent advances in Multimodal Large Language Models (MLLMs) have shown
promising results in integrating diverse modalities such as texts and images.
MLLMs are heavily influenced by modality bias, often relying on language while
under-utilizing other modalities like visual inputs. This position paper argues
that MLLMs are deeply affected by modality bias. Firstly, we diagnose the
current state of modality bias, highlighting its manifestations across various
tasks. Secondly, we propose a systematic research road-map related to modality
bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and
offer actionable suggestions for future research to mitigate it. To
substantiate these findings, we conduct experiments that demonstrate the
influence of each factor: 1. Data Characteristics: Language data is compact and
abstract, while visual data is redundant and complex, creating an inherent
imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The
dominance of pretrained language models in MLLMs leads to overreliance on
language and neglect of visual information. 3. Training Objectives: Current
objectives often fail to promote balanced cross-modal alignment, resulting in
shortcut learning biased toward language. These findings highlight the need for
balanced training strategies and model architectures to better integrate
multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle
these challenges and drive innovation in MLLM research. Our work provides a
fresh perspective on modality bias in MLLMs and offers insights for developing
more robust and generalizable multimodal systems-advancing progress toward
Artificial General Intelligence.
6836db002cbd03bbddcc8696
Xu Zheng
claimed_verified
2025-06-12T12:43:26.034Z
65961a7daa677adc97d8e33f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/FkwP-qqxjAhPeaeQ2Y5sv.jpeg
Xu Zheng
user
xz287
6836db002cbd03bbddcc8697
Chenfei Liao
claimed_verified
2025-05-28T10:11:13.295Z
6806464ed918f6d2fee2bc8b
https://cdn-avatars.huggingface.co/v1/production/uploads/6806464ed918f6d2fee2bc8b/rgpG2oO0m6PT0KltCF_Wf.jpeg
Chenfei Liao
user
Chenfei-Liao
6836db002cbd03bbddcc8698
Yuqian Fu
6836db002cbd03bbddcc8699
Kaiyu Lei
6836db002cbd03bbddcc869a
Yuanhuiyi Lyu
6836db002cbd03bbddcc869b
Lutao Jiang
6836db002cbd03bbddcc869c
Bin Ren
6836db002cbd03bbddcc869d
Jialei Chen
6836db002cbd03bbddcc869e
Jiawen Wang
6836db002cbd03bbddcc869f
Chengxin Li
6836db002cbd03bbddcc86a0
Linfeng Zhang
6836db002cbd03bbddcc86a1
Danda Pani Paudel
6836db002cbd03bbddcc86a2
Xuanjing Huang
6836db002cbd03bbddcc86a3
Yu-Gang Jiang
6836db002cbd03bbddcc86a4
Nicu Sebe
6836db002cbd03bbddcc86a5
Dacheng Tao
6836db002cbd03bbddcc86a6
Luc Van Gool
6836db002cbd03bbddcc86a7
Xuming Hu
MLLMs are Deeply Affected by Modality Bias
Recent advances in Multimodal Large Language Models (MLLMs) have shown
promising results in integrating diverse modalities such as texts and images.
MLLMs are heavily influenced by modality bias, often relying on language while
under-utilizing other modalities like visual inputs. This position paper argues
that MLLMs are deeply affected by modality bias. Firstly, we diagnose the
current state of modality bias, highlighting its manifestations across various
tasks. Secondly, we propose a systematic research road-map related to modality
bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and
offer actionable suggestions for future research to mitigate it. To
substantiate these findings, we conduct experiments that demonstrate the
influence of each factor: 1. Data Characteristics: Language data is compact and
abstract, while visual data is redundant and complex, creating an inherent
imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The
dominance of pretrained language models in MLLMs leads to overreliance on
language and neglect of visual information. 3. Training Objectives: Current
objectives often fail to promote balanced cross-modal alignment, resulting in
shortcut learning biased toward language. These findings highlight the need for
balanced training strategies and model architectures to better integrate
multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle
these challenges and drive innovation in MLLM research. Our work provides a
fresh perspective on modality bias in MLLMs and offers insights for developing
more robust and generalizable multimodal systems-advancing progress toward
Artificial General Intelligence.
MLLMs exhibit modality bias, favoring language over other modalities like visual inputs, which impedes balanced multimodal integration and necessitates research into balanced strategies and architectures.
Multimodal Large Language Models MLLMs modality bias language data visual data pretrained language models cross-modal alignment shortcut learning balanced training strategies