Evaluating and Steering Modality Preferences in Multimodal Large
  Language Model
Multimodal large language models (MLLMs) have achieved remarkable performance
on complex tasks with multimodal context. However, it is still understudied
whether they exhibit modality preference when processing multimodal contexts.
To study this question, we first build a MC\textsuperscript{2}
benchmark under controlled evidence conflict scenarios to systematically
evaluate modality preference, which is the tendency to favor one modality over
another when making decisions based on multimodal conflicting evidence. Our
extensive evaluation reveals that all 18 tested MLLMs generally demonstrate
clear modality bias, and modality preference can be influenced by external
interventions. An in-depth analysis reveals that the preference direction can
be captured within the latent representations of MLLMs. Built on this, we
propose a probing and steering method based on representation engineering to
explicitly control modality preference without additional fine-tuning or
carefully crafted prompts. Our method effectively amplifies modality preference
toward a desired direction and applies to downstream tasks such as
hallucination mitigation and multimodal machine translation, yielding promising
improvements.
683da8dc28c2a6768816306f
Yu Zhang
extracted_pending
2025-06-03T08:54:33.290Z
65fe9599d74aed6c3e6d93a2
https://cdn-avatars.huggingface.co/v1/production/uploads/65fe9599d74aed6c3e6d93a2/WxPDKPJ6-p0_xu0dzOYXw.jpeg
Yu Zhang
user
271754echo
683da8dc28c2a67688163070
Jinlong Ma
683da8dc28c2a67688163071
Yongshuai Hou
683da8dc28c2a67688163072
Xuefeng Bai
683da8dc28c2a67688163073
Kehai Chen
683da8dc28c2a67688163074
Yang Xiang
683da8dc28c2a67688163075
Jun Yu
683da8dc28c2a67688163076
Min Zhang
Evaluating and Steering Modality Preferences in Multimodal Large
  Language Model
Multimodal large language models (MLLMs) have achieved remarkable performance
on complex tasks with multimodal context. However, it is still understudied
whether they exhibit modality preference when processing multimodal contexts.
To study this question, we first build a MC\textsuperscript{2}
benchmark under controlled evidence conflict scenarios to systematically
evaluate modality preference, which is the tendency to favor one modality over
another when making decisions based on multimodal conflicting evidence. Our
extensive evaluation reveals that all 18 tested MLLMs generally demonstrate
clear modality bias, and modality preference can be influenced by external
interventions. An in-depth analysis reveals that the preference direction can
be captured within the latent representations of MLLMs. Built on this, we
propose a probing and steering method based on representation engineering to
explicitly control modality preference without additional fine-tuning or
carefully crafted prompts. Our method effectively amplifies modality preference
toward a desired direction and applies to downstream tasks such as
hallucination mitigation and multimodal machine translation, yielding promising
improvements.
MLLMs exhibit modality bias in multimodal processing, which can be controlled using a representation engineering method to improve tasks like hallucination mitigation and multimodal machine translation.
Multimodal large language models MLLMs MC\textsuperscript{2} benchmark modality preference modality bias latent representations representation engineering probing steering method hallucination mitigation multimodal machine translation