title: Evaluating and Steering Modality Preferences in Multimodal Large
    Language Model
summary: Multimodal large language models (MLLMs) have achieved remarkable performance
  on complex tasks with multimodal context. However, it is still understudied
  whether they exhibit modality preference when processing multimodal contexts.
  To study this question, we first build a MC\textsuperscript{2}
  benchmark under controlled evidence conflict scenarios to systematically
  evaluate modality preference, which is the tendency to favor one modality over
  another when making decisions based on multimodal conflicting evidence. Our
  extensive evaluation reveals that all 18 tested MLLMs generally demonstrate
  clear modality bias, and modality preference can be influenced by external
  interventions. An in-depth analysis reveals that the preference direction can
  be captured within the latent representations of MLLMs. Built on this, we
  propose a probing and steering method based on representation engineering to
  explicitly control modality preference without additional fine-tuning or
  carefully crafted prompts. Our method effectively amplifies modality preference
  toward a desired direction and applies to downstream tasks such as
  hallucination mitigation and multimodal machine translation, yielding promising
  improvements.
paper_authors._id: 683da8dc28c2a6768816306f
paper_authors.name: Yu Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-03T08:54:33.290Z
paper_authors._id: 65fe9599d74aed6c3e6d93a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65fe9599d74aed6c3e6d93a2/WxPDKPJ6-p0_xu0dzOYXw.jpeg
paper_authors.fullname: Yu Zhang
paper_authors.type: user
paper_authors.user: 271754echo
paper_authors._id: 683da8dc28c2a67688163070
paper_authors.name: Jinlong Ma
paper_authors._id: 683da8dc28c2a67688163071
paper_authors.name: Yongshuai Hou
paper_authors._id: 683da8dc28c2a67688163072
paper_authors.name: Xuefeng Bai
paper_authors._id: 683da8dc28c2a67688163073
paper_authors.name: Kehai Chen
paper_authors._id: 683da8dc28c2a67688163074
paper_authors.name: Yang Xiang
paper_authors._id: 683da8dc28c2a67688163075
paper_authors.name: Jun Yu
paper_authors._id: 683da8dc28c2a67688163076
paper_authors.name: Min Zhang
paper_title: Evaluating and Steering Modality Preferences in Multimodal Large
    Language Model
paper_summary: Multimodal large language models (MLLMs) have achieved remarkable performance
  on complex tasks with multimodal context. However, it is still understudied
  whether they exhibit modality preference when processing multimodal contexts.
  To study this question, we first build a MC\textsuperscript{2}
  benchmark under controlled evidence conflict scenarios to systematically
  evaluate modality preference, which is the tendency to favor one modality over
  another when making decisions based on multimodal conflicting evidence. Our
  extensive evaluation reveals that all 18 tested MLLMs generally demonstrate
  clear modality bias, and modality preference can be influenced by external
  interventions. An in-depth analysis reveals that the preference direction can
  be captured within the latent representations of MLLMs. Built on this, we
  propose a probing and steering method based on representation engineering to
  explicitly control modality preference without additional fine-tuning or
  carefully crafted prompts. Our method effectively amplifies modality preference
  toward a desired direction and applies to downstream tasks such as
  hallucination mitigation and multimodal machine translation, yielding promising
  improvements.
paper_ai_summary: MLLMs exhibit modality bias in multimodal processing, which can be controlled using a representation engineering method to improve tasks like hallucination mitigation and multimodal machine translation.
paper_ai_keywords: Multimodal large language models MLLMs MC\textsuperscript{2} benchmark modality preference modality bias latent representations representation engineering probing steering method hallucination mitigation multimodal machine translation