title: Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
summary: Vision-Language Models (VLMs) excel in integrating visual and textual
  information for vision-centric tasks, but their handling of inconsistencies
  between modalities is underexplored. We investigate VLMs' modality preferences
  when faced with visual data and varied textual inputs in vision-centered
  settings. By introducing textual variations to four vision-centric tasks and
  evaluating ten Vision-Language Models (VLMs), we discover a ``blind faith
  in text'' phenomenon: VLMs disproportionately trust textual data over visual
  data when inconsistencies arise, leading to significant performance drops under
  corrupted text and raising safety concerns. We analyze factors influencing this
  text bias, including instruction prompts, language model size, text relevance,
  token order, and the interplay between visual and textual certainty. While
  certain factors, such as scaling up the language model size, slightly mitigate
  text bias, others like token order can exacerbate it due to positional biases
  inherited from language models. To address this issue, we explore supervised
  fine-tuning with text augmentation and demonstrate its effectiveness in
  reducing text bias. Additionally, we provide a theoretical analysis suggesting
  that the blind faith in text phenomenon may stem from an imbalance of pure text
  and multi-modal data during training. Our findings highlight the need for
  balanced training and careful consideration of modality interactions in VLMs to
  enhance their robustness and reliability in handling multi-modal data
  inconsistencies.
paper_authors._id: 67c90dad6f3ef3c2c77689b0
paper_authors.name: Ailin Deng
paper_authors._id: 67c90dad6f3ef3c2c77689b1
paper_authors.name: Tri Cao
paper_authors._id: 67c90dad6f3ef3c2c77689b2
paper_authors.name: Zhirui Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:03:34.042Z
paper_authors._id: 67cbb6ea2cc05acaab023f75
paper_authors.avatarUrl: /avatars/79272c8889a8c472cf75172ead72daea.svg
paper_authors.fullname: Zhirui Chen
paper_authors.type: user
paper_authors.user: ryanchen42
paper_authors._id: 67c90dad6f3ef3c2c77689b3
paper_authors.name: Bryan Hooi
paper_title: Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
paper_summary: Vision-Language Models (VLMs) excel in integrating visual and textual
  information for vision-centric tasks, but their handling of inconsistencies
  between modalities is underexplored. We investigate VLMs' modality preferences
  when faced with visual data and varied textual inputs in vision-centered
  settings. By introducing textual variations to four vision-centric tasks and
  evaluating ten Vision-Language Models (VLMs), we discover a ``blind faith
  in text'' phenomenon: VLMs disproportionately trust textual data over visual
  data when inconsistencies arise, leading to significant performance drops under
  corrupted text and raising safety concerns. We analyze factors influencing this
  text bias, including instruction prompts, language model size, text relevance,
  token order, and the interplay between visual and textual certainty. While
  certain factors, such as scaling up the language model size, slightly mitigate
  text bias, others like token order can exacerbate it due to positional biases
  inherited from language models. To address this issue, we explore supervised
  fine-tuning with text augmentation and demonstrate its effectiveness in
  reducing text bias. Additionally, we provide a theoretical analysis suggesting
  that the blind faith in text phenomenon may stem from an imbalance of pure text
  and multi-modal data during training. Our findings highlight the need for
  balanced training and careful consideration of modality interactions in VLMs to
  enhance their robustness and reliability in handling multi-modal data
  inconsistencies.
paper_ai_keywords: Vision-Language Models (VLMs) blind faith in text modality preferences textual variations vision-centric tasks text bias instruction prompts language model size token order positional biases multi-modal data supervised fine-tuning text augmentation balanced training modality interactions