Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
Vision-Language Models (VLMs) excel in integrating visual and textual
information for vision-centric tasks, but their handling of inconsistencies
between modalities is underexplored. We investigate VLMs' modality preferences
when faced with visual data and varied textual inputs in vision-centered
settings. By introducing textual variations to four vision-centric tasks and
evaluating ten Vision-Language Models (VLMs), we discover a ``blind faith
in text'' phenomenon: VLMs disproportionately trust textual data over visual
data when inconsistencies arise, leading to significant performance drops under
corrupted text and raising safety concerns. We analyze factors influencing this
text bias, including instruction prompts, language model size, text relevance,
token order, and the interplay between visual and textual certainty. While
certain factors, such as scaling up the language model size, slightly mitigate
text bias, others like token order can exacerbate it due to positional biases
inherited from language models. To address this issue, we explore supervised
fine-tuning with text augmentation and demonstrate its effectiveness in
reducing text bias. Additionally, we provide a theoretical analysis suggesting
that the blind faith in text phenomenon may stem from an imbalance of pure text
and multi-modal data during training. Our findings highlight the need for
balanced training and careful consideration of modality interactions in VLMs to
enhance their robustness and reliability in handling multi-modal data
inconsistencies.
67c90dad6f3ef3c2c77689b0
Ailin Deng
67c90dad6f3ef3c2c77689b1
Tri Cao
extracted_pending
2025-05-13T11:49:04.927Z
68233194672810620978209b
/avatars/feeac9d17567c22cc74d32d553a0357a.svg
Tri Cao
user
tricao1105
67c90dad6f3ef3c2c77689b2
Zhirui Chen
claimed_verified
2025-03-10T08:03:34.042Z
67cbb6ea2cc05acaab023f75
/avatars/79272c8889a8c472cf75172ead72daea.svg
Zhirui Chen
user
ryanchen42
67c90dad6f3ef3c2c77689b3
Bryan Hooi
Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
Vision-Language Models (VLMs) excel in integrating visual and textual
information for vision-centric tasks, but their handling of inconsistencies
between modalities is underexplored. We investigate VLMs' modality preferences
when faced with visual data and varied textual inputs in vision-centered
settings. By introducing textual variations to four vision-centric tasks and
evaluating ten Vision-Language Models (VLMs), we discover a ``blind faith
in text'' phenomenon: VLMs disproportionately trust textual data over visual
data when inconsistencies arise, leading to significant performance drops under
corrupted text and raising safety concerns. We analyze factors influencing this
text bias, including instruction prompts, language model size, text relevance,
token order, and the interplay between visual and textual certainty. While
certain factors, such as scaling up the language model size, slightly mitigate
text bias, others like token order can exacerbate it due to positional biases
inherited from language models. To address this issue, we explore supervised
fine-tuning with text augmentation and demonstrate its effectiveness in
reducing text bias. Additionally, we provide a theoretical analysis suggesting
that the blind faith in text phenomenon may stem from an imbalance of pure text
and multi-modal data during training. Our findings highlight the need for
balanced training and careful consideration of modality interactions in VLMs to
enhance their robustness and reliability in handling multi-modal data
inconsistencies.
Vision-Language Models disproportionately trust textual data over visual data in vision-centric tasks when inconsistencies arise, leading to performance drops and safety concerns, which can be partially addressed through supervised fine-tuning with text augmentation.
vision-language models VLMs modality preferences vision-centric tasks text bias instruction prompts token order cross-modal certainty blind faith in text supervised fine-tuning text augmentation