Are Vision Language Models Texture or Shape Biased and Can We Steer
  Them?
Vision language models (VLMs) have drastically changed the computer vision
model landscape in only a few years, opening an exciting array of new
applications from zero-shot image classification, over to image captioning, and
visual question answering. Unlike pure vision models, they offer an intuitive
way to access visual content through language prompting. The wide applicability
of such models encourages us to ask whether they also align with human vision -
specifically, how far they adopt human-induced visual biases through multimodal
fusion, or whether they simply inherit biases from pure vision models. One
important visual bias is the texture vs. shape bias, or the dominance of local
over global information. In this paper, we study this bias in a wide range of
popular VLMs. Interestingly, we find that VLMs are often more shape-biased than
their vision encoders, indicating that visual biases are modulated to some
extent through text in multimodal models. If text does indeed influence visual
biases, this suggests that we may be able to steer visual biases not just
through visual input but also through language: a hypothesis that we confirm
through extensive experiments. For instance, we are able to steer shape bias
from as low as 49% to as high as 72% through prompting alone. For now, the
strong human bias towards shape (96%) remains out of reach for all tested VLMs.
674c73752f5974eb9a7ed124
Paul Gavrikov
claimed_verified
2024-12-02T08:50:16.337Z
6266c07e7a1f5a1562c4113b
https://cdn-avatars.huggingface.co/v1/production/uploads/6266c07e7a1f5a1562c4113b/GEt70X7-UmNIbECDA0eVs.jpeg
Paul Gavrikov
user
paulgavrikov
674c73752f5974eb9a7ed125
Jovita Lukasik
674c73752f5974eb9a7ed126
Steffen Jung
674c73752f5974eb9a7ed127
Robert Geirhos
claimed_verified
2025-01-17T17:22:38.274Z
673bbe0d7dfcdedd52619ec2
/avatars/531a44f05d0c738bbe3e028c76c2e948.svg
Robert Geirhos
user
rgeirhos
674c73752f5974eb9a7ed128
Bianca Lamm
674c73752f5974eb9a7ed129
Muhammad Jehanzeb Mirza
674c73752f5974eb9a7ed12a
Margret Keuper
674c73752f5974eb9a7ed12b
Janis Keuper
claimed_verified
2025-03-14T08:59:55.771Z
6519013e3fe8cc63b4c9b8d3
https://cdn-avatars.huggingface.co/v1/production/uploads/6519013e3fe8cc63b4c9b8d3/ShO-DM3nWhKTeg_qJ3p0P.jpeg
Janis Keuper
user
jkHSO
Are Vision Language Models Texture or Shape Biased and Can We Steer
  Them?
Vision language models (VLMs) have drastically changed the computer vision
model landscape in only a few years, opening an exciting array of new
applications from zero-shot image classification, over to image captioning, and
visual question answering. Unlike pure vision models, they offer an intuitive
way to access visual content through language prompting. The wide applicability
of such models encourages us to ask whether they also align with human vision -
specifically, how far they adopt human-induced visual biases through multimodal
fusion, or whether they simply inherit biases from pure vision models. One
important visual bias is the texture vs. shape bias, or the dominance of local
over global information. In this paper, we study this bias in a wide range of
popular VLMs. Interestingly, we find that VLMs are often more shape-biased than
their vision encoders, indicating that visual biases are modulated to some
extent through text in multimodal models. If text does indeed influence visual
biases, this suggests that we may be able to steer visual biases not just
through visual input but also through language: a hypothesis that we confirm
through extensive experiments. For instance, we are able to steer shape bias
from as low as 49% to as high as 72% through prompting alone. For now, the
strong human bias towards shape (96%) remains out of reach for all tested VLMs.
Vision-language models exhibit biases that can be influenced by language prompts, often showing more shape bias compared to pure vision models.
vision language models VLMs zero-shot image classification image captioning visual question answering multimodal fusion texture vs. shape bias shape bias visual encoders