title: Are Vision Language Models Texture or Shape Biased and Can We Steer
    Them?
summary: Vision language models (VLMs) have drastically changed the computer vision
  model landscape in only a few years, opening an exciting array of new
  applications from zero-shot image classification, over to image captioning, and
  visual question answering. Unlike pure vision models, they offer an intuitive
  way to access visual content through language prompting. The wide applicability
  of such models encourages us to ask whether they also align with human vision -
  specifically, how far they adopt human-induced visual biases through multimodal
  fusion, or whether they simply inherit biases from pure vision models. One
  important visual bias is the texture vs. shape bias, or the dominance of local
  over global information. In this paper, we study this bias in a wide range of
  popular VLMs. Interestingly, we find that VLMs are often more shape-biased than
  their vision encoders, indicating that visual biases are modulated to some
  extent through text in multimodal models. If text does indeed influence visual
  biases, this suggests that we may be able to steer visual biases not just
  through visual input but also through language: a hypothesis that we confirm
  through extensive experiments. For instance, we are able to steer shape bias
  from as low as 49% to as high as 72% through prompting alone. For now, the
  strong human bias towards shape (96%) remains out of reach for all tested VLMs.
paper_authors._id: 674c73752f5974eb9a7ed124
paper_authors.name: Paul Gavrikov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:50:16.337Z
paper_authors._id: 6266c07e7a1f5a1562c4113b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6266c07e7a1f5a1562c4113b/GEt70X7-UmNIbECDA0eVs.jpeg
paper_authors.fullname: Paul Gavrikov
paper_authors.type: user
paper_authors.user: paulgavrikov
paper_authors._id: 674c73752f5974eb9a7ed125
paper_authors.name: Jovita Lukasik
paper_authors._id: 674c73752f5974eb9a7ed126
paper_authors.name: Steffen Jung
paper_authors._id: 674c73752f5974eb9a7ed127
paper_authors.name: Robert Geirhos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T17:22:38.274Z
paper_authors._id: 673bbe0d7dfcdedd52619ec2
paper_authors.avatarUrl: /avatars/531a44f05d0c738bbe3e028c76c2e948.svg
paper_authors.fullname: Robert Geirhos
paper_authors.type: user
paper_authors.user: rgeirhos
paper_authors._id: 674c73752f5974eb9a7ed128
paper_authors.name: Bianca Lamm
paper_authors._id: 674c73752f5974eb9a7ed129
paper_authors.name: Muhammad Jehanzeb Mirza
paper_authors._id: 674c73752f5974eb9a7ed12a
paper_authors.name: Margret Keuper
paper_authors._id: 674c73752f5974eb9a7ed12b
paper_authors.name: Janis Keuper
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T08:59:55.771Z
paper_authors._id: 6519013e3fe8cc63b4c9b8d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6519013e3fe8cc63b4c9b8d3/ShO-DM3nWhKTeg_qJ3p0P.jpeg
paper_authors.fullname: Janis Keuper
paper_authors.type: user
paper_authors.user: jkHSO
paper_title: Are Vision Language Models Texture or Shape Biased and Can We Steer
    Them?
paper_summary: Vision language models (VLMs) have drastically changed the computer vision
  model landscape in only a few years, opening an exciting array of new
  applications from zero-shot image classification, over to image captioning, and
  visual question answering. Unlike pure vision models, they offer an intuitive
  way to access visual content through language prompting. The wide applicability
  of such models encourages us to ask whether they also align with human vision -
  specifically, how far they adopt human-induced visual biases through multimodal
  fusion, or whether they simply inherit biases from pure vision models. One
  important visual bias is the texture vs. shape bias, or the dominance of local
  over global information. In this paper, we study this bias in a wide range of
  popular VLMs. Interestingly, we find that VLMs are often more shape-biased than
  their vision encoders, indicating that visual biases are modulated to some
  extent through text in multimodal models. If text does indeed influence visual
  biases, this suggests that we may be able to steer visual biases not just
  through visual input but also through language: a hypothesis that we confirm
  through extensive experiments. For instance, we are able to steer shape bias
  from as low as 49% to as high as 72% through prompting alone. For now, the
  strong human bias towards shape (96%) remains out of reach for all tested VLMs.
paper_ai_keywords: Vision language models (VLMs) Zero-shot image classification Image captioning Visual question answering Multimodal fusion Texture vs. shape bias Shape-biased CLiiped