title: Beyond Recognition: Evaluating Visual Perspective Taking in Vision
    Language Models
summary: We investigate the ability of Vision Language Models (VLMs) to perform visual
  perspective taking using a novel set of visual tasks inspired by established
  human tests. Our approach leverages carefully controlled scenes, in which a
  single humanoid minifigure is paired with a single object. By systematically
  varying spatial configurations - such as object position relative to the
  humanoid minifigure and the humanoid minifigure's orientation - and using both
  bird's-eye and surface-level views, we created 144 unique visual tasks. Each
  visual task is paired with a series of 7 diagnostic questions designed to
  assess three levels of visual cognition: scene understanding, spatial
  reasoning, and visual perspective taking. Our evaluation of several
  state-of-the-art models, including GPT-4-Turbo, GPT-4o,
  Llama-3.2-11B-Vision-Instruct, and variants of Claude Sonnet, reveals that
  while they excel in scene understanding, the performance declines significantly
  on spatial reasoning and further deteriorates on perspective-taking. Our
  analysis suggests a gap between surface-level object recognition and the deeper
  spatial and perspective reasoning required for complex visual tasks, pointing
  to the need for integrating explicit geometric representations and tailored
  training protocols in future VLM development.
paper_authors._id: 681c7a3829ba66a745217db5
paper_authors.name: Gracjan Góral
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T09:58:02.558Z
paper_authors._id: 63caf7ce9f78909f9f81eb72
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63caf7ce9f78909f9f81eb72/2mLUr_DdEom28DjodWimv.jpeg
paper_authors.fullname: Gracjan Goral
paper_authors.type: user
paper_authors.user: Gracjan
paper_authors._id: 681c7a3829ba66a745217db6
paper_authors.name: Alicja Ziarko
paper_authors._id: 681c7a3829ba66a745217db7
paper_authors.name: Piotr Miłoś
paper_authors._id: 681c7a3829ba66a745217db8
paper_authors.name: Michał Nauman
paper_authors._id: 681c7a3829ba66a745217db9
paper_authors.name: Maciej Wołczyk
paper_authors._id: 681c7a3829ba66a745217dba
paper_authors.name: Michał Kosiński
paper_title: Beyond Recognition: Evaluating Visual Perspective Taking in Vision
    Language Models
paper_summary: We investigate the ability of Vision Language Models (VLMs) to perform visual
  perspective taking using a novel set of visual tasks inspired by established
  human tests. Our approach leverages carefully controlled scenes, in which a
  single humanoid minifigure is paired with a single object. By systematically
  varying spatial configurations - such as object position relative to the
  humanoid minifigure and the humanoid minifigure's orientation - and using both
  bird's-eye and surface-level views, we created 144 unique visual tasks. Each
  visual task is paired with a series of 7 diagnostic questions designed to
  assess three levels of visual cognition: scene understanding, spatial
  reasoning, and visual perspective taking. Our evaluation of several
  state-of-the-art models, including GPT-4-Turbo, GPT-4o,
  Llama-3.2-11B-Vision-Instruct, and variants of Claude Sonnet, reveals that
  while they excel in scene understanding, the performance declines significantly
  on spatial reasoning and further deteriorates on perspective-taking. Our
  analysis suggests a gap between surface-level object recognition and the deeper
  spatial and perspective reasoning required for complex visual tasks, pointing
  to the need for integrating explicit geometric representations and tailored
  training protocols in future VLM development.
paper_ai_summary: State-of-the-art Vision Language Models excel in scene understanding but struggle with spatial reasoning and visual perspective taking in controlled visual tasks.
paper_ai_keywords: Vision Language Models visual perspective taking spatial configurations scene understanding spatial reasoning bird's-eye view surface-level view GPT-4-Turbo GPT-4o Llama-3.2-11B-Vision-Instruct Claude Sonnet geometric representations tailored training protocols