Beyond Recognition: Evaluating Visual Perspective Taking in Vision
  Language Models
We investigate the ability of Vision Language Models (VLMs) to perform visual
perspective taking using a novel set of visual tasks inspired by established
human tests. Our approach leverages carefully controlled scenes, in which a
single humanoid minifigure is paired with a single object. By systematically
varying spatial configurations - such as object position relative to the
humanoid minifigure and the humanoid minifigure's orientation - and using both
bird's-eye and surface-level views, we created 144 unique visual tasks. Each
visual task is paired with a series of 7 diagnostic questions designed to
assess three levels of visual cognition: scene understanding, spatial
reasoning, and visual perspective taking. Our evaluation of several
state-of-the-art models, including GPT-4-Turbo, GPT-4o,
Llama-3.2-11B-Vision-Instruct, and variants of Claude Sonnet, reveals that
while they excel in scene understanding, the performance declines significantly
on spatial reasoning and further deteriorates on perspective-taking. Our
analysis suggests a gap between surface-level object recognition and the deeper
spatial and perspective reasoning required for complex visual tasks, pointing
to the need for integrating explicit geometric representations and tailored
training protocols in future VLM development.
681c7a3829ba66a745217db5
Gracjan Góral
claimed_verified
2025-05-08T09:58:02.558Z
63caf7ce9f78909f9f81eb72
https://cdn-avatars.huggingface.co/v1/production/uploads/63caf7ce9f78909f9f81eb72/2mLUr_DdEom28DjodWimv.jpeg
Gracjan Goral
user
Gracjan
681c7a3829ba66a745217db6
Alicja Ziarko
681c7a3829ba66a745217db7
Piotr Miłoś
681c7a3829ba66a745217db8
Michał Nauman
681c7a3829ba66a745217db9
Maciej Wołczyk
681c7a3829ba66a745217dba
Michał Kosiński
Beyond Recognition: Evaluating Visual Perspective Taking in Vision
  Language Models
We investigate the ability of Vision Language Models (VLMs) to perform visual
perspective taking using a novel set of visual tasks inspired by established
human tests. Our approach leverages carefully controlled scenes, in which a
single humanoid minifigure is paired with a single object. By systematically
varying spatial configurations - such as object position relative to the
humanoid minifigure and the humanoid minifigure's orientation - and using both
bird's-eye and surface-level views, we created 144 unique visual tasks. Each
visual task is paired with a series of 7 diagnostic questions designed to
assess three levels of visual cognition: scene understanding, spatial
reasoning, and visual perspective taking. Our evaluation of several
state-of-the-art models, including GPT-4-Turbo, GPT-4o,
Llama-3.2-11B-Vision-Instruct, and variants of Claude Sonnet, reveals that
while they excel in scene understanding, the performance declines significantly
on spatial reasoning and further deteriorates on perspective-taking. Our
analysis suggests a gap between surface-level object recognition and the deeper
spatial and perspective reasoning required for complex visual tasks, pointing
to the need for integrating explicit geometric representations and tailored
training protocols in future VLM development.
State-of-the-art Vision Language Models excel in scene understanding but struggle with spatial reasoning and visual perspective taking in controlled visual tasks.
Vision Language Models visual perspective taking spatial configurations scene understanding spatial reasoning bird's-eye view surface-level view GPT-4-Turbo GPT-4o Llama-3.2-11B-Vision-Instruct Claude Sonnet geometric representations tailored training protocols