title: Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery
    Simulation
summary: We present a framework for perspective-aware reasoning in vision-language
  models (VLMs) through mental imagery simulation. Perspective-taking, the
  ability to perceive an environment or situation from an alternative viewpoint,
  is a key benchmark for human-level visual understanding, essential for
  environmental interaction and collaboration with autonomous agents. Despite
  advancements in spatial reasoning within VLMs, recent research has shown that
  modern VLMs significantly lack perspective-aware reasoning capabilities and
  exhibit a strong bias toward egocentric interpretations. To bridge the gap
  between VLMs and human perception, we focus on the role of mental imagery,
  where humans perceive the world through abstracted representations that
  facilitate perspective shifts. Motivated by this, we propose a framework for
  perspective-aware reasoning, named Abstract Perspective Change (APC), that
  effectively leverages vision foundation models, such as object detection,
  segmentation, and orientation estimation, to construct scene abstractions and
  enable perspective transformations. Our experiments on synthetic and real-image
  benchmarks, compared with various VLMs, demonstrate significant improvements in
  perspective-aware reasoning with our framework, further outperforming
  fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.
paper_authors._id: 680af2bf2fa10fbf21684bde
paper_authors.name: Phillip Y. Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-26T08:53:40.936Z
paper_authors._id: 6342796a0875f2c99cfd313b
paper_authors.avatarUrl: /avatars/98575092404c4197b20c929a6499a015.svg
paper_authors.fullname: Yuseung "Phillip" Lee
paper_authors.type: user
paper_authors.user: phillipinseoul
paper_authors._id: 680af2bf2fa10fbf21684bdf
paper_authors.name: Jihyeon Je
paper_authors._id: 680af2bf2fa10fbf21684be0
paper_authors.name: Chanho Park
paper_authors._id: 680af2bf2fa10fbf21684be1
paper_authors.name: Mikaela Angelina Uy
paper_authors._id: 680af2bf2fa10fbf21684be2
paper_authors.name: Leonidas Guibas
paper_authors._id: 680af2bf2fa10fbf21684be3
paper_authors.name: Minhyuk Sung
paper_title: Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery
    Simulation
paper_summary: We present a framework for perspective-aware reasoning in vision-language
  models (VLMs) through mental imagery simulation. Perspective-taking, the
  ability to perceive an environment or situation from an alternative viewpoint,
  is a key benchmark for human-level visual understanding, essential for
  environmental interaction and collaboration with autonomous agents. Despite
  advancements in spatial reasoning within VLMs, recent research has shown that
  modern VLMs significantly lack perspective-aware reasoning capabilities and
  exhibit a strong bias toward egocentric interpretations. To bridge the gap
  between VLMs and human perception, we focus on the role of mental imagery,
  where humans perceive the world through abstracted representations that
  facilitate perspective shifts. Motivated by this, we propose a framework for
  perspective-aware reasoning, named Abstract Perspective Change (APC), that
  effectively leverages vision foundation models, such as object detection,
  segmentation, and orientation estimation, to construct scene abstractions and
  enable perspective transformations. Our experiments on synthetic and real-image
  benchmarks, compared with various VLMs, demonstrate significant improvements in
  perspective-aware reasoning with our framework, further outperforming
  fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.
paper_ai_summary: A framework named Abstract Perspective Change (APC) enhances perspective-aware reasoning in vision-language models by utilizing vision foundation models to create scene abstractions and enable perspective shifts.
paper_ai_keywords: vision-language models mental imagery simulation perspective-taking spatial reasoning scene abstractions perspective transformations object detection segmentation orientation estimation synthesis Abstract Perspective Change (APC)