title: Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual
    Simulations
summary: Spatial cognition is essential for human intelligence, enabling
  problem-solving through visual simulations rather than solely relying on verbal
  reasoning. However, existing AI benchmarks primarily assess verbal reasoning,
  neglecting the complexities of non-verbal, multi-step visual simulation. We
  introduce STARE(Spatial Transformations and Reasoning Evaluation), a benchmark
  designed to rigorously evaluate multimodal large language models on tasks
  better solved through multi-step visual simulation. STARE features 4K tasks
  spanning foundational geometric transformations (2D and 3D), integrated spatial
  reasoning (cube net folding and tangram puzzles), and real-world spatial
  reasoning (perspective and temporal reasoning), reflecting practical cognitive
  challenges like object assembly, mechanical diagram interpretation, and
  everyday spatial navigation. Our evaluations show that models excel at
  reasoning over simpler 2D transformations, but perform close to random chance
  on more complex tasks like 3D cube net folding and tangram puzzles that require
  multi-step visual simulations. Humans achieve near-perfect accuracy but take
  considerable time (up to 28.9s) on complex tasks, significantly speeding up
  (down by 7.5 seconds on average) with intermediate visual simulations. In
  contrast, models exhibit inconsistent performance gains from visual
  simulations, improving on most tasks but declining in specific cases like
  tangram puzzles (GPT-4o, o1) and cube net folding (Claude-3.5, Gemini-2.0
  Flash), indicating that models may not know how to effectively leverage
  intermediate visual information.
paper_authors._id: 68426296b8d07a60074e866a
paper_authors.name: Linjie Li
paper_authors._id: 68426296b8d07a60074e866b
paper_authors.name: Mahtab Bigverdi
paper_authors._id: 68426296b8d07a60074e866c
paper_authors.name: Jiawei Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:40:38.249Z
paper_authors._id: 645b4819f9d4ec91fdd54852
paper_authors.avatarUrl: /avatars/e12efb8e030688a0afcc72176b453fb3.svg
paper_authors.fullname: Jiawei Gu
paper_authors.type: user
paper_authors.user: kuvvi
paper_authors._id: 68426296b8d07a60074e866d
paper_authors.name: Zixian Ma
paper_authors._id: 68426296b8d07a60074e866e
paper_authors.name: Yinuo Yang
paper_authors._id: 68426296b8d07a60074e866f
paper_authors.name: Ziang Li
paper_authors._id: 68426296b8d07a60074e8670
paper_authors.name: Yejin Choi
paper_authors._id: 68426296b8d07a60074e8671
paper_authors.name: Ranjay Krishna
paper_title: Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual
    Simulations
paper_summary: Spatial cognition is essential for human intelligence, enabling
  problem-solving through visual simulations rather than solely relying on verbal
  reasoning. However, existing AI benchmarks primarily assess verbal reasoning,
  neglecting the complexities of non-verbal, multi-step visual simulation. We
  introduce STARE(Spatial Transformations and Reasoning Evaluation), a benchmark
  designed to rigorously evaluate multimodal large language models on tasks
  better solved through multi-step visual simulation. STARE features 4K tasks
  spanning foundational geometric transformations (2D and 3D), integrated spatial
  reasoning (cube net folding and tangram puzzles), and real-world spatial
  reasoning (perspective and temporal reasoning), reflecting practical cognitive
  challenges like object assembly, mechanical diagram interpretation, and
  everyday spatial navigation. Our evaluations show that models excel at
  reasoning over simpler 2D transformations, but perform close to random chance
  on more complex tasks like 3D cube net folding and tangram puzzles that require
  multi-step visual simulations. Humans achieve near-perfect accuracy but take
  considerable time (up to 28.9s) on complex tasks, significantly speeding up
  (down by 7.5 seconds on average) with intermediate visual simulations. In
  contrast, models exhibit inconsistent performance gains from visual
  simulations, improving on most tasks but declining in specific cases like
  tangram puzzles (GPT-4o, o1) and cube net folding (Claude-3.5, Gemini-2.0
  Flash), indicating that models may not know how to effectively leverage
  intermediate visual information.
paper_ai_summary: A new benchmark evaluates multimodal models on visual simulation tasks, revealing varying model performances compared to human accuracy and the impact of intermediate visual simulations.
paper_ai_keywords: spatial cognition visual simulations verbal reasoning multimodal large language models STARE spatial transformations geometric transformations integrated spatial reasoning real-world spatial reasoning visual reasoning intermediate visual simulations