Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual
  Simulations
Spatial cognition is essential for human intelligence, enabling
problem-solving through visual simulations rather than solely relying on verbal
reasoning. However, existing AI benchmarks primarily assess verbal reasoning,
neglecting the complexities of non-verbal, multi-step visual simulation. We
introduce STARE(Spatial Transformations and Reasoning Evaluation), a benchmark
designed to rigorously evaluate multimodal large language models on tasks
better solved through multi-step visual simulation. STARE features 4K tasks
spanning foundational geometric transformations (2D and 3D), integrated spatial
reasoning (cube net folding and tangram puzzles), and real-world spatial
reasoning (perspective and temporal reasoning), reflecting practical cognitive
challenges like object assembly, mechanical diagram interpretation, and
everyday spatial navigation. Our evaluations show that models excel at
reasoning over simpler 2D transformations, but perform close to random chance
on more complex tasks like 3D cube net folding and tangram puzzles that require
multi-step visual simulations. Humans achieve near-perfect accuracy but take
considerable time (up to 28.9s) on complex tasks, significantly speeding up
(down by 7.5 seconds on average) with intermediate visual simulations. In
contrast, models exhibit inconsistent performance gains from visual
simulations, improving on most tasks but declining in specific cases like
tangram puzzles (GPT-4o, o1) and cube net folding (Claude-3.5, Gemini-2.0
Flash), indicating that models may not know how to effectively leverage
intermediate visual information.
68426296b8d07a60074e866a
Linjie Li
68426296b8d07a60074e866b
Mahtab Bigverdi
68426296b8d07a60074e866c
Jiawei Gu
claimed_verified
2025-06-06T07:40:38.249Z
645b4819f9d4ec91fdd54852
/avatars/e12efb8e030688a0afcc72176b453fb3.svg
Jiawei Gu
user
kuvvi
68426296b8d07a60074e866d
Zixian Ma
68426296b8d07a60074e866e
Yinuo Yang
68426296b8d07a60074e866f
Ziang Li
68426296b8d07a60074e8670
Yejin Choi
68426296b8d07a60074e8671
Ranjay Krishna
Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual
  Simulations
Spatial cognition is essential for human intelligence, enabling
problem-solving through visual simulations rather than solely relying on verbal
reasoning. However, existing AI benchmarks primarily assess verbal reasoning,
neglecting the complexities of non-verbal, multi-step visual simulation. We
introduce STARE(Spatial Transformations and Reasoning Evaluation), a benchmark
designed to rigorously evaluate multimodal large language models on tasks
better solved through multi-step visual simulation. STARE features 4K tasks
spanning foundational geometric transformations (2D and 3D), integrated spatial
reasoning (cube net folding and tangram puzzles), and real-world spatial
reasoning (perspective and temporal reasoning), reflecting practical cognitive
challenges like object assembly, mechanical diagram interpretation, and
everyday spatial navigation. Our evaluations show that models excel at
reasoning over simpler 2D transformations, but perform close to random chance
on more complex tasks like 3D cube net folding and tangram puzzles that require
multi-step visual simulations. Humans achieve near-perfect accuracy but take
considerable time (up to 28.9s) on complex tasks, significantly speeding up
(down by 7.5 seconds on average) with intermediate visual simulations. In
contrast, models exhibit inconsistent performance gains from visual
simulations, improving on most tasks but declining in specific cases like
tangram puzzles (GPT-4o, o1) and cube net folding (Claude-3.5, Gemini-2.0
Flash), indicating that models may not know how to effectively leverage
intermediate visual information.
A new benchmark evaluates multimodal models on visual simulation tasks, revealing varying model performances compared to human accuracy and the impact of intermediate visual simulations.
spatial cognition visual simulations verbal reasoning multimodal large language models STARE spatial transformations geometric transformations integrated spatial reasoning real-world spatial reasoning visual reasoning intermediate visual simulations