Spatial Mental Modeling from Limited Views
Can Vision Language Models (VLMs) imagine the full scene from just a few
views, like humans do? Humans form spatial mental models, internal
representations of unseen space, to reason about layout, perspective, and
motion. Our new MindCube benchmark with 21,154 questions across 3,268 images
exposes this critical gap, where existing VLMs exhibit near-random performance.
Using MindCube, we systematically evaluate how well VLMs build robust spatial
mental models through representing positions (cognitive mapping), orientations
(perspective-taking), and dynamics (mental simulation for "what-if" movements).
We then explore three approaches to help VLMs approximate spatial mental
models, including unseen intermediate views, natural language reasoning chains,
and cognitive maps. The significant improvement comes from a synergistic
approach, "map-then-reason", that jointly trains the model to first generate a
cognitive map and then reason upon it. By training models to reason over these
internal maps, we boosted accuracy from 37.8% to 60.8% (+23.0%). Adding
reinforcement learning pushed performance even further to 70.7% (+32.9%). Our
key insight is that such scaffolding of spatial mental models, actively
constructing and utilizing internal structured spatial representations with
flexible reasoning processes, significantly improves understanding of
unobservable space.
685e93e2efec81aca075fd2a
Baiqiao Yin
685e93e2efec81aca075fd2b
Qineng Wang
claimed_verified
2025-07-01T10:14:36.351Z
640c3b28c5fa12d61a50cd92
/avatars/81556de3214c848b3c3e118f50fd2968.svg
Qineng Wang
user
Inevitablevalor
685e93e2efec81aca075fd2c
Pingyue Zhang
685e93e2efec81aca075fd2d
Jianshu Zhang
claimed_verified
2025-07-02T07:41:17.538Z
65d8b0f0661492b25c6623de
https://cdn-avatars.huggingface.co/v1/production/uploads/65d8b0f0661492b25c6623de/c6LPDse8NIV_3BHIu8dYe.png
Jianshu Zhang
user
Sterzhang
685e93e2efec81aca075fd2e
Kangrui Wang
685e93e2efec81aca075fd2f
Zihan Wang
685e93e2efec81aca075fd30
Jieyu Zhang
685e93e2efec81aca075fd31
Keshigeyan Chandrasegaran
claimed_verified
2025-07-08T08:06:59.489Z
65455f38c194936469130551
https://cdn-avatars.huggingface.co/v1/production/uploads/65455f38c194936469130551/nbZ25fFyZCymPEETOUauO.png
Keshigeyan Chandrasegaran
user
keshik6
685e93e2efec81aca075fd32
Han Liu
685e93e2efec81aca075fd33
Ranjay Krishna
685e93e2efec81aca075fd34
Saining Xie
685e93e2efec81aca075fd35
Manling Li
685e93e2efec81aca075fd36
Jiajun Wu
685e93e2efec81aca075fd37
Li Fei-Fei
Spatial Mental Modeling from Limited Views
Can Vision Language Models (VLMs) imagine the full scene from just a few
views, like humans do? Humans form spatial mental models, internal
representations of unseen space, to reason about layout, perspective, and
motion. Our new MindCube benchmark with 21,154 questions across 3,268 images
exposes this critical gap, where existing VLMs exhibit near-random performance.
Using MindCube, we systematically evaluate how well VLMs build robust spatial
mental models through representing positions (cognitive mapping), orientations
(perspective-taking), and dynamics (mental simulation for "what-if" movements).
We then explore three approaches to help VLMs approximate spatial mental
models, including unseen intermediate views, natural language reasoning chains,
and cognitive maps. The significant improvement comes from a synergistic
approach, "map-then-reason", that jointly trains the model to first generate a
cognitive map and then reason upon it. By training models to reason over these
internal maps, we boosted accuracy from 37.8% to 60.8% (+23.0%). Adding
reinforcement learning pushed performance even further to 70.7% (+32.9%). Our
key insight is that such scaffolding of spatial mental models, actively
constructing and utilizing internal structured spatial representations with
flexible reasoning processes, significantly improves understanding of
unobservable space.
A new benchmark, MindCube, shows that VLMs can improve their understanding of unseen spaces by forming internal spatial representations and reasoning over them.
Vision Language Models MindCube spatial mental models cognitive mapping perspective-taking mental simulation cognitive maps map-then-reason reinforcement learning