Do Vision-Language Models Have Internal World Models? Towards an Atomic
  Evaluation
Internal world models (WMs) enable agents to understand the world's state and
predict transitions, serving as the basis for advanced deliberative reasoning.
Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and
Gemini, exhibit potential as general-purpose WMs. While the latest studies have
evaluated and shown limitations in specific capabilities such as visual
understanding, a systematic evaluation of VLMs' fundamental WM abilities
remains absent. Drawing on comparative psychology and cognitive science, we
propose a two-stage framework that assesses Perception (visual, spatial,
temporal, quantitative, and motion) and Prediction (mechanistic simulation,
transitive inference, compositional inference) to provide an atomic evaluation
of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale
benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse
simulated environments with controlled counterfactual simulations. Through 660
experiments on 15 latest commercial and open-source VLMs, we find that these
models exhibit striking limitations in basic world modeling abilities. For
instance, almost all models perform at near-random accuracy when distinguishing
motion trajectories. Additionally, they lack disentangled understanding --
e.g., some models tend to believe blue objects move faster than green ones.
More rich results and analyses reveal significant gaps between VLMs and
human-level world modeling.
6862b46b3b7fc785c7ca5e1b
Qiyue Gao
claimed_verified
2025-07-03T07:33:00.560Z
65a0b8de51e699b22a6da8ec
/avatars/b133a48e5011041fa840d94e0c1060ec.svg
Qiyue Gao
user
BertG666
6862b46b3b7fc785c7ca5e1c
Xinyu Pi
6862b46b3b7fc785c7ca5e1d
Kevin Liu
6862b46b3b7fc785c7ca5e1e
Junrong Chen
claimed_verified
2025-07-09T08:52:57.796Z
66abcc9d308c1f8ae9e1c0de
/avatars/f00796ea1b95cd5c8caf4f392d1e9874.svg
Junrong Chen
user
JunrongChen2004
6862b46b3b7fc785c7ca5e1f
Ruolan Yang
6862b46b3b7fc785c7ca5e20
Xinqi Huang
6862b46b3b7fc785c7ca5e21
Xinyu Fang
6862b46b3b7fc785c7ca5e22
Lu Sun
6862b46b3b7fc785c7ca5e23
Gautham Kishore
claimed_verified
2025-07-02T07:41:04.988Z
616c60ff2e49649b9f3b34ef
/avatars/ad161dd9eb8a2955540ec556dfaa92d7.svg
Gautham Kishore
user
gauthk6
6862b46b3b7fc785c7ca5e24
Bo Ai
6862b46b3b7fc785c7ca5e25
Stone Tao
6862b46b3b7fc785c7ca5e26
Mengyang Liu
6862b46b3b7fc785c7ca5e27
Jiaxi Yang
6862b46b3b7fc785c7ca5e28
Chao-Jung Lai
6862b46b3b7fc785c7ca5e29
Chuanyang Jin
claimed_verified
2025-07-02T07:41:07.215Z
64beb69801f1983a86a05de2
https://cdn-avatars.huggingface.co/v1/production/uploads/64beb69801f1983a86a05de2/tFyCoqZ6gT8NWkZfuncID.jpeg
Chuanyang Jin
user
Chuanyang-Jin
6862b46b3b7fc785c7ca5e2a
Jiannan Xiang
6862b46b3b7fc785c7ca5e2b
Benhao Huang
claimed_verified
2025-07-01T10:14:12.282Z
64b22e6b0a54158d66f18688
https://cdn-avatars.huggingface.co/v1/production/uploads/64b22e6b0a54158d66f18688/Cbl3oMMMANbnCMoSUYenI.png
Benhao Huang
user
HuskyDoge
6862b46b3b7fc785c7ca5e2c
Zeming Chen
6862b46b3b7fc785c7ca5e2d
David Danks
6862b46b3b7fc785c7ca5e2e
Hao Su
6862b46b3b7fc785c7ca5e2f
Tianmin Shu
6862b46b3b7fc785c7ca5e30
Ziqiao Ma
claimed_verified
2025-07-04T07:50:46.927Z
630cfc45b66f088d547b2768
https://cdn-avatars.huggingface.co/v1/production/uploads/630cfc45b66f088d547b2768/ge2BZLDYeTlpnlIlraHNw.png
Martin Ziqiao Ma
user
marstin
6862b46b3b7fc785c7ca5e31
Lianhui Qin
6862b46b3b7fc785c7ca5e32
Zhiting Hu
Do Vision-Language Models Have Internal World Models? Towards an Atomic
  Evaluation
Internal world models (WMs) enable agents to understand the world's state and
predict transitions, serving as the basis for advanced deliberative reasoning.
Recent large Vision-Language Models (VLMs), such as OpenAI o3, GPT-4o and
Gemini, exhibit potential as general-purpose WMs. While the latest studies have
evaluated and shown limitations in specific capabilities such as visual
understanding, a systematic evaluation of VLMs' fundamental WM abilities
remains absent. Drawing on comparative psychology and cognitive science, we
propose a two-stage framework that assesses Perception (visual, spatial,
temporal, quantitative, and motion) and Prediction (mechanistic simulation,
transitive inference, compositional inference) to provide an atomic evaluation
of VLMs as WMs. Guided by this framework, we introduce WM-ABench, a large-scale
benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse
simulated environments with controlled counterfactual simulations. Through 660
experiments on 15 latest commercial and open-source VLMs, we find that these
models exhibit striking limitations in basic world modeling abilities. For
instance, almost all models perform at near-random accuracy when distinguishing
motion trajectories. Additionally, they lack disentangled understanding --
e.g., some models tend to believe blue objects move faster than green ones.
More rich results and analyses reveal significant gaps between VLMs and
human-level world modeling.
A benchmark framework evaluates the world modeling capabilities of Vision-Language Models, highlighting their limitations in perception and prediction.
internal world models Vision-Language Models o3 GPT-4o Gemini comparative psychology cognitive science perceptual evaluation prediction evaluation mechanical simulation transitive inference compositional inference WM-ABench simulated environments disentangled understanding