WorldSimBench: Towards Video Generation Models as World Simulators
Recent advancements in predictive models have demonstrated exceptional
capabilities in predicting the future state of objects and scenes. However, the
lack of categorization based on inherent characteristics continues to hinder
the progress of predictive model development. Additionally, existing benchmarks
are unable to effectively evaluate higher-capability, highly embodied
predictive models from an embodied perspective. In this work, we classify the
functionalities of predictive models into a hierarchy and take the first step
in evaluating World Simulators by proposing a dual evaluation framework called
WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and
Implicit Manipulative Evaluation, encompassing human preference assessments
from the visual perspective and action-level evaluations in embodied tasks,
covering three representative embodied scenarios: Open-Ended Embodied
Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit
Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment
dataset based on fine-grained human feedback, which we use to train a Human
Preference Evaluator that aligns with human perception and explicitly assesses
the visual fidelity of World Simulators. In the Implicit Manipulative
Evaluation, we assess the video-action consistency of World Simulators by
evaluating whether the generated situation-aware video can be accurately
translated into the correct control signals in dynamic environments. Our
comprehensive evaluation offers key insights that can drive further innovation
in video generation models, positioning World Simulators as a pivotal
advancement toward embodied artificial intelligence.
6719dd0c0a835a6043db72e5
Yiran Qin
claimed_verified
2025-03-21T11:50:02.506Z
658a6c1399ed106ac8c822b1
https://cdn-avatars.huggingface.co/v1/production/uploads/658a6c1399ed106ac8c822b1/Wk2KXCcK39rUvXx6mpmGD.jpeg
yiranqin
user
IranQin
6719dd0c0a835a6043db72e6
Zhelun Shi
admin_assigned
2024-10-24T08:33:12.384Z
64f5d8b05619fe1fe06ed1a7
/avatars/61016a0533760dafacefe8ce6e344318.svg
Zhelun Shi
user
CoachXP
6719dd0c0a835a6043db72e7
Jiwen Yu
claimed_verified
2025-01-15T08:49:49.417Z
64105a6d14215c0775dfdd14
https://cdn-avatars.huggingface.co/v1/production/uploads/64105a6d14215c0775dfdd14/-VX-cUYOLjHIg7QnWhRGG.jpeg
Jiwen Yu
user
VictorYuki
6719dd0c0a835a6043db72e8
Xijun Wang
6719dd0c0a835a6043db72e9
Enshen Zhou
claimed_verified
2024-10-24T07:15:21.418Z
63f08dc79cf89c9ed1bb89cd
/avatars/37290358ad00bbd752f519cfdec02f3e.svg
Zhoues
user
Zhoues
6719dd0c0a835a6043db72ea
Lijun Li
admin_assigned
2024-10-24T08:32:08.764Z
64631d6cd4d34b01f45b013f
/avatars/1119ca2fa9e9893cbe856560ac67b2c6.svg
JL
user
LIJUNLI
6719dd0c0a835a6043db72eb
Zhenfei Yin
admin_assigned
2024-10-24T08:31:58.215Z
64e314ad24809d7fa0f20fbc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bHE0w_hjDFvU-Aul0_E7g.jpeg
Zhenfei Yin
user
JeremyYin
6719dd0c0a835a6043db72ec
Xihui Liu
admin_assigned
2024-10-24T08:31:51.537Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
6719dd0c0a835a6043db72ed
Lu Sheng
claimed_verified
2024-12-06T15:57:51.791Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
6719dd0c0a835a6043db72ee
Jing Shao
6719dd0c0a835a6043db72ef
Lei Bai
6719dd0c0a835a6043db72f0
Wanli Ouyang
6719dd0c0a835a6043db72f1
Ruimao Zhang
admin_assigned
2024-10-24T08:33:32.983Z
65324848220f490dbaa3b8c9
/avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
Zhang
user
Ruimao
WorldSimBench: Towards Video Generation Models as World Simulators
Recent advancements in predictive models have demonstrated exceptional
capabilities in predicting the future state of objects and scenes. However, the
lack of categorization based on inherent characteristics continues to hinder
the progress of predictive model development. Additionally, existing benchmarks
are unable to effectively evaluate higher-capability, highly embodied
predictive models from an embodied perspective. In this work, we classify the
functionalities of predictive models into a hierarchy and take the first step
in evaluating World Simulators by proposing a dual evaluation framework called
WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and
Implicit Manipulative Evaluation, encompassing human preference assessments
from the visual perspective and action-level evaluations in embodied tasks,
covering three representative embodied scenarios: Open-Ended Embodied
Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit
Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment
dataset based on fine-grained human feedback, which we use to train a Human
Preference Evaluator that aligns with human perception and explicitly assesses
the visual fidelity of World Simulators. In the Implicit Manipulative
Evaluation, we assess the video-action consistency of World Simulators by
evaluating whether the generated situation-aware video can be accurately
translated into the correct control signals in dynamic environments. Our
comprehensive evaluation offers key insights that can drive further innovation
in video generation models, positioning World Simulators as a pivotal
advancement toward embodied artificial intelligence.
A dual evaluation framework, WorldSimBench, is proposed to assess World Simulators from both visual and action perspectives using explicit perceptual evaluation and implicit manipulative evaluation in embodied scenarios.
World Simulators WorldSimBench Explicit Perceptual Evaluation Implicit Manipulative Evaluation HF-Embodied Dataset Human Preference Evaluator video-action consistency embodied artificial intelligence