title: WorldSimBench: Towards Video Generation Models as World Simulators
summary: Recent advancements in predictive models have demonstrated exceptional
  capabilities in predicting the future state of objects and scenes. However, the
  lack of categorization based on inherent characteristics continues to hinder
  the progress of predictive model development. Additionally, existing benchmarks
  are unable to effectively evaluate higher-capability, highly embodied
  predictive models from an embodied perspective. In this work, we classify the
  functionalities of predictive models into a hierarchy and take the first step
  in evaluating World Simulators by proposing a dual evaluation framework called
  WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and
  Implicit Manipulative Evaluation, encompassing human preference assessments
  from the visual perspective and action-level evaluations in embodied tasks,
  covering three representative embodied scenarios: Open-Ended Embodied
  Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit
  Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment
  dataset based on fine-grained human feedback, which we use to train a Human
  Preference Evaluator that aligns with human perception and explicitly assesses
  the visual fidelity of World Simulators. In the Implicit Manipulative
  Evaluation, we assess the video-action consistency of World Simulators by
  evaluating whether the generated situation-aware video can be accurately
  translated into the correct control signals in dynamic environments. Our
  comprehensive evaluation offers key insights that can drive further innovation
  in video generation models, positioning World Simulators as a pivotal
  advancement toward embodied artificial intelligence.
paper_authors._id: 6719dd0c0a835a6043db72e5
paper_authors.name: Yiran Qin
paper_authors._id: 6719dd0c0a835a6043db72e6
paper_authors.name: Zhelun Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:33:12.384Z
paper_authors._id: 64f5d8b05619fe1fe06ed1a7
paper_authors.avatarUrl: /avatars/61016a0533760dafacefe8ce6e344318.svg
paper_authors.fullname: Zhelun Shi
paper_authors.type: user
paper_authors.user: CoachXP
paper_authors._id: 6719dd0c0a835a6043db72e7
paper_authors.name: Jiwen Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:49:49.417Z
paper_authors._id: 64105a6d14215c0775dfdd14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64105a6d14215c0775dfdd14/-VX-cUYOLjHIg7QnWhRGG.jpeg
paper_authors.fullname: Jiwen Yu
paper_authors.type: user
paper_authors.user: VictorYuki
paper_authors._id: 6719dd0c0a835a6043db72e8
paper_authors.name: Xijun Wang
paper_authors._id: 6719dd0c0a835a6043db72e9
paper_authors.name: Enshen Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:21.418Z
paper_authors._id: 63f08dc79cf89c9ed1bb89cd
paper_authors.avatarUrl: /avatars/37290358ad00bbd752f519cfdec02f3e.svg
paper_authors.fullname: Zhoues
paper_authors.type: user
paper_authors.user: Zhoues
paper_authors._id: 6719dd0c0a835a6043db72ea
paper_authors.name: Lijun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:32:08.764Z
paper_authors._id: 64631d6cd4d34b01f45b013f
paper_authors.avatarUrl: /avatars/1119ca2fa9e9893cbe856560ac67b2c6.svg
paper_authors.fullname: JL
paper_authors.type: user
paper_authors.user: LIJUNLI
paper_authors._id: 6719dd0c0a835a6043db72eb
paper_authors.name: Zhenfei Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:31:58.215Z
paper_authors._id: 64e314ad24809d7fa0f20fbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bHE0w_hjDFvU-Aul0_E7g.jpeg
paper_authors.fullname: Zhenfei Yin
paper_authors.type: user
paper_authors.user: JeremyYin
paper_authors._id: 6719dd0c0a835a6043db72ec
paper_authors.name: Xihui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:31:51.537Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 6719dd0c0a835a6043db72ed
paper_authors.name: Lu Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:57:51.791Z
paper_authors._id: 65b722dbe02a17f0f8d1cc6b
paper_authors.avatarUrl: /avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
paper_authors.fullname: Lu Sheng
paper_authors.type: user
paper_authors.user: lsheng2024
paper_authors._id: 6719dd0c0a835a6043db72ee
paper_authors.name: Jing Shao
paper_authors._id: 6719dd0c0a835a6043db72ef
paper_authors.name: Lei Bai
paper_authors._id: 6719dd0c0a835a6043db72f0
paper_authors.name: Wanli Ouyang
paper_authors._id: 6719dd0c0a835a6043db72f1
paper_authors.name: Ruimao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:33:32.983Z
paper_authors._id: 65324848220f490dbaa3b8c9
paper_authors.avatarUrl: /avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Ruimao
paper_title: WorldSimBench: Towards Video Generation Models as World Simulators
paper_summary: Recent advancements in predictive models have demonstrated exceptional
  capabilities in predicting the future state of objects and scenes. However, the
  lack of categorization based on inherent characteristics continues to hinder
  the progress of predictive model development. Additionally, existing benchmarks
  are unable to effectively evaluate higher-capability, highly embodied
  predictive models from an embodied perspective. In this work, we classify the
  functionalities of predictive models into a hierarchy and take the first step
  in evaluating World Simulators by proposing a dual evaluation framework called
  WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and
  Implicit Manipulative Evaluation, encompassing human preference assessments
  from the visual perspective and action-level evaluations in embodied tasks,
  covering three representative embodied scenarios: Open-Ended Embodied
  Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit
  Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment
  dataset based on fine-grained human feedback, which we use to train a Human
  Preference Evaluator that aligns with human perception and explicitly assesses
  the visual fidelity of World Simulators. In the Implicit Manipulative
  Evaluation, we assess the video-action consistency of World Simulators by
  evaluating whether the generated situation-aware video can be accurately
  translated into the correct control signals in dynamic environments. Our
  comprehensive evaluation offers key insights that can drive further innovation
  in video generation models, positioning World Simulators as a pivotal
  advancement toward embodied artificial intelligence.
paper_ai_keywords: predictive models World Simulators WorldSimBench Explicit Perceptual Evaluation Implicit Manipulative Evaluation HF-Embodied Dataset Human Preference Evaluator video-action consistency situation-aware video control signals dynamic environments video generation models embodied artificial intelligence