title: EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied
    World Models
summary: Recent advances in creative AI have enabled the synthesis of high-fidelity
  images and videos conditioned on language instructions. Building on these
  developments, text-to-video diffusion models have evolved into embodied world
  models (EWMs) capable of generating physically plausible scenes from language
  commands, effectively bridging vision and action in embodied AI applications.
  This work addresses the critical challenge of evaluating EWMs beyond general
  perceptual metrics to ensure the generation of physically grounded and
  action-consistent behaviors. We propose the Embodied World Model Benchmark
  (EWMBench), a dedicated framework designed to evaluate EWMs based on three key
  aspects: visual scene consistency, motion correctness, and semantic alignment.
  Our approach leverages a meticulously curated dataset encompassing diverse
  scenes and motion patterns, alongside a comprehensive multi-dimensional
  evaluation toolkit, to assess and compare candidate models. The proposed
  benchmark not only identifies the limitations of existing video generation
  models in meeting the unique requirements of embodied tasks but also provides
  valuable insights to guide future advancements in the field. The dataset and
  evaluation tools are publicly available at
  https://github.com/AgibotTech/EWMBench.
paper_authors._id: 6826ae4611765454f5757d7c
paper_authors.name: Hu Yue
paper_authors._id: 6826ae4611765454f5757d7d
paper_authors.name: Siyuan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:16:33.829Z
paper_authors._id: 63c7a33121bd95f80ed74652
paper_authors.avatarUrl: /avatars/7dd59afea785a2bff0ec2b757abd474e.svg
paper_authors.fullname: Siyuan Huang
paper_authors.type: user
paper_authors.user: thuhsy
paper_authors._id: 6826ae4611765454f5757d7e
paper_authors.name: Yue Liao
paper_authors._id: 6826ae4611765454f5757d7f
paper_authors.name: Shengcong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:16:15.416Z
paper_authors._id: 6575f9aeca03b6c514fe6e5c
paper_authors.avatarUrl: /avatars/a6e9d428beaa124ee989d702b9bf4f85.svg
paper_authors.fullname: Shengcong Chen
paper_authors.type: user
paper_authors.user: Shengcong
paper_authors._id: 6826ae4611765454f5757d80
paper_authors.name: Pengfei Zhou
paper_authors._id: 6826ae4611765454f5757d81
paper_authors.name: Liliang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:15:58.407Z
paper_authors._id: 640b00555a9c21b95c6449b3
paper_authors.avatarUrl: /avatars/5fa43b956f3acc671f033e31b7ca76c5.svg
paper_authors.fullname: Liliang Chen
paper_authors.type: user
paper_authors.user: pathcn
paper_authors._id: 6826ae4611765454f5757d82
paper_authors.name: Maoqing Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:15:51.039Z
paper_authors._id: 67739bfa64e8b7438ae68eb4
paper_authors.avatarUrl: /avatars/15193bfbce487b2de4ce8c86bd18885a.svg
paper_authors.fullname: Maoqing Yao
paper_authors.type: user
paper_authors.user: AutobotZero
paper_authors._id: 6826ae4611765454f5757d83
paper_authors.name: Guanghui Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-16T08:15:44.660Z
paper_authors._id: 646ec9b135f55eb49e405faa
paper_authors.avatarUrl: /avatars/a17194be585d20e2a021e77a5a20e213.svg
paper_authors.fullname: Guanghui Ren
paper_authors.type: user
paper_authors.user: sundrops
paper_title: EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied
    World Models
paper_summary: Recent advances in creative AI have enabled the synthesis of high-fidelity
  images and videos conditioned on language instructions. Building on these
  developments, text-to-video diffusion models have evolved into embodied world
  models (EWMs) capable of generating physically plausible scenes from language
  commands, effectively bridging vision and action in embodied AI applications.
  This work addresses the critical challenge of evaluating EWMs beyond general
  perceptual metrics to ensure the generation of physically grounded and
  action-consistent behaviors. We propose the Embodied World Model Benchmark
  (EWMBench), a dedicated framework designed to evaluate EWMs based on three key
  aspects: visual scene consistency, motion correctness, and semantic alignment.
  Our approach leverages a meticulously curated dataset encompassing diverse
  scenes and motion patterns, alongside a comprehensive multi-dimensional
  evaluation toolkit, to assess and compare candidate models. The proposed
  benchmark not only identifies the limitations of existing video generation
  models in meeting the unique requirements of embodied tasks but also provides
  valuable insights to guide future advancements in the field. The dataset and
  evaluation tools are publicly available at
  https://github.com/AgibotTech/EWMBench.
paper_ai_summary: A new benchmark framework evaluates text-to-video diffusion models in embodied AI for visual, motion, and semantic consistency, using a diverse dataset and evaluation toolkit.
paper_ai_keywords: text-to-video diffusion models embodied world models visual scene consistency motion correctness semantic alignment evaluation framework benchmark