title: Towards World Simulator: Crafting Physical Commonsense-Based Benchmark
    for Video Generation
summary: Text-to-video (T2V) models like Sora have made significant strides in
  visualizing complex prompts, which is increasingly viewed as a promising path
  towards constructing the universal world simulator. Cognitive psychologists
  believe that the foundation for achieving this goal is the ability to
  understand intuitive physics. However, the capacity of these models to
  accurately represent intuitive physics remains largely unexplored. To bridge
  this gap, we introduce PhyGenBench, a comprehensive Physics
  Generation Benchmark designed to evaluate physical
  commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully
  crafted prompts across 27 distinct physical laws, spanning four fundamental
  domains, which could comprehensively assesses models' understanding of physical
  commonsense. Alongside PhyGenBench, we propose a novel evaluation framework
  called PhyGenEval. This framework employs a hierarchical evaluation structure
  utilizing appropriate advanced vision-language models and large language models
  to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can
  conduct large-scale automated assessments of T2V models' understanding of
  physical commonsense, which align closely with human feedback. Our evaluation
  results and in-depth analysis demonstrate that current models struggle to
  generate videos that comply with physical commonsense. Moreover, simply scaling
  up models or employing prompt engineering techniques is insufficient to fully
  address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We
  hope this study will inspire the community to prioritize the learning of
  physical commonsense in these models beyond entertainment applications. We will
  release the data and codes at https://github.com/OpenGVLab/PhyGenBench
paper_authors._id: 67073022327cec6882e5d4c9
paper_authors.name: Fanqing Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:27:47.816Z
paper_authors._id: 640b37b2bab5ca8fbe7df8f2
paper_authors.avatarUrl: /avatars/c7bef45efad6a0d911a720e2236fcba5.svg
paper_authors.fullname: fanqing meng
paper_authors.type: user
paper_authors.user: FanqingM
paper_authors._id: 67073022327cec6882e5d4ca
paper_authors.name: Jiaqi Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:28:33.069Z
paper_authors._id: 6630b287ce65a66ed8f04eba
paper_authors.avatarUrl: /avatars/18ad0e3b453021ea6006e23333e92c12.svg
paper_authors.fullname: liaojiaqi
paper_authors.type: user
paper_authors.user: ljq940913
paper_authors._id: 67073022327cec6882e5d4cb
paper_authors.name: Xinyu Tan
paper_authors._id: 67073022327cec6882e5d4cc
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:29:10.044Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 67073022327cec6882e5d4cd
paper_authors.name: Quanfeng Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:29:17.379Z
paper_authors._id: 653a483dacdeea08424ef55d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tYG2isIZBLCaBbWABatgK.png
paper_authors.fullname: Quanfeng Lu
paper_authors.type: user
paper_authors.user: hflqf88888
paper_authors._id: 67073022327cec6882e5d4ce
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:29:23.401Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 67073022327cec6882e5d4cf
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:23.224Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 67073022327cec6882e5d4d0
paper_authors.name: Dianqi Li
paper_authors._id: 67073022327cec6882e5d4d1
paper_authors.name: Yu Qiao
paper_authors._id: 67073022327cec6882e5d4d2
paper_authors.name: Ping Luo
paper_title: Towards World Simulator: Crafting Physical Commonsense-Based Benchmark
    for Video Generation
paper_summary: Text-to-video (T2V) models like Sora have made significant strides in
  visualizing complex prompts, which is increasingly viewed as a promising path
  towards constructing the universal world simulator. Cognitive psychologists
  believe that the foundation for achieving this goal is the ability to
  understand intuitive physics. However, the capacity of these models to
  accurately represent intuitive physics remains largely unexplored. To bridge
  this gap, we introduce PhyGenBench, a comprehensive Physics
  Generation Benchmark designed to evaluate physical
  commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully
  crafted prompts across 27 distinct physical laws, spanning four fundamental
  domains, which could comprehensively assesses models' understanding of physical
  commonsense. Alongside PhyGenBench, we propose a novel evaluation framework
  called PhyGenEval. This framework employs a hierarchical evaluation structure
  utilizing appropriate advanced vision-language models and large language models
  to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can
  conduct large-scale automated assessments of T2V models' understanding of
  physical commonsense, which align closely with human feedback. Our evaluation
  results and in-depth analysis demonstrate that current models struggle to
  generate videos that comply with physical commonsense. Moreover, simply scaling
  up models or employing prompt engineering techniques is insufficient to fully
  address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We
  hope this study will inspire the community to prioritize the learning of
  physical commonsense in these models beyond entertainment applications. We will
  release the data and codes at https://github.com/OpenGVLab/PhyGenBench
paper_ai_keywords: Text-to-video (T2V) PhyGenBench intuitive physics common sense correctness advanced vision-language models large language models hierarchical evaluation structure