Towards World Simulator: Crafting Physical Commonsense-Based Benchmark
  for Video Generation
Text-to-video (T2V) models like Sora have made significant strides in
visualizing complex prompts, which is increasingly viewed as a promising path
towards constructing the universal world simulator. Cognitive psychologists
believe that the foundation for achieving this goal is the ability to
understand intuitive physics. However, the capacity of these models to
accurately represent intuitive physics remains largely unexplored. To bridge
this gap, we introduce PhyGenBench, a comprehensive Physics
Generation Benchmark designed to evaluate physical
commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully
crafted prompts across 27 distinct physical laws, spanning four fundamental
domains, which could comprehensively assesses models' understanding of physical
commonsense. Alongside PhyGenBench, we propose a novel evaluation framework
called PhyGenEval. This framework employs a hierarchical evaluation structure
utilizing appropriate advanced vision-language models and large language models
to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can
conduct large-scale automated assessments of T2V models' understanding of
physical commonsense, which align closely with human feedback. Our evaluation
results and in-depth analysis demonstrate that current models struggle to
generate videos that comply with physical commonsense. Moreover, simply scaling
up models or employing prompt engineering techniques is insufficient to fully
address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We
hope this study will inspire the community to prioritize the learning of
physical commonsense in these models beyond entertainment applications. We will
release the data and codes at https://github.com/OpenGVLab/PhyGenBench
67073022327cec6882e5d4c9
Fanqing Meng
admin_assigned
2024-10-10T08:27:47.816Z
640b37b2bab5ca8fbe7df8f2
/avatars/c7bef45efad6a0d911a720e2236fcba5.svg
fanqing meng
user
FanqingM
67073022327cec6882e5d4ca
Jiaqi Liao
admin_assigned
2024-10-10T08:28:33.069Z
6630b287ce65a66ed8f04eba
/avatars/18ad0e3b453021ea6006e23333e92c12.svg
liaojiaqi
user
ljq940913
67073022327cec6882e5d4cb
Xinyu Tan
67073022327cec6882e5d4cc
Wenqi Shao
admin_assigned
2024-10-10T08:29:10.044Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
67073022327cec6882e5d4cd
Quanfeng Lu
admin_assigned
2024-10-10T08:29:17.379Z
653a483dacdeea08424ef55d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tYG2isIZBLCaBbWABatgK.png
Quanfeng Lu
user
hflqf88888
67073022327cec6882e5d4ce
Kaipeng Zhang
admin_assigned
2024-10-10T08:29:23.401Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
67073022327cec6882e5d4cf
Yu Cheng
claimed_verified
2024-10-11T07:44:23.224Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
67073022327cec6882e5d4d0
Dianqi Li
67073022327cec6882e5d4d1
Yu Qiao
67073022327cec6882e5d4d2
Ping Luo
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark
  for Video Generation
Text-to-video (T2V) models like Sora have made significant strides in
visualizing complex prompts, which is increasingly viewed as a promising path
towards constructing the universal world simulator. Cognitive psychologists
believe that the foundation for achieving this goal is the ability to
understand intuitive physics. However, the capacity of these models to
accurately represent intuitive physics remains largely unexplored. To bridge
this gap, we introduce PhyGenBench, a comprehensive Physics
Generation Benchmark designed to evaluate physical
commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully
crafted prompts across 27 distinct physical laws, spanning four fundamental
domains, which could comprehensively assesses models' understanding of physical
commonsense. Alongside PhyGenBench, we propose a novel evaluation framework
called PhyGenEval. This framework employs a hierarchical evaluation structure
utilizing appropriate advanced vision-language models and large language models
to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can
conduct large-scale automated assessments of T2V models' understanding of
physical commonsense, which align closely with human feedback. Our evaluation
results and in-depth analysis demonstrate that current models struggle to
generate videos that comply with physical commonsense. Moreover, simply scaling
up models or employing prompt engineering techniques is insufficient to fully
address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We
hope this study will inspire the community to prioritize the learning of
physical commonsense in these models beyond entertainment applications. We will
release the data and codes at https://github.com/OpenGVLab/PhyGenBench
PhyGenBench and PhyGenEval assess text-to-video models' understanding of physical commonsense, revealing shortcomings that cannot be fully addressed by scaling models or prompt engineering.
text-to-video models physics generation benchmark PhyGenBench PhyGenEval physical commonsense vision-language models large language models automated assessments