How Far is Video Generation from World Model: A Physical Law Perspective
OpenAI's Sora highlights the potential of video generation for developing
world models that adhere to fundamental physical laws. However, the ability of
video generation models to discover such laws purely from visual data without
human priors can be questioned. A world model learning the true law should give
predictions robust to nuances and correctly extrapolate on unseen scenarios. In
this work, we evaluate across three key scenarios: in-distribution,
out-of-distribution, and combinatorial generalization. We developed a 2D
simulation testbed for object movement and collisions to generate videos
deterministically governed by one or more classical mechanics laws. This
provides an unlimited supply of data for large-scale experimentation and
enables quantitative evaluation of whether the generated videos adhere to
physical laws. We trained diffusion-based video generation models to predict
object movements based on initial frames. Our scaling experiments show perfect
generalization within the distribution, measurable scaling behavior for
combinatorial generalization, but failure in out-of-distribution scenarios.
Further experiments reveal two key insights about the generalization mechanisms
of these models: (1) the models fail to abstract general physical rules and
instead exhibit "case-based" generalization behavior, i.e., mimicking the
closest training example; (2) when generalizing to new cases, models are
observed to prioritize different factors when referencing training data: color
> size > velocity > shape. Our study suggests that scaling alone is
insufficient for video generation models to uncover fundamental physical laws,
despite its role in Sora's broader success. See our project page at
https://phyworld.github.io
67298a7f6f4fe77234c6a1ff
Bingyi Kang
admin_assigned
2024-11-05T14:04:24.046Z
647b5fef6a79fbf5e996c47c
https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
Bingyi Kang
user
bykang
67298a7f6f4fe77234c6a200
Yang Yue
claimed_verified
2024-11-05T07:57:49.069Z
649d475111592b1a765ac1a3
https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
Yang Yue
user
Yang130
67298a7f6f4fe77234c6a201
Rui Lu
claimed_verified
2025-04-22T10:17:23.104Z
64b77916964f6e7bf32bb428
/avatars/631b0324b4d713d351abbedb468658dc.svg
Rui Lu
user
RayLuTHU
67298a7f6f4fe77234c6a202
Zhijie Lin
67298a7f6f4fe77234c6a203
Yang Zhao
67298a7f6f4fe77234c6a204
Kaixin Wang
67298a7f6f4fe77234c6a205
Gao Huang
67298a7f6f4fe77234c6a206
Jiashi Feng
admin_assigned
2024-11-05T14:04:36.108Z
67298e44017b96a1d0101dc4
/avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
Jiashi Feng
user
jshfeng
How Far is Video Generation from World Model: A Physical Law Perspective
OpenAI's Sora highlights the potential of video generation for developing
world models that adhere to fundamental physical laws. However, the ability of
video generation models to discover such laws purely from visual data without
human priors can be questioned. A world model learning the true law should give
predictions robust to nuances and correctly extrapolate on unseen scenarios. In
this work, we evaluate across three key scenarios: in-distribution,
out-of-distribution, and combinatorial generalization. We developed a 2D
simulation testbed for object movement and collisions to generate videos
deterministically governed by one or more classical mechanics laws. This
provides an unlimited supply of data for large-scale experimentation and
enables quantitative evaluation of whether the generated videos adhere to
physical laws. We trained diffusion-based video generation models to predict
object movements based on initial frames. Our scaling experiments show perfect
generalization within the distribution, measurable scaling behavior for
combinatorial generalization, but failure in out-of-distribution scenarios.
Further experiments reveal two key insights about the generalization mechanisms
of these models: (1) the models fail to abstract general physical rules and
instead exhibit "case-based" generalization behavior, i.e., mimicking the
closest training example; (2) when generalizing to new cases, models are
observed to prioritize different factors when referencing training data: color
> size > velocity > shape. Our study suggests that scaling alone is
insufficient for video generation models to uncover fundamental physical laws,
despite its role in Sora's broader success. See our project page at
https://phyworld.github.io
Diffusion-based video generation models demonstrate perfect in-distribution generalization and measurable scaling behavior for combinatorial generalization but fail in out-of-distribution scenarios, prioritizing color over other factors.
diffusion-based video generation models in-distribution out-of-distribution combinatorial generalization generalization behavior case-based generalization