title: How Far is Video Generation from World Model: A Physical Law Perspective
summary: OpenAI's Sora highlights the potential of video generation for developing
  world models that adhere to fundamental physical laws. However, the ability of
  video generation models to discover such laws purely from visual data without
  human priors can be questioned. A world model learning the true law should give
  predictions robust to nuances and correctly extrapolate on unseen scenarios. In
  this work, we evaluate across three key scenarios: in-distribution,
  out-of-distribution, and combinatorial generalization. We developed a 2D
  simulation testbed for object movement and collisions to generate videos
  deterministically governed by one or more classical mechanics laws. This
  provides an unlimited supply of data for large-scale experimentation and
  enables quantitative evaluation of whether the generated videos adhere to
  physical laws. We trained diffusion-based video generation models to predict
  object movements based on initial frames. Our scaling experiments show perfect
  generalization within the distribution, measurable scaling behavior for
  combinatorial generalization, but failure in out-of-distribution scenarios.
  Further experiments reveal two key insights about the generalization mechanisms
  of these models: (1) the models fail to abstract general physical rules and
  instead exhibit "case-based" generalization behavior, i.e., mimicking the
  closest training example; (2) when generalizing to new cases, models are
  observed to prioritize different factors when referencing training data: color
  > size > velocity > shape. Our study suggests that scaling alone is
  insufficient for video generation models to uncover fundamental physical laws,
  despite its role in Sora's broader success. See our project page at
  https://phyworld.github.io
paper_authors._id: 67298a7f6f4fe77234c6a1ff
paper_authors.name: Bingyi Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:04:24.046Z
paper_authors._id: 647b5fef6a79fbf5e996c47c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
paper_authors.fullname: Bingyi Kang
paper_authors.type: user
paper_authors.user: bykang
paper_authors._id: 67298a7f6f4fe77234c6a200
paper_authors.name: Yang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:57:49.069Z
paper_authors._id: 649d475111592b1a765ac1a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
paper_authors.fullname: Yang Yue
paper_authors.type: user
paper_authors.user: Yang130
paper_authors._id: 67298a7f6f4fe77234c6a201
paper_authors.name: Rui Lu
paper_authors._id: 67298a7f6f4fe77234c6a202
paper_authors.name: Zhijie Lin
paper_authors._id: 67298a7f6f4fe77234c6a203
paper_authors.name: Yang Zhao
paper_authors._id: 67298a7f6f4fe77234c6a204
paper_authors.name: Kaixin Wang
paper_authors._id: 67298a7f6f4fe77234c6a205
paper_authors.name: Gao Huang
paper_authors._id: 67298a7f6f4fe77234c6a206
paper_authors.name: Jiashi Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:04:36.108Z
paper_authors._id: 67298e44017b96a1d0101dc4
paper_authors.avatarUrl: /avatars/1f8ed1a3e911e6a3021087b9371d284c.svg
paper_authors.fullname: Jiashi Feng
paper_authors.type: user
paper_authors.user: jshfeng
paper_title: How Far is Video Generation from World Model: A Physical Law Perspective
paper_summary: OpenAI's Sora highlights the potential of video generation for developing
  world models that adhere to fundamental physical laws. However, the ability of
  video generation models to discover such laws purely from visual data without
  human priors can be questioned. A world model learning the true law should give
  predictions robust to nuances and correctly extrapolate on unseen scenarios. In
  this work, we evaluate across three key scenarios: in-distribution,
  out-of-distribution, and combinatorial generalization. We developed a 2D
  simulation testbed for object movement and collisions to generate videos
  deterministically governed by one or more classical mechanics laws. This
  provides an unlimited supply of data for large-scale experimentation and
  enables quantitative evaluation of whether the generated videos adhere to
  physical laws. We trained diffusion-based video generation models to predict
  object movements based on initial frames. Our scaling experiments show perfect
  generalization within the distribution, measurable scaling behavior for
  combinatorial generalization, but failure in out-of-distribution scenarios.
  Further experiments reveal two key insights about the generalization mechanisms
  of these models: (1) the models fail to abstract general physical rules and
  instead exhibit "case-based" generalization behavior, i.e., mimicking the
  closest training example; (2) when generalizing to new cases, models are
  observed to prioritize different factors when referencing training data: color
  > size > velocity > shape. Our study suggests that scaling alone is
  insufficient for video generation models to uncover fundamental physical laws,
  despite its role in Sora's broader success. See our project page at
  https://phyworld.github.io
paper_ai_keywords: diffusion-based video generation models in-distribution out-of-distribution combinatorial generalization 2D simulation testbed classical mechanics laws deterministic governed scaling experiments generalization mechanisms case-based generalization behavior referencing training data