title: PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
summary: We present PhysGen, a novel image-to-video generation method that converts a
  single image and an input condition (e.g., force and torque applied to an
  object in the image) to produce a realistic, physically plausible, and
  temporally consistent video. Our key insight is to integrate model-based
  physical simulation with a data-driven video generation process, enabling
  plausible image-space dynamics. At the heart of our system are three core
  components: (i) an image understanding module that effectively captures the
  geometry, materials, and physical parameters of the image; (ii) an image-space
  dynamics simulation model that utilizes rigid-body physics and inferred
  parameters to simulate realistic behaviors; and (iii) an image-based rendering
  and refinement module that leverages generative video diffusion to produce
  realistic video footage featuring the simulated motion. The resulting videos
  are realistic in both physics and appearance and are even precisely
  controllable, showcasing superior results over existing data-driven
  image-to-video generation works through quantitative comparison and
  comprehensive user study. PhysGen's resulting videos can be used for various
  downstream applications, such as turning an image into a realistic animation or
  allowing users to interact with the image and create various dynamics. Project
  page: https://stevenlsw.github.io/physgen/
paper_authors._id: 66fa2c8ba0802002c35431ab
paper_authors.name: Shaowei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:21:21.907Z
paper_authors._id: 670462c6fd5ef6902568d7bd
paper_authors.avatarUrl: /avatars/2d6ea275ccc289f6f1b07d0c8e860888.svg
paper_authors.fullname: Shaowei Liu
paper_authors.type: user
paper_authors.user: shaoweiliu
paper_authors._id: 66fa2c8ba0802002c35431ac
paper_authors.name: Zhongzheng Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T07:16:52.841Z
paper_authors._id: 63079a70161cfe1383a14b8c
paper_authors.avatarUrl: /avatars/ea28f8d595d6d510b8e9c06ce23e75c5.svg
paper_authors.fullname: Jason Ren
paper_authors.type: user
paper_authors.user: jasonren
paper_authors._id: 66fa2c8ba0802002c35431ad
paper_authors.name: Saurabh Gupta
paper_authors._id: 66fa2c8ba0802002c35431ae
paper_authors.name: Shenlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-30T12:42:19.365Z
paper_authors._id: 62fbe0aaa80632fbd47b86f5
paper_authors.avatarUrl: /avatars/8175e3ac325745714f17dbe000ea955f.svg
paper_authors.fullname: Shenlong Wang
paper_authors.type: user
paper_authors.user: shenlong
paper_title: PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
paper_summary: We present PhysGen, a novel image-to-video generation method that converts a
  single image and an input condition (e.g., force and torque applied to an
  object in the image) to produce a realistic, physically plausible, and
  temporally consistent video. Our key insight is to integrate model-based
  physical simulation with a data-driven video generation process, enabling
  plausible image-space dynamics. At the heart of our system are three core
  components: (i) an image understanding module that effectively captures the
  geometry, materials, and physical parameters of the image; (ii) an image-space
  dynamics simulation model that utilizes rigid-body physics and inferred
  parameters to simulate realistic behaviors; and (iii) an image-based rendering
  and refinement module that leverages generative video diffusion to produce
  realistic video footage featuring the simulated motion. The resulting videos
  are realistic in both physics and appearance and are even precisely
  controllable, showcasing superior results over existing data-driven
  image-to-video generation works through quantitative comparison and
  comprehensive user study. PhysGen's resulting videos can be used for various
  downstream applications, such as turning an image into a realistic animation or
  allowing users to interact with the image and create various dynamics. Project
  page: https://stevenlsw.github.io/physgen/
paper_ai_summary: PhysGen generates realistic, physically plausible videos from a single image and input conditions by integrating physical simulation with video diffusion.
paper_ai_keywords: image-to-video generation model-based physical simulation data-driven image understanding module image-space dynamics simulation rigid-body physics generative video diffusion image-based rendering realism quantitative comparison user study