PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
We present PhysGen, a novel image-to-video generation method that converts a
single image and an input condition (e.g., force and torque applied to an
object in the image) to produce a realistic, physically plausible, and
temporally consistent video. Our key insight is to integrate model-based
physical simulation with a data-driven video generation process, enabling
plausible image-space dynamics. At the heart of our system are three core
components: (i) an image understanding module that effectively captures the
geometry, materials, and physical parameters of the image; (ii) an image-space
dynamics simulation model that utilizes rigid-body physics and inferred
parameters to simulate realistic behaviors; and (iii) an image-based rendering
and refinement module that leverages generative video diffusion to produce
realistic video footage featuring the simulated motion. The resulting videos
are realistic in both physics and appearance and are even precisely
controllable, showcasing superior results over existing data-driven
image-to-video generation works through quantitative comparison and
comprehensive user study. PhysGen's resulting videos can be used for various
downstream applications, such as turning an image into a realistic animation or
allowing users to interact with the image and create various dynamics. Project
page: https://stevenlsw.github.io/physgen/
66fa2c8ba0802002c35431ab
Shaowei Liu
claimed_verified
2024-10-08T07:21:21.907Z
670462c6fd5ef6902568d7bd
/avatars/2d6ea275ccc289f6f1b07d0c8e860888.svg
Shaowei Liu
user
shaoweiliu
66fa2c8ba0802002c35431ac
Zhongzheng Ren
claimed_verified
2024-10-01T07:16:52.841Z
63079a70161cfe1383a14b8c
/avatars/ea28f8d595d6d510b8e9c06ce23e75c5.svg
Jason Ren
user
jasonren
66fa2c8ba0802002c35431ad
Saurabh Gupta
66fa2c8ba0802002c35431ae
Shenlong Wang
admin_assigned
2024-09-30T12:42:19.365Z
62fbe0aaa80632fbd47b86f5
/avatars/8175e3ac325745714f17dbe000ea955f.svg
Shenlong Wang
user
shenlong
PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
We present PhysGen, a novel image-to-video generation method that converts a
single image and an input condition (e.g., force and torque applied to an
object in the image) to produce a realistic, physically plausible, and
temporally consistent video. Our key insight is to integrate model-based
physical simulation with a data-driven video generation process, enabling
plausible image-space dynamics. At the heart of our system are three core
components: (i) an image understanding module that effectively captures the
geometry, materials, and physical parameters of the image; (ii) an image-space
dynamics simulation model that utilizes rigid-body physics and inferred
parameters to simulate realistic behaviors; and (iii) an image-based rendering
and refinement module that leverages generative video diffusion to produce
realistic video footage featuring the simulated motion. The resulting videos
are realistic in both physics and appearance and are even precisely
controllable, showcasing superior results over existing data-driven
image-to-video generation works through quantitative comparison and
comprehensive user study. PhysGen's resulting videos can be used for various
downstream applications, such as turning an image into a realistic animation or
allowing users to interact with the image and create various dynamics. Project
page: https://stevenlsw.github.io/physgen/
PhysGen generates realistic, physically plausible videos from a single image and input conditions by integrating physical simulation with video diffusion.
image-to-video generation model-based physical simulation data-driven image understanding module image-space dynamics simulation rigid-body physics generative video diffusion image-based rendering realism quantitative comparison user study