PhysDreamer: Physics-Based Interaction with 3D Objects via Video
  Generation
Realistic object interactions are crucial for creating immersive virtual
experiences, yet synthesizing realistic 3D object dynamics in response to novel
interactions remains a significant challenge. Unlike unconditional or
text-conditioned dynamics generation, action-conditioned dynamics requires
perceiving the physical material properties of objects and grounding the 3D
motion prediction on these properties, such as object stiffness. However,
estimating physical material properties is an open problem due to the lack of
material ground-truth data, as measuring these properties for real objects is
highly difficult. We present PhysDreamer, a physics-based approach that endows
static 3D objects with interactive dynamics by leveraging the object dynamics
priors learned by video generation models. By distilling these priors,
PhysDreamer enables the synthesis of realistic object responses to novel
interactions, such as external forces or agent manipulations. We demonstrate
our approach on diverse examples of elastic objects and evaluate the realism of
the synthesized interactions through a user study. PhysDreamer takes a step
towards more engaging and realistic virtual experiences by enabling static 3D
objects to dynamically respond to interactive stimuli in a physically plausible
manner. See our project page at https://physdreamer.github.io/.
6625ee7cf8639638f91f87e8
Tianyuan Zhang
claimed_verified
2024-04-23T07:15:37.440Z
6463e30728fafa56c3f1fcbf
/avatars/6b145b5bd5e5e7a9d1d55f191174486b.svg
Tianyuan Zhang
user
YunjinZhang
6625ee7cf8639638f91f87e9
Hong-Xing Yu
6625ee7cf8639638f91f87ea
Rundi Wu
admin_assigned
2024-04-22T14:02:49.530Z
63823d59a18d1ca09653aa21
/avatars/151e1ca46627abdd99c27945ea2fa441.svg
Rundi Wu
user
rundi
6625ee7cf8639638f91f87eb
Brandon Y. Feng
admin_assigned
2024-04-22T14:03:20.820Z
634ae1fbe97a3d183f2cf063
/avatars/3e05289d1876d7b7ef2c6a595a7f39d7.svg
Brandon Feng
user
brandonyfeng
6625ee7cf8639638f91f87ec
Changxi Zheng
6625ee7cf8639638f91f87ed
Noah Snavely
admin_assigned
2024-04-22T14:03:35.164Z
632b79ca0a9dc02985af9e0b
/avatars/e2ca585398a3212b2956a14804d0ba67.svg
Noah Snavely
user
jimantha
6625ee7cf8639638f91f87ee
Jiajun Wu
admin_assigned
2024-04-22T14:03:52.084Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
6625ee7cf8639638f91f87ef
William T. Freeman
admin_assigned
2024-04-22T14:04:03.062Z
654e866a85a5e608059a9b4f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O0T97EupMHH_iXcv1xYFr.png
William Freeman
user
mrpuppt
PhysDreamer: Physics-Based Interaction with 3D Objects via Video
  Generation
Realistic object interactions are crucial for creating immersive virtual
experiences, yet synthesizing realistic 3D object dynamics in response to novel
interactions remains a significant challenge. Unlike unconditional or
text-conditioned dynamics generation, action-conditioned dynamics requires
perceiving the physical material properties of objects and grounding the 3D
motion prediction on these properties, such as object stiffness. However,
estimating physical material properties is an open problem due to the lack of
material ground-truth data, as measuring these properties for real objects is
highly difficult. We present PhysDreamer, a physics-based approach that endows
static 3D objects with interactive dynamics by leveraging the object dynamics
priors learned by video generation models. By distilling these priors,
PhysDreamer enables the synthesis of realistic object responses to novel
interactions, such as external forces or agent manipulations. We demonstrate
our approach on diverse examples of elastic objects and evaluate the realism of
the synthesized interactions through a user study. PhysDreamer takes a step
towards more engaging and realistic virtual experiences by enabling static 3D
objects to dynamically respond to interactive stimuli in a physically plausible
manner. See our project page at https://physdreamer.github.io/.
PhysDreamer utilizes video generation models to simulate realistic 3D object dynamics based on learned action-conditioned priors, enhancing immersive virtual interactions.
object dynamics video generation models action-conditioned dynamics material properties motion prediction user study virtual interactions physically plausible