title: PhysDreamer: Physics-Based Interaction with 3D Objects via Video
    Generation
summary: Realistic object interactions are crucial for creating immersive virtual
  experiences, yet synthesizing realistic 3D object dynamics in response to novel
  interactions remains a significant challenge. Unlike unconditional or
  text-conditioned dynamics generation, action-conditioned dynamics requires
  perceiving the physical material properties of objects and grounding the 3D
  motion prediction on these properties, such as object stiffness. However,
  estimating physical material properties is an open problem due to the lack of
  material ground-truth data, as measuring these properties for real objects is
  highly difficult. We present PhysDreamer, a physics-based approach that endows
  static 3D objects with interactive dynamics by leveraging the object dynamics
  priors learned by video generation models. By distilling these priors,
  PhysDreamer enables the synthesis of realistic object responses to novel
  interactions, such as external forces or agent manipulations. We demonstrate
  our approach on diverse examples of elastic objects and evaluate the realism of
  the synthesized interactions through a user study. PhysDreamer takes a step
  towards more engaging and realistic virtual experiences by enabling static 3D
  objects to dynamically respond to interactive stimuli in a physically plausible
  manner. See our project page at https://physdreamer.github.io/.
paper_authors._id: 6625ee7cf8639638f91f87e8
paper_authors.name: Tianyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-23T07:15:37.440Z
paper_authors._id: 6463e30728fafa56c3f1fcbf
paper_authors.avatarUrl: /avatars/6b145b5bd5e5e7a9d1d55f191174486b.svg
paper_authors.fullname: Tianyuan Zhang
paper_authors.type: user
paper_authors.user: YunjinZhang
paper_authors._id: 6625ee7cf8639638f91f87e9
paper_authors.name: Hong-Xing Yu
paper_authors._id: 6625ee7cf8639638f91f87ea
paper_authors.name: Rundi Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:02:49.530Z
paper_authors._id: 63823d59a18d1ca09653aa21
paper_authors.avatarUrl: /avatars/151e1ca46627abdd99c27945ea2fa441.svg
paper_authors.fullname: Rundi Wu
paper_authors.type: user
paper_authors.user: rundi
paper_authors._id: 6625ee7cf8639638f91f87eb
paper_authors.name: Brandon Y. Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:03:20.820Z
paper_authors._id: 634ae1fbe97a3d183f2cf063
paper_authors.avatarUrl: /avatars/3e05289d1876d7b7ef2c6a595a7f39d7.svg
paper_authors.fullname: Brandon Feng
paper_authors.type: user
paper_authors.user: brandonyfeng
paper_authors._id: 6625ee7cf8639638f91f87ec
paper_authors.name: Changxi Zheng
paper_authors._id: 6625ee7cf8639638f91f87ed
paper_authors.name: Noah Snavely
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:03:35.164Z
paper_authors._id: 632b79ca0a9dc02985af9e0b
paper_authors.avatarUrl: /avatars/e2ca585398a3212b2956a14804d0ba67.svg
paper_authors.fullname: Noah Snavely
paper_authors.type: user
paper_authors.user: jimantha
paper_authors._id: 6625ee7cf8639638f91f87ee
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:03:52.084Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 6625ee7cf8639638f91f87ef
paper_authors.name: William T. Freeman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:04:03.062Z
paper_authors._id: 654e866a85a5e608059a9b4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/O0T97EupMHH_iXcv1xYFr.png
paper_authors.fullname: William Freeman
paper_authors.type: user
paper_authors.user: mrpuppt
paper_title: PhysDreamer: Physics-Based Interaction with 3D Objects via Video
    Generation
paper_summary: Realistic object interactions are crucial for creating immersive virtual
  experiences, yet synthesizing realistic 3D object dynamics in response to novel
  interactions remains a significant challenge. Unlike unconditional or
  text-conditioned dynamics generation, action-conditioned dynamics requires
  perceiving the physical material properties of objects and grounding the 3D
  motion prediction on these properties, such as object stiffness. However,
  estimating physical material properties is an open problem due to the lack of
  material ground-truth data, as measuring these properties for real objects is
  highly difficult. We present PhysDreamer, a physics-based approach that endows
  static 3D objects with interactive dynamics by leveraging the object dynamics
  priors learned by video generation models. By distilling these priors,
  PhysDreamer enables the synthesis of realistic object responses to novel
  interactions, such as external forces or agent manipulations. We demonstrate
  our approach on diverse examples of elastic objects and evaluate the realism of
  the synthesized interactions through a user study. PhysDreamer takes a step
  towards more engaging and realistic virtual experiences by enabling static 3D
  objects to dynamically respond to interactive stimuli in a physically plausible
  manner. See our project page at https://physdreamer.github.io/.
paper_ai_summary: PhysDreamer utilizes video generation models to simulate realistic 3D object dynamics based on learned action-conditioned priors, enhancing immersive virtual interactions.
paper_ai_keywords: object dynamics video generation models action-conditioned dynamics material properties motion prediction user study virtual interactions physically plausible