title: PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable
    Objects from Videos
summary: Creating a physical digital twin of a real-world object has immense potential
  in robotics, content creation, and XR. In this paper, we present PhysTwin, a
  novel framework that uses sparse videos of dynamic objects under interaction to
  produce a photo- and physically realistic, real-time interactive virtual
  replica. Our approach centers on two key components: (1) a physics-informed
  representation that combines spring-mass models for realistic physical
  simulation, generative shape models for geometry, and Gaussian splats for
  rendering; and (2) a novel multi-stage, optimization-based inverse modeling
  framework that reconstructs complete geometry, infers dense physical
  properties, and replicates realistic appearance from videos. Our method
  integrates an inverse physics framework with visual perception cues, enabling
  high-fidelity reconstruction even from partial, occluded, and limited
  viewpoints. PhysTwin supports modeling various deformable objects, including
  ropes, stuffed animals, cloth, and delivery packages. Experiments show that
  PhysTwin outperforms competing methods in reconstruction, rendering, future
  prediction, and simulation under novel interactions. We further demonstrate its
  applications in interactive real-time simulation and model-based robotic motion
  planning.
paper_authors._id: 67e35159b3f0452257728c31
paper_authors.name: Hanxiao Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:44:50.816Z
paper_authors._id: 63f2bb987ddf724fbcc57ae9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f2bb987ddf724fbcc57ae9/j81g8dBAMcJchjjsRQSrP.jpeg
paper_authors.fullname: Hanxiao Jiang
paper_authors.type: user
paper_authors.user: Jianghanxiao
paper_authors._id: 67e35159b3f0452257728c32
paper_authors.name: Hao-Yu Hsu
paper_authors._id: 67e35159b3f0452257728c33
paper_authors.name: Kaifeng Zhang
paper_authors._id: 67e35159b3f0452257728c34
paper_authors.name: Hsin-Ni Yu
paper_authors._id: 67e35159b3f0452257728c35
paper_authors.name: Shenlong Wang
paper_authors._id: 67e35159b3f0452257728c36
paper_authors.name: Yunzhu Li
paper_title: PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable
    Objects from Videos
paper_summary: Creating a physical digital twin of a real-world object has immense potential
  in robotics, content creation, and XR. In this paper, we present PhysTwin, a
  novel framework that uses sparse videos of dynamic objects under interaction to
  produce a photo- and physically realistic, real-time interactive virtual
  replica. Our approach centers on two key components: (1) a physics-informed
  representation that combines spring-mass models for realistic physical
  simulation, generative shape models for geometry, and Gaussian splats for
  rendering; and (2) a novel multi-stage, optimization-based inverse modeling
  framework that reconstructs complete geometry, infers dense physical
  properties, and replicates realistic appearance from videos. Our method
  integrates an inverse physics framework with visual perception cues, enabling
  high-fidelity reconstruction even from partial, occluded, and limited
  viewpoints. PhysTwin supports modeling various deformable objects, including
  ropes, stuffed animals, cloth, and delivery packages. Experiments show that
  PhysTwin outperforms competing methods in reconstruction, rendering, future
  prediction, and simulation under novel interactions. We further demonstrate its
  applications in interactive real-time simulation and model-based robotic motion
  planning.
paper_ai_summary: PhysTwin is a framework using physics-informed and generative models to create a realistic digital twin from sparse video, supporting various deformable objects and real-time simulation.
paper_ai_keywords: physics-informed representation spring-mass models generative shape models Gaussian splats inverse modeling framework inverse physics framework visual perception cues deformable objects real-time simulation robotic motion planning