title: Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling
summary: Videos of robots interacting with objects encode rich information about the
  objects' dynamics. However, existing video prediction approaches typically do
  not explicitly account for the 3D information from videos, such as robot
  actions and objects' 3D states, limiting their use in real-world robotic
  applications. In this work, we introduce a framework to learn object dynamics
  directly from multi-view RGB videos by explicitly considering the robot's
  action trajectories and their effects on scene dynamics. We utilize the 3D
  Gaussian representation of 3D Gaussian Splatting (3DGS) to train a
  particle-based dynamics model using Graph Neural Networks. This model operates
  on sparse control particles downsampled from the densely tracked 3D Gaussian
  reconstructions. By learning the neural dynamics model on offline robot
  interaction data, our method can predict object motions under varying initial
  configurations and unseen robot actions. The 3D transformations of Gaussians
  can be interpolated from the motions of control particles, enabling the
  rendering of predicted future object states and achieving action-conditioned
  video prediction. The dynamics model can also be applied to model-based
  planning frameworks for object manipulation tasks. We conduct experiments on
  various kinds of deformable materials, including ropes, clothes, and stuffed
  animals, demonstrating our framework's ability to model complex shapes and
  dynamics. Our project page is available at https://gs-dynamics.github.io.
paper_authors._id: 671c6ae3a7c9769d4c7f7b55
paper_authors.name: Mingtong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:36:59.663Z
paper_authors._id: 671c6a3e255aa50ebb504fc5
paper_authors.avatarUrl: /avatars/b549bb19990ae21690472799817f951e.svg
paper_authors.fullname: Mingtong Zhang
paper_authors.type: user
paper_authors.user: Mingtongz
paper_authors._id: 671c6ae3a7c9769d4c7f7b56
paper_authors.name: Kaifeng Zhang
paper_authors._id: 671c6ae3a7c9769d4c7f7b57
paper_authors.name: Yunzhu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:55:25.973Z
paper_authors._id: 64a46562c641afb468a3dc04
paper_authors.avatarUrl: /avatars/bc54e5c13f657251b05bf2725086e56d.svg
paper_authors.fullname: Yunzhu Li
paper_authors.type: user
paper_authors.user: yunzhuli
paper_title: Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling
paper_summary: Videos of robots interacting with objects encode rich information about the
  objects' dynamics. However, existing video prediction approaches typically do
  not explicitly account for the 3D information from videos, such as robot
  actions and objects' 3D states, limiting their use in real-world robotic
  applications. In this work, we introduce a framework to learn object dynamics
  directly from multi-view RGB videos by explicitly considering the robot's
  action trajectories and their effects on scene dynamics. We utilize the 3D
  Gaussian representation of 3D Gaussian Splatting (3DGS) to train a
  particle-based dynamics model using Graph Neural Networks. This model operates
  on sparse control particles downsampled from the densely tracked 3D Gaussian
  reconstructions. By learning the neural dynamics model on offline robot
  interaction data, our method can predict object motions under varying initial
  configurations and unseen robot actions. The 3D transformations of Gaussians
  can be interpolated from the motions of control particles, enabling the
  rendering of predicted future object states and achieving action-conditioned
  video prediction. The dynamics model can also be applied to model-based
  planning frameworks for object manipulation tasks. We conduct experiments on
  various kinds of deformable materials, including ropes, clothes, and stuffed
  animals, demonstrating our framework's ability to model complex shapes and
  dynamics. Our project page is available at https://gs-dynamics.github.io.
paper_ai_summary: The proposed framework uses 3D Gaussian Splatting and Graph Neural Networks to predict object dynamics from multi-view RGB videos, facilitating action-conditioned video prediction and model-based planning for complex object manipulation tasks.
paper_ai_keywords: 3D Gaussian representation 3D Gaussian Splatting particle-based dynamics model Graph Neural Networks control particles action-conditioned video prediction 3D transformations model-based planning deformable materials