title: GaussianProperty: Integrating Physical Properties to 3D Gaussians with
    LMMs
summary: Estimating physical properties for visual data is a crucial task in computer
  vision, graphics, and robotics, underpinning applications such as augmented
  reality, physical simulation, and robotic grasping. However, this area remains
  under-explored due to the inherent ambiguities in physical property estimation.
  To address these challenges, we introduce GaussianProperty, a training-free
  framework that assigns physical properties of materials to 3D Gaussians.
  Specifically, we integrate the segmentation capability of SAM with the
  recognition capability of GPT-4V(ision) to formulate a global-local physical
  property reasoning module for 2D images. Then we project the physical
  properties from multi-view 2D images to 3D Gaussians using a voting strategy.
  We demonstrate that 3D Gaussians with physical property annotations enable
  applications in physics-based dynamic simulation and robotic grasping. For
  physics-based dynamic simulation, we leverage the Material Point Method (MPM)
  for realistic dynamic simulation. For robot grasping, we develop a grasping
  force prediction strategy that estimates a safe force range required for object
  grasping based on the estimated physical properties. Extensive experiments on
  material segmentation, physics-based dynamic simulation, and robotic grasping
  validate the effectiveness of our proposed method, highlighting its crucial
  role in understanding physical properties from visual data. Online demo, code,
  more cases and annotated datasets are available on
  https://Gaussian-Property.github.io{this https URL}.
paper_authors._id: 6760e4c35b8363d75f3e7241
paper_authors.name: Xinli Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:03:58.140Z
paper_authors._id: 64b4ab62eec33e27dcd733b5
paper_authors.avatarUrl: /avatars/0a9bf220c9a5efe7279f9b287b087d36.svg
paper_authors.fullname: Xinli XU
paper_authors.type: user
paper_authors.user: Xxlbigbrother
paper_authors._id: 6760e4c35b8363d75f3e7242
paper_authors.name: Wenhang Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:17:45.208Z
paper_authors._id: 64758ee7d815855e4efa206b
paper_authors.avatarUrl: /avatars/105ada08a9a982fc7b723bdc678f7e72.svg
paper_authors.fullname: wenhang ge
paper_authors.type: user
paper_authors.user: spongy
paper_authors._id: 6760e4c35b8363d75f3e7243
paper_authors.name: Dicong Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:36:46.537Z
paper_authors._id: 64c1f2bbc84cd12dd7a5468c
paper_authors.avatarUrl: /avatars/d0a9fe33e5913b1304b8530b2c43f8bc.svg
paper_authors.fullname: David Qiu
paper_authors.type: user
paper_authors.user: davidqiu1993
paper_authors._id: 6760e4c35b8363d75f3e7244
paper_authors.name: ZhiFei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:17:31.856Z
paper_authors._id: 634a91c53a0cd2d4985dc98a
paper_authors.avatarUrl: /avatars/fe680e683761c347cfa4bc9273b67b0d.svg
paper_authors.fullname: ZhiFei Chen
paper_authors.type: user
paper_authors.user: zhifeichen097
paper_authors._id: 6760e4c35b8363d75f3e7245
paper_authors.name: Dongyu Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:17:25.961Z
paper_authors._id: 64049ae20ab5e22719f35103
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678023295407-noauth.jpeg
paper_authors.fullname: Dongyu Yan
paper_authors.type: user
paper_authors.user: StarYDY
paper_authors._id: 6760e4c35b8363d75f3e7246
paper_authors.name: Zhuoyun Liu
paper_authors._id: 6760e4c35b8363d75f3e7247
paper_authors.name: Haoyu Zhao
paper_authors._id: 6760e4c35b8363d75f3e7248
paper_authors.name: Hanfeng Zhao
paper_authors._id: 6760e4c35b8363d75f3e7249
paper_authors.name: Shunsi Zhang
paper_authors._id: 6760e4c35b8363d75f3e724a
paper_authors.name: Junwei Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:16:24.900Z
paper_authors._id: 656eccea896aeb1168a2bade
paper_authors.avatarUrl: /avatars/83298bc28c9a5de3d600b5d889c1dd82.svg
paper_authors.fullname: Junwei Liang
paper_authors.type: user
paper_authors.user: junweiliang
paper_authors._id: 6760e4c35b8363d75f3e724b
paper_authors.name: Ying-Cong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T11:16:19.064Z
paper_authors._id: 655cba1d87b67834000590e8
paper_authors.avatarUrl: /avatars/3bd43b7c9351f65b8f38f4c8237a0146.svg
paper_authors.fullname: Yingcong Chen
paper_authors.type: user
paper_authors.user: yingcongchen
paper_title: GaussianProperty: Integrating Physical Properties to 3D Gaussians with
    LMMs
paper_summary: Estimating physical properties for visual data is a crucial task in computer
  vision, graphics, and robotics, underpinning applications such as augmented
  reality, physical simulation, and robotic grasping. However, this area remains
  under-explored due to the inherent ambiguities in physical property estimation.
  To address these challenges, we introduce GaussianProperty, a training-free
  framework that assigns physical properties of materials to 3D Gaussians.
  Specifically, we integrate the segmentation capability of SAM with the
  recognition capability of GPT-4V(ision) to formulate a global-local physical
  property reasoning module for 2D images. Then we project the physical
  properties from multi-view 2D images to 3D Gaussians using a voting strategy.
  We demonstrate that 3D Gaussians with physical property annotations enable
  applications in physics-based dynamic simulation and robotic grasping. For
  physics-based dynamic simulation, we leverage the Material Point Method (MPM)
  for realistic dynamic simulation. For robot grasping, we develop a grasping
  force prediction strategy that estimates a safe force range required for object
  grasping based on the estimated physical properties. Extensive experiments on
  material segmentation, physics-based dynamic simulation, and robotic grasping
  validate the effectiveness of our proposed method, highlighting its crucial
  role in understanding physical properties from visual data. Online demo, code,
  more cases and annotated datasets are available on
  https://Gaussian-Property.github.io{this https URL}.
paper_ai_summary: GaussianProperty is a training-free framework that assigns physical properties to 3D Gaussians using SAM and GPT-4V, enabling applications in physics-based dynamic simulation and robotic grasping.
paper_ai_keywords: GaussianProperty SAM GPT-4V 3D Gaussians physics-based dynamic simulation Material Point Method (MPM) robotic grasping grasping force prediction