Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal
  Rearrangement
We propose a system for rearranging objects in a scene to achieve a desired
object-scene placing relationship, such as a book inserted in an open slot of a
bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of
both scenes and objects, and is trained from demonstrations to operate directly
on 3D point clouds. Our system overcomes challenges associated with the
existence of many geometrically-similar rearrangement solutions for a given
scene. By leveraging an iterative pose de-noising training procedure, we can
fit multi-modal demonstration data and produce multi-modal outputs while
remaining precise and accurate. We also show the advantages of conditioning on
relevant local geometric features while ignoring irrelevant global structure
that harms both generalization and precision. We demonstrate our approach on
three distinct rearrangement tasks that require handling multi-modality and
generalization over object shape and pose in both simulation and the real
world. Project website, code, and videos:
https://anthonysimeonov.github.io/rpdiff-multi-modal/
64acd65c0078182535e73448
Anthony Simeonov
64acd65c0078182535e73449
Ankit Goyal
claimed_verified
2024-06-15T07:46:54.737Z
666c6f67bc840e6748554a91
/avatars/72121715e14f720e5c1d029b7f00d55d.svg
Ankit Goyal
user
ankgoyal
64acd65c0078182535e7344a
Lucas Manuelli
claimed_verified
2023-07-12T07:29:11.525Z
64ad76bc0d0826bdd8a32221
/avatars/571f75f2f5710d11b2d44af8c240dd13.svg
Lucas Manuelli
user
lmanuelli
64acd65c0078182535e7344b
Lin Yen-Chen
64acd65c0078182535e7344c
Alina Sarmiento
64acd65c0078182535e7344d
Alberto Rodriguez
64acd65c0078182535e7344e
Pulkit Agrawal
64acd65c0078182535e7344f
Dieter Fox
Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal
  Rearrangement
We propose a system for rearranging objects in a scene to achieve a desired
object-scene placing relationship, such as a book inserted in an open slot of a
bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of
both scenes and objects, and is trained from demonstrations to operate directly
on 3D point clouds. Our system overcomes challenges associated with the
existence of many geometrically-similar rearrangement solutions for a given
scene. By leveraging an iterative pose de-noising training procedure, we can
fit multi-modal demonstration data and produce multi-modal outputs while
remaining precise and accurate. We also show the advantages of conditioning on
relevant local geometric features while ignoring irrelevant global structure
that harms both generalization and precision. We demonstrate our approach on
three distinct rearrangement tasks that require handling multi-modality and
generalization over object shape and pose in both simulation and the real
world. Project website, code, and videos:
https://anthonysimeonov.github.io/rpdiff-multi-modal/
The proposed system, trained from demonstrations, rearranges objects in 3D scenes by iteratively refining poses and focusing on relevant local geometric features to handle multi-modality and generalization.
3D point clouds iterative pose de-noising multi-modal demonstration data multi-modal outputs generalization local geometric features global structure simulation real world