title: 3D Diffusion Policy
summary: Imitation learning provides an efficient way to teach robots dexterous
  skills; however, learning complex skills robustly and generalizablely usually
  consumes large amounts of human demonstrations. To tackle this challenging
  problem, we present 3D Diffusion Policy (DP3), a novel visual imitation
  learning approach that incorporates the power of 3D visual representations into
  diffusion policies, a class of conditional action generative models. The core
  design of DP3 is the utilization of a compact 3D visual representation,
  extracted from sparse point clouds with an efficient point encoder. In our
  experiments involving 72 simulation tasks, DP3 successfully handles most tasks
  with just 10 demonstrations and surpasses baselines with a 55.3% relative
  improvement. In 4 real robot tasks, DP3 demonstrates precise control with a
  high success rate of 85%, given only 40 demonstrations of each task, and shows
  excellent generalization abilities in diverse aspects, including space,
  viewpoint, appearance, and instance. Interestingly, in real robot experiments,
  DP3 rarely violates safety requirements, in contrast to baseline methods which
  frequently do, necessitating human intervention. Our extensive evaluation
  highlights the critical importance of 3D representations in real-world robot
  learning. Videos, code, and data are available on
  https://3d-diffusion-policy.github.io .
paper_authors._id: 65e9558e6434c9e6f4a30604
paper_authors.name: Yanjie Ze
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:13:40.115Z
paper_authors._id: 63509bc859bfa9a85d4220aa
paper_authors.avatarUrl: /avatars/ca2cc9b87f5ca5cd51606b2f9edf89d0.svg
paper_authors.fullname: Yanjie Ze
paper_authors.type: user
paper_authors.user: yjze
paper_authors._id: 65e9558e6434c9e6f4a30605
paper_authors.name: Gu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T09:13:37.901Z
paper_authors._id: 65e95effe19214e9d13d7a2a
paper_authors.avatarUrl: /avatars/35ae40579ac53d7b598957be9c8b4710.svg
paper_authors.fullname: Gu Zhang
paper_authors.type: user
paper_authors.user: GuZhang
paper_authors._id: 65e9558e6434c9e6f4a30606
paper_authors.name: Kangning Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T16:02:46.137Z
paper_authors._id: 63b6def76fca9d2a1902fa14
paper_authors.avatarUrl: /avatars/c7f2487450ea954e2bca4fc5a6db8eb3.svg
paper_authors.fullname: 张康宁
paper_authors.type: user
paper_authors.user: zhuiguang-ning
paper_authors._id: 65e9558e6434c9e6f4a30607
paper_authors.name: Chenyuan Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T09:49:28.551Z
paper_authors._id: 63ec3830ca08a72ba9d04b81
paper_authors.avatarUrl: /avatars/12245876937e12651dc794ca99f9109d.svg
paper_authors.fullname: chenyuan hu
paper_authors.type: user
paper_authors.user: hcy1223
paper_authors._id: 65e9558e6434c9e6f4a30608
paper_authors.name: Muhan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T09:49:34.997Z
paper_authors._id: 656492f7055bb4df5e8a3a49
paper_authors.avatarUrl: /avatars/b3b7e8e10853541a9bfa1ce7ce2217fd.svg
paper_authors.fullname: Muhan Wang
paper_authors.type: user
paper_authors.user: Sparklist
paper_authors._id: 65e9558e6434c9e6f4a30609
paper_authors.name: Huazhe Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-07T09:49:41.282Z
paper_authors._id: 649c130f92525b3ac9fbcffe
paper_authors.avatarUrl: /avatars/b60e4d6803f7e4ef42200ddcb9e5bac1.svg
paper_authors.fullname: Huazhe Xu
paper_authors.type: user
paper_authors.user: Huazhe-Xu
paper_title: 3D Diffusion Policy
paper_summary: Imitation learning provides an efficient way to teach robots dexterous
  skills; however, learning complex skills robustly and generalizablely usually
  consumes large amounts of human demonstrations. To tackle this challenging
  problem, we present 3D Diffusion Policy (DP3), a novel visual imitation
  learning approach that incorporates the power of 3D visual representations into
  diffusion policies, a class of conditional action generative models. The core
  design of DP3 is the utilization of a compact 3D visual representation,
  extracted from sparse point clouds with an efficient point encoder. In our
  experiments involving 72 simulation tasks, DP3 successfully handles most tasks
  with just 10 demonstrations and surpasses baselines with a 55.3% relative
  improvement. In 4 real robot tasks, DP3 demonstrates precise control with a
  high success rate of 85%, given only 40 demonstrations of each task, and shows
  excellent generalization abilities in diverse aspects, including space,
  viewpoint, appearance, and instance. Interestingly, in real robot experiments,
  DP3 rarely violates safety requirements, in contrast to baseline methods which
  frequently do, necessitating human intervention. Our extensive evaluation
  highlights the critical importance of 3D representations in real-world robot
  learning. Videos, code, and data are available on
  https://3d-diffusion-policy.github.io .
paper_ai_summary: 3D Diffusion Policy (DP3) utilizes 3D visual representations for efficient imitation learning, achieving robust and generalized skills with fewer demonstrations compared to existing methods.
paper_ai_keywords: 3D Diffusion Policy DP3 3D visual representations sparse point clouds point encoder diffusion policies conditional action generative models