title: Robot See Robot Do: Imitating Articulated Object Manipulation with
    Monocular 4D Reconstruction
summary: Humans can learn to manipulate new objects by simply watching others;
  providing robots with the ability to learn from such demonstrations would
  enable a natural interface specifying new behaviors. This work develops Robot
  See Robot Do (RSRD), a method for imitating articulated object manipulation
  from a single monocular RGB human demonstration given a single static
  multi-view object scan. We first propose 4D Differentiable Part Models
  (4D-DPM), a method for recovering 3D part motion from a monocular video with
  differentiable rendering. This analysis-by-synthesis approach uses part-centric
  feature fields in an iterative optimization which enables the use of geometric
  regularizers to recover 3D motions from only a single video. Given this 4D
  reconstruction, the robot replicates object trajectories by planning bimanual
  arm motions that induce the demonstrated object part motion. By representing
  demonstrations as part-centric trajectories, RSRD focuses on replicating the
  demonstration's intended behavior while considering the robot's own
  morphological limits, rather than attempting to reproduce the hand's motion. We
  evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part
  trajectories and RSRD's physical execution performance on 9 objects across 10
  trials each on a bimanual YuMi robot. Each phase of RSRD achieves an average of
  87% success rate, for a total end-to-end success rate of 60% across 90 trials.
  Notably, this is accomplished using only feature fields distilled from large
  pretrained vision models -- without any task-specific training, fine-tuning,
  dataset collection, or annotation. Project page:
  https://robot-see-robot-do.github.io
paper_authors._id: 66f6277ce5d2d88a93bc9331
paper_authors.name: Justin Kerr
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T07:41:32.308Z
paper_authors._id: 63ec69436ef3ce22b88b636d
paper_authors.avatarUrl: /avatars/83436c44a4650c8c81da769e024409a7.svg
paper_authors.fullname: Justin Kerr
paper_authors.type: user
paper_authors.user: wisepigeon
paper_authors._id: 66f6277ce5d2d88a93bc9332
paper_authors.name: Chung Min Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T13:11:50.069Z
paper_authors._id: 6348a53762c668c7b48bd796
paper_authors.avatarUrl: /avatars/a60214a0150b9c7047c6eaf482bc195b.svg
paper_authors.fullname: Chung Min Kim
paper_authors.type: user
paper_authors.user: chungmin99
paper_authors._id: 66f6277ce5d2d88a93bc9333
paper_authors.name: Mingxuan Wu
paper_authors._id: 66f6277ce5d2d88a93bc9334
paper_authors.name: Brent Yi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T13:11:35.799Z
paper_authors._id: 63e4519e54f51ea342d231f1
paper_authors.avatarUrl: /avatars/d1c63cb6a28cb8af753daa6e6bb771cd.svg
paper_authors.fullname: Brent Yi
paper_authors.type: user
paper_authors.user: brentyi
paper_authors._id: 66f6277ce5d2d88a93bc9335
paper_authors.name: Qianqian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T13:11:30.491Z
paper_authors._id: 6616f0c4c2e30710e607c2bf
paper_authors.avatarUrl: /avatars/a5941e0ed940439f4c7c67747318cbfc.svg
paper_authors.fullname: Qianqian Wang
paper_authors.type: user
paper_authors.user: qianqian68
paper_authors._id: 66f6277ce5d2d88a93bc9336
paper_authors.name: Ken Goldberg
paper_authors._id: 66f6277ce5d2d88a93bc9337
paper_authors.name: Angjoo Kanazawa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T13:11:16.275Z
paper_authors._id: 6478cf7150ff7001631679c3
paper_authors.avatarUrl: /avatars/65ec385a9cc44c972e6caf952e759ff1.svg
paper_authors.fullname: Angjoo Kanazawa
paper_authors.type: user
paper_authors.user: akanazawa
paper_title: Robot See Robot Do: Imitating Articulated Object Manipulation with
    Monocular 4D Reconstruction
paper_summary: Humans can learn to manipulate new objects by simply watching others;
  providing robots with the ability to learn from such demonstrations would
  enable a natural interface specifying new behaviors. This work develops Robot
  See Robot Do (RSRD), a method for imitating articulated object manipulation
  from a single monocular RGB human demonstration given a single static
  multi-view object scan. We first propose 4D Differentiable Part Models
  (4D-DPM), a method for recovering 3D part motion from a monocular video with
  differentiable rendering. This analysis-by-synthesis approach uses part-centric
  feature fields in an iterative optimization which enables the use of geometric
  regularizers to recover 3D motions from only a single video. Given this 4D
  reconstruction, the robot replicates object trajectories by planning bimanual
  arm motions that induce the demonstrated object part motion. By representing
  demonstrations as part-centric trajectories, RSRD focuses on replicating the
  demonstration's intended behavior while considering the robot's own
  morphological limits, rather than attempting to reproduce the hand's motion. We
  evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part
  trajectories and RSRD's physical execution performance on 9 objects across 10
  trials each on a bimanual YuMi robot. Each phase of RSRD achieves an average of
  87% success rate, for a total end-to-end success rate of 60% across 90 trials.
  Notably, this is accomplished using only feature fields distilled from large
  pretrained vision models -- without any task-specific training, fine-tuning,
  dataset collection, or annotation. Project page:
  https://robot-see-robot-do.github.io
paper_ai_keywords: 4D Differentiable Parts Models (4D-DPM) differentiable rendering semantic clinical feature fields iterative optimization geometric regularizers 3D motion recovery bimanual arm motions part-centric feature fields part-centric trajectories Internet-scale latency feature fields large pretrained vision models task-specific training fine-tuning dataset collection annotation YuMi robot