title: Giving Robots a Hand: Learning Generalizable Manipulation with
    Eye-in-Hand Human Video Demonstrations
summary: Eye-in-hand cameras have shown promise in enabling greater sample efficiency
  and generalization in vision-based robotic manipulation. However, for robotic
  imitation, it is still expensive to have a human teleoperator collect large
  amounts of expert demonstrations with a real robot. Videos of humans performing
  tasks, on the other hand, are much cheaper to collect since they eliminate the
  need for expertise in robotic teleoperation and can be quickly captured in a
  wide range of scenarios. Therefore, human video demonstrations are a promising
  data source for learning generalizable robotic manipulation policies at scale.
  In this work, we augment narrow robotic imitation datasets with broad unlabeled
  human video demonstrations to greatly enhance the generalization of eye-in-hand
  visuomotor policies. Although a clear visual domain gap exists between human
  and robot data, our framework does not need to employ any explicit domain
  adaptation method, as we leverage the partial observability of eye-in-hand
  cameras as well as a simple fixed image masking scheme. On a suite of eight
  real-world tasks involving both 3-DoF and 6-DoF robot arm control, our method
  improves the success rates of eye-in-hand manipulation policies by 58%
  (absolute) on average, enabling robots to generalize to both new environment
  configurations and new tasks that are unseen in the robot demonstration data.
  See video results at https://giving-robots-a-hand.github.io/ .
paper_authors._id: 64af7c630d79759ee65225cf
paper_authors.name: Moo Jin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:48:38.258Z
paper_authors._id: 64b017833ffa69983bb8ed37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b017833ffa69983bb8ed37/gdaFjlaJDDPtfH84b75RU.jpeg
paper_authors.fullname: Moo Jin Kim
paper_authors.type: user
paper_authors.user: moojink
paper_authors._id: 64af7c630d79759ee65225d0
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:53:23.211Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 64af7c630d79759ee65225d1
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:53:01.024Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_title: Giving Robots a Hand: Learning Generalizable Manipulation with
    Eye-in-Hand Human Video Demonstrations
paper_summary: Eye-in-hand cameras have shown promise in enabling greater sample efficiency
  and generalization in vision-based robotic manipulation. However, for robotic
  imitation, it is still expensive to have a human teleoperator collect large
  amounts of expert demonstrations with a real robot. Videos of humans performing
  tasks, on the other hand, are much cheaper to collect since they eliminate the
  need for expertise in robotic teleoperation and can be quickly captured in a
  wide range of scenarios. Therefore, human video demonstrations are a promising
  data source for learning generalizable robotic manipulation policies at scale.
  In this work, we augment narrow robotic imitation datasets with broad unlabeled
  human video demonstrations to greatly enhance the generalization of eye-in-hand
  visuomotor policies. Although a clear visual domain gap exists between human
  and robot data, our framework does not need to employ any explicit domain
  adaptation method, as we leverage the partial observability of eye-in-hand
  cameras as well as a simple fixed image masking scheme. On a suite of eight
  real-world tasks involving both 3-DoF and 6-DoF robot arm control, our method
  improves the success rates of eye-in-hand manipulation policies by 58%
  (absolute) on average, enabling robots to generalize to both new environment
  configurations and new tasks that are unseen in the robot demonstration data.
  See video results at https://giving-robots-a-hand.github.io/ .
paper_ai_keywords: eye-in-hand cameras robotic manipulation robotic imitation teleoperator human video demonstrations narrow robotic imitation datasets unsupervised learning eye-in-hand cameras robotic manipulation robotic imitation human video demonstrations narrow robotic imitation datasets unsupervised learning visuomotor policies domain gap image masking scheme 3-DoF 6-DoF robot arm control success rates unseen tasks