Giving Robots a Hand: Learning Generalizable Manipulation with
  Eye-in-Hand Human Video Demonstrations
Eye-in-hand cameras have shown promise in enabling greater sample efficiency
and generalization in vision-based robotic manipulation. However, for robotic
imitation, it is still expensive to have a human teleoperator collect large
amounts of expert demonstrations with a real robot. Videos of humans performing
tasks, on the other hand, are much cheaper to collect since they eliminate the
need for expertise in robotic teleoperation and can be quickly captured in a
wide range of scenarios. Therefore, human video demonstrations are a promising
data source for learning generalizable robotic manipulation policies at scale.
In this work, we augment narrow robotic imitation datasets with broad unlabeled
human video demonstrations to greatly enhance the generalization of eye-in-hand
visuomotor policies. Although a clear visual domain gap exists between human
and robot data, our framework does not need to employ any explicit domain
adaptation method, as we leverage the partial observability of eye-in-hand
cameras as well as a simple fixed image masking scheme. On a suite of eight
real-world tasks involving both 3-DoF and 6-DoF robot arm control, our method
improves the success rates of eye-in-hand manipulation policies by 58%
(absolute) on average, enabling robots to generalize to both new environment
configurations and new tasks that are unseen in the robot demonstration data.
See video results at https://giving-robots-a-hand.github.io/ .
64af7c630d79759ee65225cf
Moo Jin Kim
claimed_verified
2023-07-17T06:48:38.258Z
64b017833ffa69983bb8ed37
https://cdn-avatars.huggingface.co/v1/production/uploads/64b017833ffa69983bb8ed37/gdaFjlaJDDPtfH84b75RU.jpeg
Moo Jin Kim
user
moojink
64af7c630d79759ee65225d0
Jiajun Wu
admin_assigned
2023-07-13T09:53:23.211Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
64af7c630d79759ee65225d1
Chelsea Finn
admin_assigned
2023-07-13T09:53:01.024Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
Giving Robots a Hand: Learning Generalizable Manipulation with
  Eye-in-Hand Human Video Demonstrations
Eye-in-hand cameras have shown promise in enabling greater sample efficiency
and generalization in vision-based robotic manipulation. However, for robotic
imitation, it is still expensive to have a human teleoperator collect large
amounts of expert demonstrations with a real robot. Videos of humans performing
tasks, on the other hand, are much cheaper to collect since they eliminate the
need for expertise in robotic teleoperation and can be quickly captured in a
wide range of scenarios. Therefore, human video demonstrations are a promising
data source for learning generalizable robotic manipulation policies at scale.
In this work, we augment narrow robotic imitation datasets with broad unlabeled
human video demonstrations to greatly enhance the generalization of eye-in-hand
visuomotor policies. Although a clear visual domain gap exists between human
and robot data, our framework does not need to employ any explicit domain
adaptation method, as we leverage the partial observability of eye-in-hand
cameras as well as a simple fixed image masking scheme. On a suite of eight
real-world tasks involving both 3-DoF and 6-DoF robot arm control, our method
improves the success rates of eye-in-hand manipulation policies by 58%
(absolute) on average, enabling robots to generalize to both new environment
configurations and new tasks that are unseen in the robot demonstration data.
See video results at https://giving-robots-a-hand.github.io/ .
Combining narrow robotic imitation datasets with broad human video demonstrations improves generalization of eye-in-hand visuomotor policies using partial observability and image masking.
eye-in-hand cameras robotic manipulation imitation learning human video demonstrations visual domain gap domain adaptation image masking 3-DoF 6-DoF visuomotor policies