Decomposing the Generalization Gap in Imitation Learning for Visual
  Robotic Manipulation
What makes generalization hard for imitation learning in visual robotic
manipulation? This question is difficult to approach at face value, but the
environment from the perspective of a robot can often be decomposed into
enumerable factors of variation, such as the lighting conditions or the
placement of the camera. Empirically, generalization to some of these factors
have presented a greater obstacle than others, but existing work sheds little
light on precisely how much each factor contributes to the generalization gap.
Towards an answer to this question, we study imitation learning policies in
simulation and on a real robot language-conditioned manipulation task to
quantify the difficulty of generalization to different (sets of) factors. We
also design a new simulated benchmark of 19 tasks with 11 factors of variation
to facilitate more controlled evaluations of generalization. From our study, we
determine an ordering of factors based on generalization difficulty, that is
consistent across simulation and our real robot setup.
64ab998b31f620201d04bcfc
Annie Xie
admin_assigned
2023-07-10T09:59:35.854Z
649e23bb3e5e75047640b3b5
/avatars/019d008a5bd7057b8665e1ef856f13a7.svg
Annie Xie
user
anniexie
64ab998b31f620201d04bcfd
Lisa Lee
64ab998b31f620201d04bcfe
Ted Xiao
64ab998b31f620201d04bcff
Chelsea Finn
claimed_verified
2023-07-11T07:54:07.806Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
Decomposing the Generalization Gap in Imitation Learning for Visual
  Robotic Manipulation
What makes generalization hard for imitation learning in visual robotic
manipulation? This question is difficult to approach at face value, but the
environment from the perspective of a robot can often be decomposed into
enumerable factors of variation, such as the lighting conditions or the
placement of the camera. Empirically, generalization to some of these factors
have presented a greater obstacle than others, but existing work sheds little
light on precisely how much each factor contributes to the generalization gap.
Towards an answer to this question, we study imitation learning policies in
simulation and on a real robot language-conditioned manipulation task to
quantify the difficulty of generalization to different (sets of) factors. We
also design a new simulated benchmark of 19 tasks with 11 factors of variation
to facilitate more controlled evaluations of generalization. From our study, we
determine an ordering of factors based on generalization difficulty, that is
consistent across simulation and our real robot setup.
The study investigates the generalization challenges in imitation learning for visual robotic manipulation by quantifying the impact of different factors of variation in both simulation and real-world settings.
imitation learning visual robotic manipulation factors of variation generalization difficulty