title: Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
summary: We present a framework for robot skill acquisition, which 1) efficiently
  scale up data generation of language-labelled robot data and 2) effectively
  distills this data down into a robust multi-task language-conditioned
  visuo-motor policy. For (1), we use a large language model (LLM) to guide
  high-level planning, and sampling-based robot planners (e.g. motion or grasp
  samplers) for generating diverse and rich manipulation trajectories. To
  robustify this data-collection process, the LLM also infers a code-snippet for
  the success condition of each task, simultaneously enabling the data-collection
  process to detect failure and retry as well as the automatic labeling of
  trajectories with success/failure. For (2), we extend the diffusion policy
  single-task behavior-cloning approach to multi-task settings with language
  conditioning. Finally, we propose a new multi-task benchmark with 18 tasks
  across five domains to test long-horizon behavior, common-sense reasoning,
  tool-use, and intuitive physics. We find that our distilled policy successfully
  learned the robust retrying behavior in its data collection policy, while
  improving absolute success rates by 34.8% on average across five domains. The
  benchmark, code, and qualitative results are on our website
  https://www.cs.columbia.edu/~huy/scalingup/
paper_authors._id: 64c316045b7862ce872fa6a7
paper_authors.name: Huy Ha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-28T16:17:00.967Z
paper_authors._id: 636a8b90c95145940bfdec8c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667926906903-noauth.jpeg
paper_authors.fullname: Huy Ha
paper_authors.type: user
paper_authors.user: huy-ha
paper_authors._id: 64c316045b7862ce872fa6a8
paper_authors.name: Pete Florence
paper_authors._id: 64c316045b7862ce872fa6a9
paper_authors.name: Shuran Song
paper_title: Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
paper_summary: We present a framework for robot skill acquisition, which 1) efficiently
  scale up data generation of language-labelled robot data and 2) effectively
  distills this data down into a robust multi-task language-conditioned
  visuo-motor policy. For (1), we use a large language model (LLM) to guide
  high-level planning, and sampling-based robot planners (e.g. motion or grasp
  samplers) for generating diverse and rich manipulation trajectories. To
  robustify this data-collection process, the LLM also infers a code-snippet for
  the success condition of each task, simultaneously enabling the data-collection
  process to detect failure and retry as well as the automatic labeling of
  trajectories with success/failure. For (2), we extend the diffusion policy
  single-task behavior-cloning approach to multi-task settings with language
  conditioning. Finally, we propose a new multi-task benchmark with 18 tasks
  across five domains to test long-horizon behavior, common-sense reasoning,
  tool-use, and intuitive physics. We find that our distilled policy successfully
  learned the robust retrying behavior in its data collection policy, while
  improving absolute success rates by 34.8% on average across five domains. The
  benchmark, code, and qualitative results are on our website
  https://www.cs.columbia.edu/~huy/scalingup/
paper_ai_summary: The framework scales up the generation of language-labeled robot data using a large language model and sampling-based planners, then distills it into a multi-task language-conditioned visuo-motor policy, achieving improved success rates on a new multi-task benchmark.
paper_ai_keywords: large language model sampling-based robot planners diffusion policy multi-task behavior-cloning multi-task benchmark language-conditioned visuo-motor policy