title: RACER: Rich Language-Guided Failure Recovery Policies for Imitation
    Learning
summary: Developing robust and correctable visuomotor policies for robotic
  manipulation is challenging due to the lack of self-recovery mechanisms from
  failures and the limitations of simple language instructions in guiding robot
  actions. To address these issues, we propose a scalable data generation
  pipeline that automatically augments expert demonstrations with failure
  recovery trajectories and fine-grained language annotations for training. We
  then introduce Rich languAge-guided failure reCovERy (RACER), a
  supervisor-actor framework, which combines failure recovery data with rich
  language descriptions to enhance robot control. RACER features a
  vision-language model (VLM) that acts as an online supervisor, providing
  detailed language guidance for error correction and task execution, and a
  language-conditioned visuomotor policy as an actor to predict the next actions.
  Our experimental results show that RACER outperforms the state-of-the-art
  Robotic View Transformer (RVT) on RLbench across various evaluation settings,
  including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
  unseen tasks, achieving superior performance in both simulated and real world
  environments. Videos and code are available at:
  https://rich-language-failure-recovery.github.io.
paper_authors._id: 66f227620fa7f61205acaa41
paper_authors.name: Yinpei Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:54.065Z
paper_authors._id: 648b889575d21f74acf9cbb1
paper_authors.avatarUrl: /avatars/0ffee0809f9e2670821721c005d8dde4.svg
paper_authors.fullname: Dai
paper_authors.type: user
paper_authors.user: Yinpei
paper_authors._id: 66f227620fa7f61205acaa42
paper_authors.name: Jayjun Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-24T10:19:27.756Z
paper_authors._id: 62622f39aee4bea02143fe35
paper_authors.avatarUrl: /avatars/ba50d75c1d591678347a108d16976c9e.svg
paper_authors.fullname: Jayjun Lee
paper_authors.type: user
paper_authors.user: newwater
paper_authors._id: 66f227620fa7f61205acaa43
paper_authors.name: Nima Fazeli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-24T10:19:34.083Z
paper_authors._id: 66919fa1306a261f1850b27e
paper_authors.avatarUrl: /avatars/34d7956986bef23d01c2bd4ebd4e9765.svg
paper_authors.fullname: Nima Fazeli
paper_authors.type: user
paper_authors.user: nimafazeli
paper_authors._id: 66f227620fa7f61205acaa44
paper_authors.name: Joyce Chai
paper_title: RACER: Rich Language-Guided Failure Recovery Policies for Imitation
    Learning
paper_summary: Developing robust and correctable visuomotor policies for robotic
  manipulation is challenging due to the lack of self-recovery mechanisms from
  failures and the limitations of simple language instructions in guiding robot
  actions. To address these issues, we propose a scalable data generation
  pipeline that automatically augments expert demonstrations with failure
  recovery trajectories and fine-grained language annotations for training. We
  then introduce Rich languAge-guided failure reCovERy (RACER), a
  supervisor-actor framework, which combines failure recovery data with rich
  language descriptions to enhance robot control. RACER features a
  vision-language model (VLM) that acts as an online supervisor, providing
  detailed language guidance for error correction and task execution, and a
  language-conditioned visuomotor policy as an actor to predict the next actions.
  Our experimental results show that RACER outperforms the state-of-the-art
  Robotic View Transformer (RVT) on RLbench across various evaluation settings,
  including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
  unseen tasks, achieving superior performance in both simulated and real world
  environments. Videos and code are available at:
  https://rich-language-failure-recovery.github.io.
paper_ai_summary: RACER, a supervisor-actor framework enhanced with failure recovery data and rich language descriptions, improves robotic manipulation by outperforming RVT in various tasks and environments.
paper_ai_keywords: visuomotor policies robotic manipulation failure recovery data generation pipeline Rich languAge-guided failure reCovERy (RACER) supervisor-actor framework vision-language model (VLM) language-conditioned visuomotor policy RLbench long-horizon tasks dynamic goal-change tasks zero-shot unseen tasks