RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning
Developing robust and correctable visuomotor policies for robotic
manipulation is challenging due to the lack of self-recovery mechanisms from
failures and the limitations of simple language instructions in guiding robot
actions. To address these issues, we propose a scalable data generation
pipeline that automatically augments expert demonstrations with failure
recovery trajectories and fine-grained language annotations for training. We
then introduce Rich languAge-guided failure reCovERy (RACER), a
supervisor-actor framework, which combines failure recovery data with rich
language descriptions to enhance robot control. RACER features a
vision-language model (VLM) that acts as an online supervisor, providing
detailed language guidance for error correction and task execution, and a
language-conditioned visuomotor policy as an actor to predict the next actions.
Our experimental results show that RACER outperforms the state-of-the-art
Robotic View Transformer (RVT) on RLbench across various evaluation settings,
including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
unseen tasks, achieving superior performance in both simulated and real world
environments. Videos and code are available at:
https://rich-language-failure-recovery.github.io.
66f227620fa7f61205acaa41
Yinpei Dai
claimed_verified
2024-09-24T07:52:54.065Z
648b889575d21f74acf9cbb1
/avatars/0ffee0809f9e2670821721c005d8dde4.svg
Dai
user
Yinpei
66f227620fa7f61205acaa42
Jayjun Lee
admin_assigned
2024-09-24T10:19:27.756Z
62622f39aee4bea02143fe35
/avatars/ba50d75c1d591678347a108d16976c9e.svg
Jayjun Lee
user
newwater
66f227620fa7f61205acaa43
Nima Fazeli
admin_assigned
2024-09-24T10:19:34.083Z
66919fa1306a261f1850b27e
/avatars/34d7956986bef23d01c2bd4ebd4e9765.svg
Nima Fazeli
user
nimafazeli
66f227620fa7f61205acaa44
Joyce Chai
RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning
Developing robust and correctable visuomotor policies for robotic
manipulation is challenging due to the lack of self-recovery mechanisms from
failures and the limitations of simple language instructions in guiding robot
actions. To address these issues, we propose a scalable data generation
pipeline that automatically augments expert demonstrations with failure
recovery trajectories and fine-grained language annotations for training. We
then introduce Rich languAge-guided failure reCovERy (RACER), a
supervisor-actor framework, which combines failure recovery data with rich
language descriptions to enhance robot control. RACER features a
vision-language model (VLM) that acts as an online supervisor, providing
detailed language guidance for error correction and task execution, and a
language-conditioned visuomotor policy as an actor to predict the next actions.
Our experimental results show that RACER outperforms the state-of-the-art
Robotic View Transformer (RVT) on RLbench across various evaluation settings,
including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
unseen tasks, achieving superior performance in both simulated and real world
environments. Videos and code are available at:
https://rich-language-failure-recovery.github.io.
RACER, a supervisor-actor framework enhanced with failure recovery data and rich language descriptions, improves robotic manipulation by outperforming RVT in various tasks and environments.
visuomotor policies robotic manipulation failure recovery data generation pipeline Rich languAge-guided failure reCovERy (RACER) supervisor-actor framework vision-language model (VLM) language-conditioned visuomotor policy RLbench long-horizon tasks dynamic goal-change tasks zero-shot unseen tasks