RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning
<|reference_start|>RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning: Developing robust and correctable visuomotor policies for robotic manipulation is challenging due to the lack of self-recovery mechanisms from failures and the limitations of simple language instructions in guiding robot actions. To address these issues, we propose a scalable data generation pipeline that automatically augments expert demonstrations with failure recovery trajectories and fine-grained language annotations for training. We then introduce Rich languAge-guided failure reCovERy (RACER), a supervisor-actor framework, which combines failure recovery data with rich language descriptions to enhance robot control. RACER features a vision-language model (VLM) that acts as an online supervisor, providing detailed language guidance for error correction and task execution, and a language-conditioned visuomotor policy as an actor to predict the next actions. Our experimental results show that RACER outperforms the state-of-the-art Robotic View Transformer (RVT) on RLbench across various evaluation settings, including standard long-horizon tasks, dynamic goal-change tasks and zero-shot unseen tasks, achieving superior performance in both simulated and real world environments. Videos and code are available at: https://rich-language-failure-recovery.github.io.<|reference_end|>
@article{dai2024racer:,
    title={RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning},
    author={Yinpei Dai, Jayjun Lee, Nima Fazeli, Joyce Chai},
    journal={arXiv preprint arXiv:2409.14674},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14674},
    primaryClass={cs.RO cs.CL cs.CV}
}