Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
<|reference_start|>Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation: Despite significant progress in robotics and embodied AI in recent years, deploying robots for long-horizon tasks remains a great challenge. Majority of prior arts adhere to an open-loop philosophy and lack real-time feedback, leading to error accumulation and undesirable robustness. A handful of approaches have endeavored to establish feedback mechanisms leveraging pixel-level differences or pre-trained visual representations, yet their efficacy and adaptability have been found to be constrained. Inspired by classic closed-loop control systems, we propose CLOVER, a closed-loop visuomotor control framework that incorporates feedback mechanisms to improve adaptive robotic control. CLOVER consists of a text-conditioned video diffusion model for generating visual plans as reference inputs, a measurable embedding space for accurate error quantification, and a feedback-driven controller that refines actions from feedback and initiates replans as needed. Our framework exhibits notable advancement in real-world robotic tasks and achieves state-of-the-art on CALVIN benchmark, improving by 8% over previous open-loop counterparts. Code and checkpoints are maintained at https://github.com/OpenDriveLab/CLOVER.<|reference_end|>
@article{bu2024closed-loop,
    title={Closed-Loop Visuomotor Control with Generative Expectation for Robotic
  Manipulation},
    author={Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan,
  Ping Luo, Heming Cui, Yi Ma, Hongyang Li},
    journal={arXiv preprint arXiv:2409.09016},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09016},
    primaryClass={cs.RO}
}