Never-Ending Behavior-Cloning Agent for Robotic Manipulation
<|reference_start|>Never-Ending Behavior-Cloning Agent for Robotic Manipulation: Relying on multi-modal observations, embodied robots could perform multiple robotic manipulation tasks in unstructured real-world environments. However, most language-conditioned behavior-cloning agents still face existing long-standing challenges, i.e., 3D scene representation and human-level task learning, when adapting into new sequential tasks in practical scenarios. We here investigate these above challenges with NBAgent in embodied robots, a pioneering language-conditioned Never-ending Behavior-cloning Agent. It can continually learn observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes, respectively. Specifically, we propose a skill-sharedsemantic rendering module and a skill-shared representation distillation module to effectively learn 3D scene semantics from skill-shared attribute, further tackling 3D scene representation overlooking. Meanwhile, we establish a skill-specific evolving planner to perform manipulation knowledge decoupling, which can continually embed novel skill-specific knowledge like human from latent and low-rank space. Finally, we design a never-ending embodied robot manipulation benchmark, and expensive experiments demonstrate the significant performance of our method. Visual results, code, and dataset are provided at: https://neragent.github.io.<|reference_end|>
@article{liang2024never-ending,
    title={Never-Ending Behavior-Cloning Agent for Robotic Manipulation},
    author={Wenqi Liang, Gan Sun, Qian He, Yu Ren, Jiahua Dong and Yang Cong},
    journal={arXiv preprint arXiv:2403.00336},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00336},
    primaryClass={cs.RO cs.AI}
}