Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity
<|reference_start|>Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity: We present an embodied AI system which receives open-ended natural language instructions from a human, and controls two arms to collaboratively accomplish potentially long-horizon tasks over a large workspace. Our system is modular: it deploys state of the art Large Language Models for task planning,Vision-Language models for semantic perception, and Point Cloud transformers for grasping. With semantic and physical safety in mind, these modules are interfaced with a real-time trajectory optimizer and a compliant tracking controller to enable human-robot proximity. We demonstrate performance for the following tasks: bi-arm sorting, bottle opening, and trash disposal tasks. These are done zero-shot where the models used have not been trained with any real world data from this bi-arm robot, scenes or workspace.Composing both learning- and non-learning-based components in a modular fashion with interpretable inputs and outputs allows the user to easily debug points of failures and fragilities. One may also in-place swap modules to improve the robustness of the overall platform, for instance with imitation-learned policies.<|reference_end|>
@article{varley2024embodied,
    title={Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity},
    author={Jake Varley, Sumeet Singh, Deepali Jain, Krzysztof Choromanski, Andy
  Zeng, Somnath Basu Roy Chowdhury, Avinava Dubey, Vikas Sindhwani},
    journal={arXiv preprint arXiv:2404.03570},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.03570},
    primaryClass={cs.RO}
}