Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model
<|reference_start|>Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model: With the power of large language models (LLMs), open-ended embodied agents can flexibly understand human instructions, generate interpretable guidance strategies, and output executable actions. Nowadays, Multi-modal Language Models~(MLMs) integrate multi-modal signals into LLMs, further bringing richer perception to entity agents and allowing embodied agents to perceive world-understanding tasks more delicately. However, existing works: 1) operate independently by agents, each containing multiple LLMs, from perception to action, resulting in gaps between complex tasks and execution; 2) train MLMs on static data, struggling with dynamics in open-ended scenarios; 3) input prior knowledge directly as prompts, suppressing application flexibility. We propose STEVE-2, a hierarchical knowledge distillation framework for open-ended embodied tasks, characterized by 1) a hierarchical system for multi-granular task division, 2) a mirrored distillation method for parallel simulation data, and 3) an extra expert model for bringing additional knowledge into parallel simulation. After distillation, embodied agents can complete complex, open-ended tasks without additional expert guidance, utilizing the performance and knowledge of a versatile MLM. Extensive evaluations on navigation and creation tasks highlight the superior performance of STEVE-2 in open-ended tasks, with $1.4 \times$ - $7.3 \times$ in performance.<|reference_end|>
@article{zhao2024do,
    title={Do We Really Need a Complex Agent System? Distill Embodied Agent into a
  Single Model},
    author={Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo,
  Yanting Zhang, Hongwei Wang and Gaoang Wang},
    journal={arXiv preprint arXiv:2404.04619},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.04619},
    primaryClass={cs.AI cs.CV}
}