LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence
<|reference_start|>LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence: Due to the need to interact with the real world, embodied agents are required to possess comprehensive prior knowledge, long-horizon planning capability, and a swift response speed. Despite recent large language model (LLM) based agents achieving promising performance, they still exhibit several limitations. For instance, the output of LLMs is a descriptive sentence, which is ambiguous when determining specific actions. To address these limitations, we introduce the large auto-regressive model (LARM). LARM leverages both text and multi-view images as input and predicts subsequent actions in an auto-regressive manner. To train LARM, we develop a novel data format named auto-regressive node transmission structure and assemble a corresponding dataset. Adopting a two-phase training regimen, LARM successfully harvests enchanted equipment in Minecraft, which demands significantly more complex decision-making chains than the highest achievements of prior best methods. Besides, the speed of LARM is 6.8x faster.<|reference_end|>
@article{li2024larm:,
    title={LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence},
    author={Zhuoling Li, Xiaogang Xu, Zhenhua Xu, SerNam Lim, Hengshuang Zhao},
    journal={arXiv preprint arXiv:2405.17424},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17424},
    primaryClass={cs.CV}
}