EPO: Hierarchical LLM Agents with Environment Preference Optimization
<|reference_start|>EPO: Hierarchical LLM Agents with Environment Preference Optimization: Long-horizon decision-making tasks present significant challenges for LLM-based agents due to the need for extensive planning over multiple steps. In this paper, we propose a hierarchical framework that decomposes complex tasks into manageable subgoals, utilizing separate LLMs for subgoal prediction and low-level action generation. To address the challenge of creating training signals for unannotated datasets, we develop a reward model that leverages multimodal environment feedback to automatically generate reward signals. We introduce Environment Preference Optimization (EPO), a novel method that generates preference signals from the environment's feedback and uses them to train LLM-based agents. Extensive experiments on ALFRED demonstrate the state-of-the-art performance of our framework, achieving first place on the ALFRED public leaderboard and showcasing its potential to improve long-horizon decision-making in diverse environments.<|reference_end|>
@article{zhao2024epo:,
    title={EPO: Hierarchical LLM Agents with Environment Preference Optimization},
    author={Qi Zhao, Haotian Fu, Chen Sun, George Konidaris},
    journal={arXiv preprint arXiv:2408.16090},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.16090},
    primaryClass={cs.LG}
}