Open-World Reinforcement Learning over Long Short-Term Imagination
<|reference_start|>Open-World Reinforcement Learning over Long Short-Term Imagination: Training visual reinforcement learning agents in a high-dimensional open world presents significant challenges. While various model-based methods have improved sample efficiency by learning interactive world models, these agents tend to be "short-sighted", as they are typically trained on short snippets of imagined experiences. We argue that the primary obstacle in open-world decision-making is improving the efficiency of off-policy exploration across an extensive state space. In this paper, we present LS-Imagine, which extends the imagination horizon within a limited number of state transition steps, enabling the agent to explore behaviors that potentially lead to promising long-term feedback. The foundation of our approach is to build a long short-term world model. To achieve this, we simulate goal-conditioned jumpy state transitions and compute corresponding affordance maps by zooming in on specific areas within single images. This facilitates the integration of direct long-term values into behavior learning. Our method demonstrates significant improvements over state-of-the-art techniques in MineDojo.<|reference_end|>
@article{li2024open-world,
    title={Open-World Reinforcement Learning over Long Short-Term Imagination},
    author={Jiajian Li, Qi Wang, Yunbo Wang, Xin Jin, Yang Li, Wenjun Zeng,
  Xiaokang Yang},
    journal={arXiv preprint arXiv:2410.03618},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03618},
    primaryClass={cs.LG}
}