Tackling Long-Horizon Tasks with Model-based Offline Reinforcement Learning
<|reference_start|>Tackling Long-Horizon Tasks with Model-based Offline Reinforcement Learning: Model-based offline reinforcement learning (RL) is a compelling approach that addresses the challenge of learning from limited, static data by generating imaginary trajectories using learned models. However, it falls short in solving long-horizon tasks due to high bias in value estimation from model rollouts. In this paper, we introduce a novel model-based offline RL method, Lower Expectile Q-learning (LEQ), which enhances long-horizon task performance by mitigating the high bias in model-based value estimation via expectile regression of $\lambda$-returns. Our empirical results show that LEQ significantly outperforms previous model-based offline RL methods on long-horizon tasks, such as the D4RL AntMaze tasks, matching or surpassing the performance of model-free approaches. Our experiments demonstrate that expectile regression, $\lambda$-returns, and critic training on offline data are all crucial for addressing long-horizon tasks. Additionally, LEQ achieves performance comparable to the state-of-the-art model-based and model-free offline RL methods on the NeoRL benchmark and the D4RL MuJoCo Gym tasks.<|reference_end|>
@article{park2024tackling,
    title={Tackling Long-Horizon Tasks with Model-based Offline Reinforcement
  Learning},
    author={Kwanyoung Park and Youngwoon Lee},
    journal={arXiv preprint arXiv:2407.00699},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.00699},
    primaryClass={cs.LG cs.AI}
}