Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning
<|reference_start|>Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning: Model-based methods in reinforcement learning offer a promising approach to enhance data efficiency by facilitating policy exploration within a dynamics model. However, accurately predicting sequential steps in the dynamics model remains a challenge due to the bootstrapping prediction, which attributes the next state to the prediction of the current state. This leads to accumulated errors during model roll-out. In this paper, we propose the Any-step Dynamics Model (ADM) to mitigate the compounding error by reducing bootstrapping prediction to direct prediction. ADM allows for the use of variable-length plans as inputs for predicting future states without frequent bootstrapping. We design two algorithms, ADMPO-ON and ADMPO-OFF, which apply ADM in online and offline model-based frameworks, respectively. In the online setting, ADMPO-ON demonstrates improved sample efficiency compared to previous state-of-the-art methods. In the offline setting, ADMPO-OFF not only demonstrates superior performance compared to recent state-of-the-art offline approaches but also offers better quantification of model uncertainty using only a single ADM.<|reference_end|>
@article{lin2024any-step,
    title={Any-step Dynamics Model Improves Future Predictions for Online and
  Offline Reinforcement Learning},
    author={Haoxin Lin, Yu-Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing
  Jia, Junyin Ye, Jiaji Zhang, Yang Yu},
    journal={arXiv preprint arXiv:2405.17031},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17031},
    primaryClass={cs.LG}
}