FOSP: Fine-tuning Offline Safe Policy through World Models
<|reference_start|>FOSP: Fine-tuning Offline Safe Policy through World Models: Model-based Reinforcement Learning (RL) has shown its high training efficiency and capability of handling high-dimensional tasks. Regarding safety issues, safe model-based RL can achieve nearly zero-cost performance and effectively manage the trade-off between performance and safety. Nevertheless, prior works still pose safety challenges due to the online exploration in real-world deployment. To address this, some offline RL methods have emerged as solutions, which learn from a static dataset in a safe way by avoiding interactions with the environment. In this paper, we aim to further enhance safety during the deployment stage for vision-based robotic tasks by fine-tuning an offline-trained policy. We incorporate in-sample optimization, model-based policy expansion, and reachability guidance to construct a safe offline-to-online framework. Moreover, our method proves to improve the generalization of offline policy in unseen safety-constrained scenarios. Finally, the efficiency of our method is validated on simulation benchmarks with five vision-only tasks and a real robot by solving some deployment problems using limited data.<|reference_end|>
@article{cao2024fosp:,
    title={FOSP: Fine-tuning Offline Safe Policy through World Models},
    author={Chenyang Cao, Yucheng Xin, Silang Wu, Longxiang He, Zichen Yan, Junbo
  Tan, Xueqian Wang},
    journal={arXiv preprint arXiv:2407.04942},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.04942},
    primaryClass={cs.RO cs.LG}
}