Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning
<|reference_start|>Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning: Offline reinforcement learning learns from a static dataset without interacting with the environment, which ensures security and thus owns a good prospect of application. However, directly applying naive reinforcement learning methods usually fails in an offline environment due to function approximation errors caused by out-of-distribution(OOD) actions. To solve this problem, existing algorithms mainly penalize the Q-value of OOD actions, the quality of whose constraints also matter. Imprecise constraints may lead to suboptimal solutions, while precise constraints require significant computational costs. In this paper, we propose a novel count-based method for continuous domains, called Grid-Mapping Pseudo-Count method(GPC), to penalize the Q-value appropriately and reduce the computational cost. The proposed method maps the state and action space to discrete space and constrains their Q-values through the pseudo-count. It is theoretically proved that only a few conditions are needed to obtain accurate uncertainty constraints in the proposed method. Moreover, we develop a Grid-Mapping Pseudo-Count Soft Actor-Critic(GPC-SAC) algorithm using GPC under the Soft Actor-Critic(SAC) framework to demonstrate the effectiveness of GPC. The experimental results on D4RL benchmark datasets show that GPC-SAC has better performance and less computational cost compared to other algorithms.<|reference_end|>
@article{shen2024grid-mapping,
    title={Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning},
    author={Yi Shen, Hanyan Huang},
    journal={arXiv preprint arXiv:2404.02545},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.02545},
    primaryClass={cs.LG cs.AI}
}