Probabilistic Subgoal Representations for Hierarchical Reinforcement learning
<|reference_start|>Probabilistic Subgoal Representations for Hierarchical Reinforcement learning: In goal-conditioned hierarchical reinforcement learning (HRL), a high-level policy specifies a subgoal for the low-level policy to reach. Effective HRL hinges on a suitable subgoal represen tation function, abstracting state space into latent subgoal space and inducing varied low-level behaviors. Existing methods adopt a subgoal representation that provides a deterministic mapping from state space to latent subgoal space. Instead, this paper utilizes Gaussian Processes (GPs) for the first probabilistic subgoal representation. Our method employs a GP prior on the latent subgoal space to learn a posterior distribution over the subgoal representation functions while exploiting the long-range correlation in the state space through learnable kernels. This enables an adaptive memory that integrates long-range subgoal information from prior planning steps allowing to cope with stochastic uncertainties. Furthermore, we propose a novel learning objective to facilitate the simultaneous learning of probabilistic subgoal representations and policies within a unified framework. In experiments, our approach outperforms state-of-the-art baselines in standard benchmarks but also in environments with stochastic elements and under diverse reward conditions. Additionally, our model shows promising capabilities in transferring low-level policies across different tasks.<|reference_end|>
@article{wang2024probabilistic,
    title={Probabilistic Subgoal Representations for Hierarchical Reinforcement
  learning},
    author={Vivienne Huiling Wang, Tinghuai Wang, Wenyan Yang, Joni-Kristian
  K"am"ar"ainen, Joni Pajarinen},
    journal={arXiv preprint arXiv:2406.16707},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.16707},
    primaryClass={cs.LG cs.AI}
}