Synthesis of Hierarchical Controllers Based on Deep Reinforcement Learning Policies
<|reference_start|>Synthesis of Hierarchical Controllers Based on Deep Reinforcement Learning Policies: We propose a novel approach to the problem of controller design for environments modeled as Markov decision processes (MDPs). Specifically, we consider a hierarchical MDP a graph with each vertex populated by an MDP called a "room". We first apply deep reinforcement learning (DRL) to obtain low-level policies for each room, scaling to large rooms of unknown structure. We then apply reactive synthesis to obtain a high-level planner that chooses which low-level policy to execute in each room. The central challenge in synthesizing the planner is the need for modeling rooms. We address this challenge by developing a DRL procedure to train concise "latent" policies together with PAC guarantees on their performance. Unlike previous approaches, ours circumvents a model distillation step. Our approach combats sparse rewards in DRL and enables reusability of low-level policies. We demonstrate feasibility in a case study involving agent navigation amid moving obstacles.<|reference_end|>
@article{delgrange2024synthesis,
    title={Synthesis of Hierarchical Controllers Based on Deep Reinforcement
  Learning Policies},
    author={Florent Delgrange, Guy Avni, Anna Lukina, Christian Schilling, Ann
  Now'e, and Guillermo A. P'erez},
    journal={arXiv preprint arXiv:2402.13785},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.13785},
    primaryClass={cs.AI}
}