Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks
Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level. We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos https://sites.google.com/view/hide-rl
The submission proposes a novel method for explicit decomposition of hierarchical policies for long-horizon navigation tasks. The approach proposes to separate a policy into 3 modules, high-level planner, intermediate planner and low-level control. The evaluation shows that explicit decomposition is well suited for generalisation across a limited set of RL domains. The proposed method integrates aspects from a variety of recent work including planning layers from value propagation networks, hindsight training paradigms from hierarchical actor critic and hindsight experience replay and related techniques. The variety of different techniques combined instead of a single main contribution renders it challenging to follow all aspects and in particular to trace relevant contributions to performance - which is rendered harder by a limited evaluation section. While the approach shows good performance against a couple of start of the art methods, it is necessary to provide sufficient ablations to enable long-term insights for the community. The submission high level goal (explicit decomposition and information asymmetry) is clear, the execution involves the combination of many existing techniques plus variations such that it is hard to make solid statements about the relevance of any part. It is commendable that the authors have introduced adaptations and improvements to their baselines for a stronger and fairer comparison but the evaluation remains very limited. I suggest to run different domains as given by other domains from OpenAI gym or DeepMind control suite. But more importantly I suggest to run further ablations without the intermediate planning layer & with absolute goal positions. Furthermore, since HAC seems to perform worse when combined with the proposed low and mid level policies (RelHAC), it would make sense to compare to the proposed high-level policy using low and mid level from HAC instead. The submission provides an overall interesting perspective but makes it hard to narrow down on contribution and important insights by being unclear in formulation and providing only very limited ablations. Minor issues include: - Missing description of the mid level policy - what encourages the proposal of closer short-term goals. - Missing literature on information asymmetry in RL (e.g. see Tirumala et al 2019 ‘Exploiting Hierarchy for Learning and Transfer in KL-regularized RL’) - Unclear description of how models that get attached to existing planning layers have been trained (Sec 5.3). - Additional unclear description in the description of the experiments and method sections (4.2, 4.3)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks
Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level. We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos https://sites.google.com/view/hide-rl
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The submission proposes a novel method for explicit decomposition of hierarchical policies for long-horizon navigation tasks. The approach proposes to separate a policy into 3 modules, high-level planner, intermediate planner and low-level control. The evaluation shows that explicit decomposition is well suited for generalisation across a limited set of RL domains. The proposed method integrates aspects from a variety of recent work including planning layers from value propagation networks, hindsight training paradigms from hierarchical actor critic and hindsight experience replay and related techniques. The variety of different techniques combined instead of a single main contribution renders it challenging to follow all aspects and in particular to trace relevant contributions to performance - which is rendered harder by a limited evaluation section. While the approach shows good performance against a couple of start of the art methods, it is necessary to provide sufficient ablations to enable long-term insights for the community. The submission high level goal (explicit decomposition and information asymmetry) is clear, the execution involves the combination of many existing techniques plus variations such that it is hard to make solid statements about the relevance of any part. It is commendable that the authors have introduced adaptations and improvements to their baselines for a stronger and fairer comparison but the evaluation remains very limited. I suggest to run different domains as given by other domains from OpenAI gym or DeepMind control suite. But more importantly I suggest to run further ablations without the intermediate planning layer & with absolute goal positions. Furthermore, since HAC seems to perform worse when combined with the proposed low and mid level policies (RelHAC), it would make sense to compare to the proposed high-level policy using low and mid level from HAC instead. The submission provides an overall interesting perspective but makes it hard to narrow down on contribution and important insights by being unclear in formulation and providing only very limited ablations. Minor issues include: - Missing description of the mid level policy - what encourages the proposal of closer short-term goals. - Missing literature on information asymmetry in RL (e.g. see Tirumala et al 2019 ‘Exploiting Hierarchy for Learning and Transfer in KL-regularized RL’) - Unclear description of how models that get attached to existing planning layers have been trained (Sec 5.3). - Additional unclear description in the description of the experiments and method sections (4.2, 4.3)
assistant
The submission proposes a novel method for explicit decomposition of hierarchical policies for long-horizon navigation tasks. The approach proposes to separate a policy into 3 modules, high-level planner, intermediate planner and low-level control. The evaluation shows that explicit decomposition is well suited for generalisation across a limited set of RL domains. The proposed method integrates aspects from a variety of recent work including planning layers from value propagation networks, hindsight training paradigms from hierarchical actor critic and hindsight experience replay and related techniques. The variety of different techniques combined instead of a single main contribution renders it challenging to follow all aspects and in particular to trace relevant contributions to performance - which is rendered harder by a limited evaluation section. While the approach shows good performance against a couple of start of the art methods, it is necessary to provide sufficient ablations to enable long-term insights for the community. The submission high level goal (explicit decomposition and information asymmetry) is clear, the execution involves the combination of many existing techniques plus variations such that it is hard to make solid statements about the relevance of any part. It is commendable that the authors have introduced adaptations and improvements to their baselines for a stronger and fairer comparison but the evaluation remains very limited. I suggest to run different domains as given by other domains from OpenAI gym or DeepMind control suite. But more importantly I suggest to run further ablations without the intermediate planning layer & with absolute goal positions. Furthermore, since HAC seems to perform worse when combined with the proposed low and mid level policies (RelHAC), it would make sense to compare to the proposed high-level policy using low and mid level from HAC instead. The submission provides an overall interesting perspective but makes it hard to narrow down on contribution and important insights by being unclear in formulation and providing only very limited ablations. Minor issues include: - Missing description of the mid level policy - what encourages the proposal of closer short-term goals. - Missing literature on information asymmetry in RL (e.g. see Tirumala et al 2019 ‘Exploiting Hierarchy for Learning and Transfer in KL-regularized RL’) - Unclear description of how models that get attached to existing planning layers have been trained (Sec 5.3). - Additional unclear description in the description of the experiments and method sections (4.2, 4.3)