Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level. We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos https://sites.google.com/view/hide-rl.
The paper proposes a neat framework for creating HRL framework that will be able to generalize its application to slightly different environment layout. This is done via an image-based top-down from as input to the high level. An intermediate layer is used to help create more fine-grained goal specification for a final goal-based control layer. These layers are trained together using HAC. Overall, the method shows promise but there needs to be more analysis to understand which parts of this combination of ideas are the most important. The results are also only shown for a single environment. Last, the generalization analysis in the paper does not appear to be overly thorough. It would be good to perform this on more than one type of environment also the random environment is not very random.
More detailed comments:
- The results seem very similar to some of the work in "Universal Planning Networks" that did not need a more complex HRL design to achieve subgoal specification via images. This should be discussed more in the paper.
- The authors point out that the use of relative goal positions "ensures generalization to new environments", this is a rather strong statement. The use of relative goal specification my help improve generalization but that can only be shown empirically.
- The demonstration to show that the method generalizes to other configuration after being trained on a fixed environment should be evaluated over many randomly generated environments so that we have a non-biased estimate of the true generalization performance. In Table 2, it is rather surprising that the HiDe trained model does better on the "Random" environment vs the "FOrward" environment it is trained on. Can more details be provided on how the "Random" environment is created? Are the locations of the walls randomized? Is the initial position of the agent and goal randomize?
- The video seems to contradict the ordering of operations for training the planning network. The video suggests that first it is learned with the Ant then transferred to the ball which is less complex to control.
- At the beginning of the prior work section, it is noted that many other methods require prior knowledge of the environment I would say this method also requires certain kinds of prior knowledge about the task. For example, a top-down view of the environment is needed which is not often feasible.
- HIRO and HAC use a more proprioceptive state space but I don't think the sharing of global states is intentional. I am not convinced that this choice, in particular, is what makes the approaches prone to overfitting.
- You show a comparison to the "windows" created from your method vs a fixed neighbourhood. Do you perform any empirical evidence that your introduced methods provide an improvement over this fixed window?
- It is mentioned at the end of section 4.1 that MVProp is differential so it can be trained with the Bellman error objective. Because many policies are being trained concurrently does the MVProp attention model need to be recomputed after every sub-policy update? Does the frequency of updates have a large effect on performance?
- Section 4.2 introduces an interface layer that is not a very common practice. It would be good to include an ablation study of the effects of this introduced layer.
- In section 4.3 it says that the control layer is the only layer with access to the agent's proprioceptive state. Would it not be good to at least include the agent facing direction or current average velocity to higher layers to improve the attention mask estimation?
- In figure 5 it says HIRO converges the fastest because it has dense rewards. Can you be more specific? Also, If different agents are using different reward signals I am not sure this evaluation is a fair comparison.
- Are tables 2 and 3 just for the HiDe algorithm? Is it possible to include data for the other algorithms? 
- You perform an experiment to train HiDe with random initial and goal locations for comparison. I think running this comparison for HIRO and HAC would be a good additional point of comparison. This would help the reader know if the generalization is not biased to the particular initial environment configuration for Maze Forward.
- In the generalization analysis for the paper, how is the analysis performed? There are percentages for the success of the policy, where does the randomness come from is the agent state and goal are always fixed? Are these averaged because the agent has a stochastic policy during evaluation? If this is the case how many random trajectories are collected to compute these statistics?
- It would be very helpful to have a description of the algorithm in the paper. How the algorithm works is not very clear and some details about how the goal and states are passed to the different policies would be very helpful if anyone wanted to reimplement this work.
Updated comments:
- The addition of more results and added analysis helps show the improvement of this method over the most related baselines.