LEARNING MULTI-LEVEL HIERARCHIES WITH HINDSIGHT
Multi-level hierarchies have the potential to accelerate learning in sparse reward tasks because they can divide a problem into a set of short horizon subproblems. In order to realize this potential, Hierarchical Reinforcement Learning (HRL) algorithms need to be able to learn the multiple levels within a hierarchy in parallel, so these simpler subproblems can be solved simultaneously. Yet most existing HRL methods that can learn hierarchies are not able to efficiently learn multiple levels of policies at the same time, particularly in continuous domains. To address this problem, we introduce a framework that can learn multiple levels of policies in parallel. Our approach consists of two main components: (i) a particular hierarchical architecture and (ii) a method for jointly learning multiple levels of policies. The hierarchies produced by our framework are comprised of a set of nested, goal-conditioned policies that use the state space to decompose a task into short subtasks. All policies in the hierarchy are learned simultaneously using two types of hindsight transitions. We demonstrate experimentally in both grid world and simulated robotics domains that our approach can significantly accelerate learning relative to other non-hierarchical and hierarchical methods. Indeed, our framework is the first to successfully learn 3-level hierarchies in parallel in tasks with continuous state and action spaces. We also present a video of our results and software to implement our framework.
This paper proposed a framework that can improve the performances of reinforcement learning algorithms in tasks that involve long time horizons and sparse rewards. The proposed method is a hierarchical reinforcement learning framework that can use policy hierarchies with an arbitrary number of levels. To improve the sample efficiency in the learning process, the authors proposed to apply the hindsight experience replay mechanism at each level. Also, in order to avoid the actor function to output an unrealistic subgoal, the authors proposed the subgoal testing technique. 
The proposed framework is interesting. And the example in Section 3.5 clearly demonstrate how this framework works. The authors proposed to solve a UMDP by solving a hierarchy of k UMDPs, where k is a hyperparameter. Each level (except for the bottom most level) will output subgoal states for the next level to achieve. This hierarchy is reasonable and easy to understand. However, from the definition on Page 3, it seems that all of the intermediate levels i (the case where 0 < i < k - 1) has the same state and action spaces. They are all equal to the state set of the original UMDP. Under this setting, will adding more intermediate levels help improve the performance a lot? We only see results with at most one intermediate level in the experiment. It will be better if the authors can show results on more levels (i.e. at least 4 levels in total). 
Moreover, the proposed framework has a policy limit parameter T, meaning that we only consider if a goal can be achieved within T steps or not, at each level. Is this parameter necessary to be the same for all levels? Also, it will be better if the author can show some results on the performances of the proposed method according to different values for T. The authors also proposed the subgoal testing technique. It is also better if the authors can show some performance comparisons on the cases with and without this technique.
The authors claimed that their method has the advantage over some existing HRL methods (e.g. the Option-Critic Architecture [1]) that their method can use policy hierarchies with an arbitrary number of levels while these methods can only use policy hierarchies with two levels. In the experiments, the authors also showed that, in some of their experiments, the 3-layer agent (with 2 subgoal layers) outperforms the 2-layer agent (with 1 subgoal layer), under their framework. However, the authors did not compare their 2-layer agent's performance with these existing HRL methods, which means that we do not know if their 3-layer agent's performance is better than that of some of the existing 2-layer agent methods. In addition to that, as I mentioned before, it is better if the authors can show experiment results on more levels (e.g. 4 levels and more) to show that their method can perform well in practice for policy hierarchies with many levels.
References:
[1] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. CoRR, abs/1609.05140, 2016.