Hierarchical reinforcement learning is a promising approach to tackle long-horizon decision-making problems with sparse rewards. Unfortunately, most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task. Leaving the skills fixed can lead to significant sub-optimality in the transfer setting. In this work, we propose a novel algorithm to discover a set of skills, and continuously adapt them along with the higher level even when training on a new task. Our main contributions are two-fold. First, we derive a new hierarchical policy gradient with an unbiased latent-dependent baseline, and we introduce Hierarchical Proximal Policy Optimization (HiPPO), an on-policy method to efficiently train all levels of the hierarchy jointly. Second, we propose a method of training time-abstractions that improves the robustness of the obtained skills to environment changes. Code and videos are available here.
This paper is under the topic of hierarchical reinforcement learning. The motivation of this paper is "most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task." The paper proposes a method to learn higher-level skill selection and lower-level skill improvement jointly.
What I like in this paper:
    1. The paper, in general, is well-written so that I can understand it well.
    2. Experiments are question-driven and provide interesting results.
    3. Theories are closely related to the algorithm.
Key reasons for my rejection:
    1. My biggest concern is the motivation of this paper. 
    The joint learning of higher-level policy and lower-level skill discovery is not rare in modern literature. Some works are even cited in this paper, for example, option-critic, feudal network, etc. These methods fix their skills in the new task, not because they are inherently not able to do so, but because they want to demonstrate that the learned skills can be reused in new tasks, even if there is no further adaptation. I agree with the author that the agent needs to adapt its skills when faced with new tasks. But I don't think most works are limited in this aspect, as claimed by the paper in the abstract.
    2. I think the author didn't justify his key design choices well. 
    This paper is under the research area of "hierarchical reinforcement learning." However, just like temporal abstraction, the HRL is a general idea instead of an existing problem formulation or a particular algorithm. It seems that the author is not aware of this point as the paper claims a particular way of achieving HRL is the HRL itself (in section 4.1 "In the context of HRL, a hierarchical policy with a manager πθh(zt|st) selects every p time-steps one of n sub-policies to execute."). I would like to see the paper takes the responsibility to justify the reason it follows this particular way. There are two more key decisions the paper proposed but not fully justified and analyzed.
        1. Why is random length a valid choice? The paper doesn't tell readers the consequence of this design choice. For example, what about the optimality of the solution? Since bounding the random length needs prior knowledge, how difficult is it to come up with the prior knowledge. Is the algorithm sensitive to prior knowledge?
        2. Why is it fine to assume "for each action, there is just one sub-policy that gives it high probability"? What would be the consequence of this assumption? Well, the extreme case is each action is only being chosen by one sub-policy. Therefore, executing the sub-policy becomes executing a repeated sequence of the same action. Obviously, this is a kind of temporal abstraction but is a very limited one. 
Other small issues:
Section 2: "... maximize the discounted expected reward ..." should be "... maximize the discounted expected return ...".
Section 2: the horizon T in the definition in \eta should be H.
Section 4: the advantage function is not defined.