Learning Abstract Options
Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning. One popular approach for addressing this challenge is the options framework [29]. However, only recently in [1] was a policy gradient theorem derived for online learning of general purpose options in an end to end fashion. In this work, we extend previous work on this topic that only focuses on learning a two-level hierarchy including options and primitive actions to enable learning simultaneously at multiple resolutions in time. We achieve this by considering an arbitrarily deep hierarchy of options where high level temporally extended options are composed of lower level options with finer resolutions in time. We extend results from [1] and derive policy gradient theorems for a deep hierarchy of options. Our proposed hierarchical option-critic architecture is capable of learning internal policies, termination conditions, and hierarchical compositions over options without the need for any intrinsic rewards or subgoals. Our empirical results in both discrete and continuous environments demonstrate the efficiency of our framework.
The paper extends the Option-Critic policy gradient theorems to multiple levels of abstractions (i.e. it allows options-over options at multiple levels). The results they present apply to execution models where action choices are conditioned on a vector [o^1, o_2, ..., o_N] representing abstract action choices at levels 1,2,...,N (1 being the highest). These choices are determined using policies \pi (o^j | o_{j-1}, o_{j-2}, ..., 1). Moreover, the conditioning is maintained until a termination signal, modeled in a similar way, is triggered at some level j, at which point [o_j , o_{j+1}, ..., o_N] are updated.  I consider this line of work very important, as it provides an investigation in the use of multiple levels of temporal abstractions, something that in most previous work was only alluded but never provided with clear derivations or empirical results. This paper provides these extensions. Although the paper is overall very well presented, there are some major technical details that are not properly described which makes me slightly skeptical of accepting the paper. See below for aspects that the authors should consider in improving the quality of the paper:  Major issues:  • From Eq (10), it seems that \beta(s, o^1, ..., o^N) is the prob. that at least one of the options terminate. For example, you use (1-\beta(s, o^1, ..., o^N)) for the event that none terminate.  Still, on L170 you give a different definition:  \beta(s, o^1, ..., o^{N-2}) is the prob. that o_{N-2} terminates given that o_{N-1} terminated and we were executing o^1, o_2, ..., o_{N-1}. This alternative definition is also used in the algorithm provided in the supplemental material. Although it makes no difference when N=2, it seems like a big inconsistency for N \geq 3. Please clarify the semantics of \beta to validate equation 10 and the Markov Chain in the supplemental material. • The paper argues for higher level abstractions, but nowhere in the experimental section do the authors try to illustrate the learned abstractions nor do they assess whether the resulting abstract options are degenerate (i.e. either never terminate or terminate immediately).  • L268- L270: Can you please add the details on how you ensured a fair comparison between the 2-level and multi-level architectures? I think this is a crucial aspect which should not be left out in the supplemental material. E.g. what was the numerical limit to put on the number of options at each level?  • as claimed in the abstract, the work "[learns abstract options] without the need for any intrinsic reward (IR) or subgoals", but it fails to explain why this is necessarily an advantage. I would argue that many of the problems associated with option-critic can be addressed by learning IR and subgoals (e.g. as demonstrated by work on Feudal Networks). The authors should clarify their perspective on the disadvantage of using IR and subgoals. •  Minor:  • nipick : what about "Hierarchical Option Critic" for a more concise title?   • L52: benefits • L74: Can you give some thoughts on learning both latent goal representations and options achieving these subgoals? Te statement on L74 seems to imply that these are orthogonal.  • L91: \pi: S \to (A \to [0,1]) for consistency with definition of P on L88. • L99:  P(s_t=s | s_0, \pi_\theta) -  state transitions are dependent on the policy • L110 - L129: You provide too much details coming from the Option-Critic paper - it would be much more useful to replace the policy gradient results (which are indirectly presented in 4.3) with a diagram that concisely illustrates the building blocks in Option-Critic as related to your work (e.g. the dependence between Q_\Omega, U, Q_U, V_\Omega, A_\Omega).  • Section 4 + supplemental work: consider using o_{1:k} instead of "o^1, o^2, ..., o^k". All results would read much easier.  • L185 (eq. 8) typo? Q_\Omega -> Q_U? • nipick: L351: remove "highly" + L352:  I don't think the experimental results support the statement as used in the conclusion. The results are interesting and illustrate in some cases the benefits of multiple levels of abstraction, but I would be reserved in using such superlatives.  ----After author feedback---- I would like to thank the authors for the additional feedback they provided, which includes clarifications to some of the issues I have raised. I still believe that the a proper qualitative analysis of the options learned at the higher levels is missing. I also find that a proper comparison to Feudal Networks is necessary for this work, and the author feedback is not very convincing on the many interesting aspects that Feudal Networks are using, such as intrinsic reward.