Data-Efficient Hierarchical Reinforcement Learning
Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higherand lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higher-and lower-level policies using substantially fewer environment interactions than on-policy algorithms. We term the resulting HRL agent HIRO and find that it is generally applicable and highly sample-efficient. Our experiments show that HIRO can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations, 1 learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.
Summary =======  The authors present a heirarchical reinforcement learning approach which learns at two levels, a higher level agent that is learning to perform actions in the form of medium term goals (changes in the state variable) and a low level agent that is aiming to (and rewarded for) achieving these medium term goals by performing atomic level actions.  The key contributions identified by the authors are that learning at both lower and higher level are off-policy and take advantage of recent developments in off-policy learning. The authors say that the more challenging aspect of this, is the off policy learning at the higher level, as the actions (sub-goals) chosen during early experience are not effectively met by the low level policy. Their solution is to instead replace (or augment) high level experience with synthetic high level actions (sub-goals) which would be more likely to have happened based on the current instantiation of the low level controller.  An additional key feature is that the sub-goals, rather than given in terms of absolute (observed) states, are instead given in terms of relative states (deltas), and there is a mechanism to update this sub-goal appropirately as the low level controller advances. As a side note, I suggest the authors look to the work of George Konidaris, in particular (Konidaris and Barto, 2007), in which agent centric formalisms are championed as most suitable for transfer learning (something the authors say they are interested in investigating in future work).  My concerns are relatively minor and a brief overview is as follows:   * I think that the cited paper [24] (though unpublished), is slightly closer to this work than the authors allow.   * There are some minor technical details, including that relating to prior work (e.g.  DDPG method [25], UVFA [34]), which could be made clearer and more explicit (see below).   * Some notation could be described more fully, and the reader guided a little to anticipate the meaning and use of certain constructions (e.g. the notation for subsequences of time series and sums over such series should be explicit, the fact that the raw tuples stored for the higher level controller will be later processed into conventional off-policy RL tuples).   * Some ideas are reiterated more than once or descriptions could be tightened up a bit, (e.g. Sections 3.2 and 3.3).  While I have a few comments on the details of the paper, I think the paper is well written and the work reproducible. The experiments, although in a limited domain, are varied and the results convincing. Moreover, I believe it presents a significant and interesting contribution to the field, and I am recommending it for acceptance.   Local references ---------------------  George Konidaris, Andrew G Barto (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. IJCAI.  Details ====== ### p3, line 104, minor point. > ...for fixed σ...  I assume this is standard deviation, but it is fairly common practice to parametrise normal distributions by variance.  ### p3, line 111 > ...specify a potentially infinite set of lower-level policies...  To be precise, I would suggest *unbounded* rather than *potentially infinite*.  ### p3, line 122 > a fixed goal transition function $g_t = h(s_{t−1} , g_{t−1 }, s_t )$  This is quite an unusual feature and is necessary here because goals (being relative) are not static. It would help to tell the reader that here.  ### p4, line 130-1 > the higher-level transition (s t:t+c−1 , g t:t+c−1 , a t:t+c−1 , R t:t+c−1 , s t+c )  The reader might be guided here too.  ### p4, line 139-40 > To maintain the same absolute position of the goal regardless of state change, the goal transition model h is defined as  Being a little picky maybe, but...What are the assumptions on your state space that ensures a relative goal can be (always) defined? And that when updated in the way suggested, will always be meaningful/consistent?   ### p5, lines 154-8, clarity > Parameterized rewards are not a new concept, and have been studied previously [34, 19]. They are a natural choice for a generally applicable HRL method and have therefore appeared as components of other HRL methods [43, 22, 30]. A significant distinction between our method and these prior approaches is that we directly use the state observation as the goal, and changes in the state observation as the action space for the higher-level policy...  The (unpublished) work [24], cited elsewhere does use states as sub-goals, and also as actions in the higher level policy. But not changes in the state as done here.    ### p5, line 159, typo > This allows the lower-level policy to begin receiving reward signals immediately, even before the lower-level policy has figured out how to reach the goal...  I think the second *lower-level*, should be *higher-level*.  ### p5, line 164, citation needed > While a number of prior works have proposed two-level HRL architectures  ### p5, line 169-70, citation needed > off-policy algorithms (often based on some variant of Q-function learning) generally exhibit substantially better sample efficiency than on-policy actor-critic or policy gradient variants.  ### p5, line 191 and Equation (5) > log probability...computed as...  I think the proportional to sign should be an equals, and this relies on an assumption about the sampling policy that should be made explicity, e.g. as a mean action with isotropic gaussian noise.  This part of the method seems the most tied to a specific formulation of the learner, and I wonder whether it could defined in more general terms first, with this as the example (I realise there are other approaches in the supplementary file). Also, authors should be clear at the outset that this is an approximate optimisation.  ### p5, line 194 > ...eight candidate goals sampled randomly from a Gaussian...  What is the Gaussian's covariance matrix?  ### p6, line 223-4, clarification when referring to ref [24] > ...[24]. In this work, the multiple layers of policies are trained jointly, while ignoring the non-stationarity problem.  The work cited is unpublished (arXiv), but I happen to have read it. In the cited work, the authors adapt a published method Hindsight Experience Replay (HER), which in their own work is used to replace the goals at lower levels with the actually achieved states. This contrasts with the currently submitted paper, in that the action of the higher level agent is the replaced entity (rather than the lower level goal), but in other respects there is some degree of similarity which could be remarked upon.  ### p6, line 231 > In our experiments, we found this technique to under-perform compared to our approach, which uses the state in its raw form. We find that this has a number of benefits.  This could be clearer.  ### p7, Figure 4 caption > extremely fast...  Use "quickly" or "rapidly".  Post Rebuttal ============  Reviewer #3's question regarding the significance of one of the results in Table 1 is valid. I still think the paper is interesting and worthy of publication, but I have moved my recommendation down to a 7 to reflect this concern.  If accepted for publication, I would encourage the authors to make clear in the camera ready version whether the values are standard deviations or standard errors or something else. The authors should also make clear their justification for the significance claim.