Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. One of the key difficulties is insufficient exploration, resulting in an agent being unable to learn robust policies. Intrinsically motivated agents can explore new behavior for their own sake rather than to directly solve external goals. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical action-value functions, operating at different temporal scales, with goal-driven intrinsically motivated deep reinforcement learning. A top-level q-value function learns a policy over intrinsic goals, while a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse and delayed feedback: (1) a complex discrete stochastic decision process with stochastic transitions, and (2) the classic ATARI game -'Montezuma's Revenge'.
The paper addresses deep hierarchical RL with a two-levels approach. Low-level controllers are used to reach goals, and a higher-level controller chooses the current goal. Goal specification is not clearly described. The model is tested first in a toy problem and then on Montezuma's revenge, a hard ATARI game. Results are preliminary, but quite promising.
The paper addresses the timely question of building a hierarchy of behaviours out of recent deep reinforcement learning methods. This is one of the hot challenges at the moment and several teams are on that (actually, the arXiv version of the paper is already cited 10 times). The paper proposes a promising contribution to the domain, but my general feeling is that the paper and the corresponding work are not mature enough for publication at NIPS: several points need to be clarified and the empirical evaluation is too preliminary.  Globally, the major issue in the field is the automatic extraction of a hierarchy from data. Here, a good deal of the hierarchy building process is left to the engineer through the identification of a set of goals. In itself, this is not a major flaw of the paper, but the limitations of the work should be discussed under this light.  About the literature review section, two points:   1°) "Our approach does not use separate Q-functions for each option, but instead treats the option as part of the input, similar to [18]. This has two advantages: (1) there is shared learning between different options, and (2) the model is potentially scalable to a large number of options."  => The statement about these two advantages should be supported either by a reference or an empirical study, where the authors would compare both approaches  2)° In the "Intrinsically motivated RL" and the "Cognitive Science and Neuroscience" subsections, nothing is said to relate the described works to the submitted paper. In particular, for the latter, the content looks quite far away. Also, the "Deep Reinforcement Learning" subsection does not bring much. So my general feeling is that this section should be more targeted to recent deep hierarchical RL work, and should save space for additional studies such as the one described in point 1 or improving the model description (see below).  By the way, given that the arXiv version of the paper was published a while before NIPS, the authors should refresh their literature review section the many recent works in the domain. E.g. Model-free episodic control, several works about macro-actions, using A3C instead of DQN (fine if that's left for future work ;)), etc.  About the model description:  The model section could be much clearer. Although the right-hand side of Fig.1 shows it, it is not obvious from the notations in the text that the controller networks take s and g as input and a as output, and that the "meta-controller" takes s as input and g as output. It would be clear if you write : Q1(A|s,g) and Q2(g|s), for instance.  Also, there should be a module (inside the meta-controller?) checking that a goal is reached. This is not described clearly. The language to specify goals is not described either (despite the connection to Duik's work about object-oriented MDPs).  The replay buffer initialization is not described.  The exploration strategy deserves its own subsection. Using a separate epsilon for each goal should be discussed. Also, the adaptive annealing method for \epsilon_1,g is not described, though this is not standard.  It should be specified that a set of goals is given as input to the algorithm. Also, the method for checking that a goal is reached should be present in the algorithm. This part is poorly specified and looks quite ad hoc (see below about Experiment 2).  About the study and results:  In Experiment 1, the structure of the controllers and meta-controller are not described (and should be). It is only specified that it is not a deep network.  In Experiment 2, the lack of a clear specification of how the goals are managed is a major issue. How many entities do you have in Montezuma's revenge? If you have two instances of the same entity, do they correspond to two different goals? Is reaching an entity with the agent the only kind of goal? What about avoiding monsters at pacMan, for instance? Even if what the authors did is completely ad hoc to a specific game, it should at least be described accurately.  The parametrization of the architecture and learning process looks quite arbitrary and should be discussed.  Finally, the small paragraph about missing ingredients convinces the reader that this work, although an interesting starting point, is not mature enough for publication in a major conference, particularly one that is biased towards clean theoretically-grounded work.