Exploration is a well known challenge in Reinforcement Learning. One principled way of overcoming this challenge is to find a hierarchical abstraction of the base problem and explore at these higher levels, rather than in the space of primitives. However, discovering a deep abstraction autonomously remains a largely unsolved problem, with practitioners typically hand-crafting these hierarchical control architectures. Recent work with multitask linear Markov decision processes, allows for the autonomous discovery of deep hierarchical abstractions, but operates exclusively in the offline setting. By extending this work, we develop an agent that is capable of incrementally growing a hierarchical representation, and using its experience to date to improve exploration.
This work builds on the ICML paper from Saxe et al (2017) in which the compositionality property of LMDPs was exploited to solve multi-task hierarchies. The paper extends this work by proposing a method that learns incrementally such hierarchies instead of pre-defining them by design. Some experimental results illustrate the method on two toy problems, a 1D corridor and a corridor of rooms.
The paper deals with an interesting and hard problem. Learning hierarchies while solving an MDP is a much harder problem than solving the flat MDP or solving the hierarchical MDP. The authors leverage the compositionality of optimal controls of the LMDP framework to learn incrementally the hierarchies. Surprisingly, the proposed method not only learns those hierarchies, but also is more effective in terms of exploration.
On the positive side, the main idea is very interesting and has a lot of potential. The authors combine many techniques under the powerful framework of LMDPs such as hierarchical RL, low-rank factorization, and count-based exploration. The authors do a good job describing their approach (at a higher level).
On the negative side, the paper looks a bit incremental, given the prior existing work. I also found the paper unclear in many aspects lacking some relevant technical details (see below). The narrative is sometimes superficial or focused mainly on intuitions and analogies. Overall, it is difficult to assess the significance of this work and the results give the impression of limited applicability, beyond the described toy problem.
1- First, in order to combine optimal controls, the LMDPs need to be solved for each different boundary state, i.e., do you require to solve as many LMDPs as possible states? If that is the case, I don't think it makes sens to talk about exploration/exploration trade-off, since you really need to visit all states a priori.
2- I cannot understand what is learned and what is required a priori. the authors state that if "the multi-scale structure of the domain is known a priori, the decomposition (...) explicitly specified". What does exactly that mean? If what the method does is an incremental version of the low-rank factorization proposed in Earle at al (2018), I think the presentation can be better described in those terms.
2- Regarding exploration/exploitation tradeoff. From section 3, it seems that authors focus on a particular spatial problem and define already some exploration choices. But this means that the choice about when a state is integrated in the current MDP is already done, so no real trade-off exists?
3- The narrative in Section 3.2 is not very rigorous. The authors just mention the computational problem to keep consistency between layers and then just argue that "in practice" using count-based exploration everything works. I think a more principle approach is necessary.
4- Experiments I: what do the authors means by "exploration"? Is it just Boltzmann exploration? I can think of an exploration strategy that would choose an unseen state with probability 1 and would bring the agent to the goal in one shot.
5- Experiments II: I like the benchmark but, how does the result depend on the structure of the problem? What happens if I the rooms have very different sizes?
6- I miss some references that are very relevant to this work:
- the ICAPS paper "Hierarchical Linearly-Solvable Markov Decision Problems" by Jonson et al. seems to be the first proposing a hierarchical embedding of LMDPs.
- other factorization techniques exist, e.g., "Incremental Stochastic Factorization for Online Reinforcement Learning", Barreto et al (AAAI' 16), to uncover an MDP structure.
There are also some minor grammar mistakes:
"passive dynamics then become" -> "passive dynamics then becomes"
"reward function r" -> "reward function R"
"Howver" -> "However"
"the room contains" -> "the room that contains"
"have simply add" -> "have simply added"
"spacial and temporal" -> "spatial and temporal"
...