Published as a conference paper at ICLR 2018 HIERARCHICAL AND INTERPRETABLE SKILL ACQUI- SITION IN MULTI-TASK REINFORCEMENT LEARNING
Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.
Summary:
This paper proposes an approach to learning hierarchical policies in a lifelong learning context. This is achieved by stacking policies - an explicit "switch" policy is then used to decide whether to execute a primitive action or call the policy of the layer below it. Additionally, each task is encoded in a human-readable template, which provides interpretability.
Review:
Overall, I found the paper to be generally well-written and the core idea to be interesting. My main concern is about the performance against existing methods (no empirical results are provided), and while it does provide interpretability, I am not sure that other approaches (e.g. Tessler et al. 2017) could not be slightly modified to do the same. I think the paper could also benefit from at least one more experiment in a different, harder domain.
I have a few questions and comments about the paper:
The first paragraph claims "This precludes transfer of previously learned simple skills to a new policy defined over a space with differing states or actions". I do not see how this approach avoids suffering from the same problem? Additionally, approaches such as agent-space options [Konidaris and Barto. Building Portable Options: Skill Transfer in Reinforcement Learning, IJCAI 2007] get around at least the state part.
I do not quite follow what is meant by "a global policy is assumed to be executable by only using local policies over specific options". It sounds like this is saying that the inter-option policy can pick only options, and not primitive actions, which is obviously untrue. Can you clarify this sentence?
In section 3.1, it may be best to mention that the policy accepts both a state and task and outputs an action. This is stated shortly afterwards, but it was confusing because section 3.1 says that there is a single policy for a set of tasks, and so obviously a normal state-action policy would not work here.
At the bottom of page 6, are there any drawbacks to the instruction policy being defined as two independent distributions? What if not all skills are applicable to all items?
In section 5, what does the "without grammar" agent entail? How is the sampling from the switch and instruction policies done in this case?
While the results in Figures 4 and 5 show improvement over a flat policy, as well as the value of using the grammar, I am *very* surprised there is no comparison to existing methods. For example, Tessler's H-DRLN seems like one obvious comparison here, since it learns when to execute a primitive action and when to reuse a skill.
There were also some typos/small issues (I may have missed some):
pg 3: "In addition, previous work usually useS..."
pg 3. "we encode a human instruction to LEARN A..." (?)
pg 4. "...with A stochastic temporal grammar..."
pg 4. "... described above through A/THE modified..."
pg 6. "...TOTALLING six colors..."
There are some issues with the references (capital letters missing e.g. Minecraft)
It also would be preferable if the figures could appear after they are referenced in the text, since it is quite confusing otherwise. For example, Figure 2 contains V(s,g), but that is only defined much later on. Also, I struggled to make out the yellow box in Figure 2, and the positioning of Figure 3 on the side is not ideal either.