We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph which describes a set of subtasks and their dependencies that are unknown to the agent. The agent needs to quickly adapt to the task over few episodes during adaptation phase to maximize the return in the test phase. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference (MSGI), which infers the latent parameter of the task by interacting with the environment and maximizes the return given the latent parameter. To facilitate learning, we adopt an intrinsic reward inspired by upper confidence bound (UCB) that encourages efficient exploration. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter, and to adapt more efficiently than existing meta RL and hierarchical RL methods.
This paper proposes a new meta-reinforcement learning algorithm, MSGI, which focuses on the problem of adapting to unseen hierarchical tasks through interaction with the environment where the external reward is sparse. The authors make use of subtask graph inference to infer the latent subtask representation of a task through interacting with the environment using an adaptation policy and then optimize the adaptation policy based on the inferred latent subtask structure. Each task in the paper is represented as a tuple of subtask precondition and subtask reward, which are inferred via logic induction and MLE of Gaussians respectively. At meta-test time, MSGI rollouts a subtask graph execution (SGE) policy based on the graph inferred from the interactions between the environment and the adaptation policy. The authors also propose a UCB-inspired intrinsic reward to encourage exploration when optimizing the adaptation policy. Experiments are conducted on two grid-world domains as well as StarCraft II.
Overall, this paper is mainly an extension of the prior work [1], which uses a subtask graph for tackling hierarchical RL problems. This work builds upon [1] by extending to meta-learning domains and studying generalization to new hierarchical tasks. While the contribution seems a bit incremental and the experimental setting is a bit unclear and limited to low-dimensional state space, the inference of task-specific subtask graphs based on past experiences and the proposal of a UCB-inspired reward shed some interesting insights on how to approach meta-hierarchical RL where long-horizon tasks and sparse rewards have been major challenges. Given some clarification on the experimental setup and additional results on more challenging domains in the author's response, I would be willing to improve my score.
Regarding the experimental setup, the set of subtasks is a Cartesian product of the set of primitive actions and a set of all types of interactive objects in the domain, while the state is represented as a binary 3-dimensional tensor indicating the position of each type of objects. Such a setup seems a bit contrived and is limited to low-dimensional state space and discrete action space, which makes me doubt its scalability to high-dimensional continuous control tasks. It would be interesting to see how/if MSGI can perform in widely used meta-RL benchmarks in Mujoco. I also wonder how MSGI can be compared to newly proposed context-based meta-RL methods such as PEARL.
As for the results, the authors don't provide an ablation study on the UCB exploration bonus though they claim they would show it in the paper. Moreover, the result of GRProp+Oracle is also missing in the comparison. I also don't understand why MSGI-Meta and RL2 would overfit in the SC2LE case and are unable to adapt to new tasks. Is that a limitation of the method? The authors also introduce MSGI-GRProp in this setting, which is never discussed before, and claim that MSGI-GRProp can successfully generalize to new tasks. It seems that the authors don't use a meta-RL agent in order to get this domain to work. I believe more discussion on this part is needed.
[1] Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero-shot
generalization with subtask dependencies. In NeurIPS, pp. 7156–7166, 2018.