Some Considerations on Learning to Explore via Meta-Reinforcement Learning
We interpret meta-reinforcement learning as the problem of learning how to quickly find a good sampling distribution in a new environment. This interpretation leads to the development of two new meta-reinforcement learning algorithms: E-MAML and E-RL 2 . Results are presented on a new environment we call 'Krazy World': a difficult high-dimensional gridworld which is designed to highlight the importance of correctly differentiating through sampling distributions in meta-reinforcement learning. Further results are presented on a set of maze environments. We show E-MAML and E-RL 2 deliver better performance than baseline algorithms on both tasks.
sampling process that maximally informs the meta-learner how it should adapt to new environments. In the context of gradient based algorithms, this means that one principled approach for learning an optimal sampling strategy is to differentiate the meta RL agent's per-task sampling process with respect to the goal of maximizing the reward attained by the agent post-adaptation. To the best of our knowledge, such a scheme is hitherto unexplored.
In this paper, we derive an algorithm for gradient-based meta-learning that explicitly optimizes the pertask sampling distributions during adaptation with respect to the expected future returns produced by it post-adaptation self. This algorithm is closely related to the recently proposed MAML algorithm [7]. For reasons that will become clear later, we call this algorithm E-MAML. Inspired by this algorithm, we develop a less principled extension of RL 2 that we call E-RL 2 . To demonstrate that this method learns more transferable structures for meta adaptation, we propose a new, high-dimensional, and dynamically-changing set of tasks over a domain we call "Krazy-World". "Krazy-World" requires the agent to learn high-level structures, and is much more challenging for state-of-the-art RL algorithms than simple locomotion tasks. 2 .
We show that our algorithms outperform baselines on "Krazy-World". To verify we are not over-fitting to this environment, we also present results on a set of maze environments.
This paper proposes extensions to two popular meta-learning techniques: MAML and RL^2, with the aim of learning an optimal sampling strategy (policy) to maximize expected returns that takes into account the effect of the initial sampling policy on final return. This was more theoretically derived for MAML by explicitly including the dependency in the expectation and differentiating to compute the appropriate gradient, which turns out to have an additional term compared to MAML. This additional term allows the policy to be more exploratory. RL^2 was modified similarly by including multiple rollouts per update, p of which contribute 0 return to the backwards pass (exploratory episodes) and k-p of which contribute the normal return. These models were evaluated on a new environment “Krazy World”, which has dynamically generated features every episode, and also a maze environment.   The authors propose a simple but useful idea, and I think it does make a contribution. Although there are a lot of interesting ideas and results, it feels like a work in progress rather than a complete piece of work. Some technical details are missing, which impact the clarity and quality, as I’ve listed below. This makes me concerned about the reproducibility of their results.  The new environment Krazy World is an interesting benchmark task for meta-learning, and I’d like to see it open-sourced if possible.   More specific comments: -It’s interesting that E-MAML does better initially but E-RL^2 does better in the end. What level of reward is optimal for the two tasks? -For E-RL^2, have you explored tuning the relative contributions of exploration on final return? That is, how is performance impacted by performing more exploratory rollouts per update? How did you settle on 3 and 2? -Are the maze layouts procedurally generated every episode? -There are a lot of typos, please proofread more carefully (e.g. page 7 “desigend” -> “designed”, page 8 “defining an intrinsic motivation signals” -> “signal” and “and the its affect” -> “and its effect”) -In section 2.5, k is both the number of rollouts for each MDP and also the task the MDP represents, so this is confusing -It’s very strange that they don’t cite Duan et al’s RL^2 paper at all, since E-RL^2 is named after it -Figure 8 (in the appendix) is not helpful at all, the axes and legends are impossible to read and axes aren’t even labeled. Why are there 3 columns? The caption only mentions 2. -What do the error shades represent in the plots? Standard deviation? Standard error? Over how many runs?  Update after reading the rebuttal: It seems like all the reviewers had major concerns regarding clarity, and the authors' response does seem to address many of these. There are a lot of things to revise however, and I'm not completely convinced the final product will be ready for publication at NIPS, given how in-progress the initial version felt. Given this, I can't at this time change my assessment, although I still tentatively recommend acceptance and encourage the authors to revise with clarity in mind.