We consider the problem of exploration in meta reinforcement learning. Two new meta reinforcement learning algorithms are suggested: E-MAML and E-RL 2 . Results are presented on a novel environment we call 'Krazy World' and a set of maze environments. We show E-MAML and E-RL 2 deliver better performance on tasks where exploration is important.
The paper proposes a trick of extending objective functions to drive exploration in meta-RL on top of two recent so-called meta-RL algorithms, Model-Agnostic Meta-Learning (MAML) and RL^2. 
Pros:
+ Quite simple but promising idea to augment exploration in MAML and RL^2 by taking initial sampling distribution into account. 
+ Excellent analysis of learning curves with variances across two different environments. Charts across different random seeds and hyperparameters indicate  reproducibility. 
Cons/Typos/Suggestions:
- The brief introduction to meta-RL is missing lots of related work - see below.
- Equation (3) and equations on the top of page 4: Mathematically, it looks better to swap \mathrm{d}\tau and \mathrm{d}\bar{\tau}, to obtain a consistent ordering with the double integrals. 
- In page 4, last paragraph before Section 5, “However, during backward pass, the future discounted returns for the policy gradient computation will zero out the contributions from exploratory episodes”:  I did not fully understand this - please explain better. 
- It is not very clear if the authors use REINFORCE or more advanced approaches like TRPO/PPO/DDPG to perform policy gradient updates?
- I'd like to see more detailed hyperparameter settings. 
- Figures 10, 11, 12, 13, 14: Too small to see clearly. I would propose to re-arrange the figures in either [2, 2]-layout, or a single column layout, particularly for Figure 14. 
- Figures 5, 6, 9: Wouldn't it be better to also use log-scale on the x-axis for consistent comparison with curves in Krazy World experiments ?
3. It could be very interesting to benchmark also in Mujoco environments, such as modified Ant Maze. 
Overall, the idea proposed in this paper is interesting. I agree with the authors that a good learner should be able to generalize to new tasks with very few trials compared with learning each task from scratch. This, however, is usually called transfer learning, not metalearning. As mentioned above, experiments in more complex, continuous control tasks with Mujoco simulators might be illuminating. 
Relation to prior work:
p 2: Authors write: "Recently, a flurry of new work in Deep Reinforcement Learning has provided the foundations for tackling RL problems that were previously thought intractable. This work includes: 1) Mnih et al. (2015; 2016), which allow for discrete control in complex environments directly from raw images. 2) Schulman et al. (2015); Mnih et al. (2016); Schulman et al. (2017); Lillicrap et al. (2015), which have allowed for high-dimensional continuous control in complex environments from raw state information."
Here it should be mentioned that the first RL for high-dimensional continuous control in complex environments from raw state information was actually published in mid 2013:
(1) Koutnik, J., Cuccu, G., Schmidhuber, J., and Gomez, F. (July 2013). Evolving large-scale neural networks for vision-based reinforcement learning. GECCO 2013, pages 1061-1068, Amsterdam. ACM.
p2: Authors write: "In practice, these methods are often not used due to difficulties with high-dimensional observations, difficulty in implementation on arbitrary domains, and lack of promising results."
Not quite true - RL robots with high-dimensional video inputs and intrinsic motivation learned to explore in 2015: 
(2) Kompella, Stollenga, Luciw, Schmidhuber. Continual curiosity-driven skill acquisition from high-dimensional video inputs for humanoid robots. Artificial Intelligence, 2015.
p2: Authors write: "Although this line of work does not explicitly deal with exploration in meta learning, it remains a large source of inspiration for this work."
p2: Authors write: "To the best of our knowledge, there does not exist any literature addressing the topic of exploration in meta RL."
But there is such literature - see the following meta-RL work where exploration is the central issue:
(3) J. Schmidhuber. Exploring the Predictable. In Ghosh, S. Tsutsui, eds., Advances in Evolutionary Computing, p. 579-612, Springer, 2002.
The RL method of this paper is the one from the original meta-RL work:
(4) J. Schmidhuber. On learning how to learn learning strategies. Technical Report FKI-198-94, Fakultät für Informatik, Technische Universität München, November 1994.
Which then led to:
(5) J. Schmidhuber, J.  Zhao, N. Schraudolph. Reinforcement learning with self-modifying policies. In S. Thrun and L. Pratt, eds., Learning to learn, Kluwer, pages 293-309, 1997.
p2: "In hierarchical RL, a major focus is on learning primitives that can be reused and strung together. These primitives will frequently enable better exploration, since they’ll often relate to better coverage over state visitation frequencies. Recent work in this direction includes (Vezhnevets et al., 2017; Bacon & Precup, 2015; Tessler et al., 2016; Rusu et al., 2016)."
These are very recent refs - one should cite original work on hierarchical RL including:
J.  Schmidhuber. Learning to generate sub-goals for action sequences. In T. Kohonen, K. Mäkisara, O. Simula, and J. Kangas, editors, Artificial Neural Networks, pages 967-972. Elsevier Science Publishers B.V., North-Holland, 1991.
M. B. Ring. Incremental Development of Complex Behaviors through Automatic Construction of Sensory-Motor Hierarchies. Machine Learning: Proceedings of the Eighth International Workshop, L. Birnbaum and G. Collins, 343-347, Morgan Kaufmann, 1991.
M. Wiering and J. Schmidhuber. HQ-Learning. Adaptive Behavior 6(2):219-246, 1997
References to original work on meta-RL are missing. How does the approach of the authors relate to the following approaches? 
(6) J. Schmidhuber. Gödel machines: Fully Self-Referential Optimal Universal Self-Improvers. In B. Goertzel and C. Pennachin, eds.: Artificial General Intelligence, p. 119-226, 2006. 
(7) J. Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook. Diploma thesis, TUM, 1987. 
 
Papers (4,5) above describe a universal self-referential, self-modifying RL machine. It can implement and run all kinds of learning algorithms on itself, but cannot learn them by gradient descent (because it's RL). Instead it uses what was later called the success-story algorithm (5) to handle all the meta-learning and meta-meta-learning etc.
Ref (7) above also has a universal programming language such that the system can learn to implement and run all kinds of computable learning algorithms, and uses what's now called Genetic Programming (GP), but applied to itself, to recursively evolve better GP methods through meta-GP and meta-meta-GP etc. 
Ref (6) is about an optimal way of learning or the initial code of a learning machine through self-modifications, again with a universal programming language such that the system can learn to implement and run all kinds of computable learning algorithms.
General recommendation: Accept, provided the comments are taken into account, and the relation to previous work is established.