We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES). Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies. We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptually simple and easy to implement. Moreover, ES-MAML can handle new types of nonsmooth adaptation operators, and other techniques for improving performance and estimation of ES methods become applicable. We show empirically that ES-MAML is competitive with existing methods and often yields better adaptation with fewer queries.
The paper proposes ES for the task of Model agnostic meta learning. Instead of the gradient-approximation which requires computing a hessian matrix, MC samples from a search distribution are used to estimate a search direction. The approach is validated on a number of experiments.
Unfortunatly, I am unable to accept this paper for a number of reasons. Mainly that the ES used is inferior and the constant step-size used can have a major effect on the experimental outcome. 
Almost all proper ES literature with real working ES algorithms are missing and ESGrad is more than 20 years behind SOTA in the field. Since ES are central to the paper, an algorithm that would not even be considered a baseline at any conference in that field is difficult to accept.
The reason for this is that nowadays all ES use dynamic sample-variances based on progress measures, e.g. Cumulative step-size adaptation and Two-Point-Adaptation as the SOTA. Without this, it can be very difficult to find reasonable solutions.
	
Most important missing references from the ES-field in this context:
	
1. and most importantly The original ES-based RL paper:
Heidrich-Meisner, Verena, and Christian Igel. "Neuroevolution strategies for episodic reinforcement learning." Journal of Algorithms 64.4 (2009): 152-168.
	
2. CMA-ES and NES
Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary computation, 11(1), 1-18.
Krause, O., Arbonès, D. R., & Igel, C. (2016). CMA-ES with optimal covariance update and storage complexity. In Advances in Neural Information Processing Systems (pp. 370-378).
Wierstra, D., Schaul, T., Peters, J., & Schmidhuber, J. (2008, June). Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence) (pp. 3381-3387). IEEE.
3. Review of SOTA in large-scale ES:
Varelas, K., Auger, A., Brockhoff, D., Hansen, N., ElHara, O. A., Semet, Y., ... & Barbaresco, F. (2018, September). A comparative study of large-scale variants of CMA-ES. In International Conference on Parallel Problem Solving from Nature (pp. 3-15). Springer, Cham.
4. Recent developments for noisy functions (also references other relevant algorithms with noise-handling)
Krause, O. (2019, July). Large-scale noise-resilient evolution-strategies. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 682-690). ACM.
Section 3.2
-Why should in (7) the same sigma be used as in (6)? Sigma, alpha etc should be learnable parameters learned by the outer ES.
-3.3.2: you are writing below (1) that rollouts come from a distribution, i.e. are stochastic. How would you implement a hill-climber in the stochastic setting? e.g. consider the case when the rewards are heavy-tailed.
- using a hill-climber goes completely against the SOTA in ES which showed repeatedly over the last 20 years that hill-climbing is inferior, especially in larger dimension search-spaces (>100).
Experiments:
- I am not an expert of MAML, but i would not consider this as different tasks, just as different environments for the same task. i.e. a circular running strategy should be optimal for all environments. but when considering different tasks, we would consider different policies to be optimal.
- The experiments use the same hyper parameters for all variants. However, i am not sure this is a fair comparison. E.g. HC has way more spread over the search-space than the other two methods for a given sigma, with following sample steps allowing for fixing the "too large" or "too small" spread.
Since the graph of the objective function is flat in a large area of the search space, the additional exploration through stocasticity alone might explain the results of Figure 1. In this case, the result would be pretty artificial, because real ES would adapt their step-size.
- Similar holds for the number of samples used by the outer ES (n, but named differently in th appendix?). The gradient-based approaches might require a lot more initial points with a smaller K , especially on the flat surfaces of the objectives.
- In Figure 3, middle image, why does the green curve appear to have decreasing performance after iteration 200?
- Figure 3/ 4.2 why do the three settings have different values for number of iterations and K? Why does L-DPP only appear in the third task?
-Section 4.3 and Figure 4: why is there no L-PG and HH-ES? the only curve which is is available for both algorithms has the same performance.