HOW TO TRAIN YOUR MAML
The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.
Paper summary - This paper provides a bag of sensible tricks for making MAML more stable, faster to learn, and better in final performance.
Quality - The quality of the work is strong: the results demonstrate that tweaks to MAML produce significant improvements in performance. However, I have some concern that certain portions of the text overclaim (see concerns section below).
Clarity - The paper is reasonably clear, with some exceptions (see concerns section).
Originality - The techniques described in the paper range from only mildly novel (e.g. MSL, DA), to very obvious (e.g. CA). Additionally, the paper's contributions amount to tweaks to a previously existing algorithm. 
Significance - The quality of the results make this a significant contribution in my view.
Pros - Good results on a problem/algorithm of great current interest.
Cons - Only presents (in some cases obvious) tweaks to a previous algorithm; clarity and overclaiming issues in the writeup.
Concerns (please address in author response)
- The paper says  "we … propose multiple ways to automate most of the hyperparameter searching required". I'm not sure that this is true. The only technique that arguably removes a hyperparameter is LSLR. Even in this case, you still have to initialize the inner loop learning rates, so I'm not convinced that even this reduces hyperparameters. Perhaps I've missed something, please clarify.
- Section 4's paragraph on LSLR seems to say that you have a single alpha for each layer of the network. If this is right, then saying your method has a "per layer gradient direction" is very confusing. Each layer's alpha modulates the magnitude of that layer's update vector, but not its direction. The per-layer alphas together modify the direction of the global update vector. Perhaps I've misunderstood; equations describing exactly what LSLR does would be helpful. In any case, this should be clarified in the text.
Suggestions (less essential than the concerns above)
- The write-up is redundant and carries unnecessary content. The paper would be better shorter (8 pages is not a minimum :)
Section 1 covers a lot of background on the basics of meta-learning background that could be skipped. Other papers you cite (e.g. the MAML paper cover this). 
    - Section 2 goes into more detail about e.g. matching nets than is necessary. 
    - Section 2 explains MAML, which is then covered in much more detail in Section 3; better to leave out the Section 2 MAML paragraph. 
    - Sections 3 and 4 are very redundant. Combine them for a shorter (i.e., better!) paper.
- The paper says, "Furthermore, for each learning rate learned, there will be N instances of that learning rate, one for each step to be taken. By doing this, the parameters are free to learn to decrease the learning rates at each step which may help alleviate overfitting." Does this happen empirically? Space could be freed up (see above) to have a figure showing whether or not this happens.
- The paper says, "we propose MAML++, an improved meta-learning framework" -- it's a little too far to call this a new framework. it's still MAML, with improvements.
Typos
- "4) increase the system’s computational overheads" -> overhead
- "composed by" -> composed of
- "Santurkar et al. (2018).", "Krizhevsky et al. (2012),",  "Finn et al. (2017) " -> misplaced citation parens
- "a method that reduce" -> reduces
- "An evaluation ran consisted" -> evaluation consisted
- The Loshchlikov and Hutter citation in the bibliography isn't right. It should be "Sgdr: Stochastic gradient descent with restarts." (2016) instead of "Fixing weight decay regularization in adam" (2017).