Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al., 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks. The mechanism behind their success, however, is poorly understood. We begin this work with an experimental analysis of MAML, finding that deep models are crucial for its success, even given sets of simple tasks where a linear model would suffice on any individual task. Furthermore, on image-recognition tasks, we find that the early layers of MAML-trained models learn task-invariant features, while later layers are used for adaptation, providing further evidence that these models require greater capacity than is strictly necessary for their individual tasks. Following our findings, we propose a method which enables better use of model capacity at inference time by separating the adaptation aspect of meta-learning into parameters that are only used for adaptation but are not part of the forward model. We find that our approach enables more effective meta-learning in smaller models, which are suitably sized for the individual tasks.
This paper analyzes the popular MAML (Model-Agnostic Meta-Learner) method, and thereafter proposes a new approach to meta-learning based on observations from empirical studies. The key idea of the work is to separate the base model and task-specific adaptation components of MAML. This decoupling of adaptation and modeling reduces the burden on the model, thus enabling smaller memory efficient deep learning models to adapt and give high performance on meta learning tasks. The paper proposes a learnable meta-optimizer consisting of a parametrized function U such that the knowledge of adaptation is embedded into its parameters (A,b), instead of forward model parameters. The computational challenges posed by the proposed method are addressed by expressing the parameter matrix A as a Knonecker product of small matrices which is more efficient from memory and time complexity view point. The results on Omniglot and CIFAR-FS are promising, and the paper shows that the proposed meta-optimize is "more expressive", as well as can adapt a shallower model to the same level of performance as MAML.
+ves:
+ The discussion on the deficiency of MAML combined with shallow models is well-supported experimentally. 
+ The idea to leverage the parameters of a meta-optimizer for adaptation instead of using model parameters is novel and interesting.
+ The paper is well-written and easy to follow. It motivates its choices well, both in the proposed method and the experiments.
+ The paper presents fair comparison in all experiments with appropriately chosen baseline models, and the proposed approach is validated for both linear as well as non linear models using benchmark datasets.
Concerns:
- While MAML was a seminal work and is widely followed, there have been many follow-ups of MAML, including another widely used method Reptile (Nichol et al, On First-Order Meta-Learning Algorithms). How is the proposed method relevant more broadly to this genre of methods? Some discussion of this would have been useful to understand the generalizability of the idea.
- The choice of the Kronecker product to handle the dimensionality of the meta-optimizer is supported by the paper, but is not very convincing. How important is this choice? What if other decompositions were used? 
- The paper seems to state that shallow models are convex (Sec 3.2); however, weight symmetry induces non-convexity even in shallow models. This perspective of the problem may not be very well-justified.
- In Sec 3.2, the paper compares the 1-step adaptation accuracy of a shallow network and a deeper 4 layered linear network and claim that shallow networks underperform. However this underperformance might be due to the difference in required number of steps to reach optimal performance by the two models, and may not be a fair comparison. Why is this conclusive inference? Considering these inferences motivate the full paper, this is important.
- All the presented results are on small CNNs. The paper motivates this as “easing the computational burden”. The original MAML work shows results on state-of-the-art convolutional and recurrent models. It may be important to show results on deeper models to be more confident about its applicability.
- Although one can obtain smaller meta-learned models using the proposed method, training via this method will incur a higher computational burden than MAML-trained deep models. The paper does not talk about this additional complexity at all. Comparisons of wall-clock times or asymptotic analysis of the proposed method w.r.t. MAML would have greatly helped understand the pros and cons of the method.
I am on the borderline on this work - it is a well-written paper with a clear objective and support. But lack of rigorous analysis of the proposed method in terms of the method (how important is the Kronecker factorization?), experiments (with deeper architectures) and a more generalizable understanding of the proposed idea seems to be limiting the work's impact. 
========POST-REBUTTAL COMMENTS===============
I thank the authors for their response, and all the efforts in the updated manuscript. Some of the clarifications sought were answered clearly. However, unfortunately, I continue to remain on the borderline on this work for the reasons below. (I would be willing to increase my rating to 4 or 5, which however are not available on the drop down, but perhaps not beyond).
- The response to AnonReviewer1 says that "there have been multiple empirical and theoretical works dedicated solely to the study of MAML [7-11]", hence supporting this work dedicating its focus to MAML alone. However, on close observation, most of these efforts are not published on peer-reviewed avenues and are only on arXiv at this time. Ref [7] (Finn and Levine, 2017) is published but has significantly stronger contributions. Considering the largely empirical nature of this work, showing its generalizability would be required, in my opinion, to make the conclusions of this work useful to the audience. Expecting that it would naturally hold for other methods like REPTILE may not be sufficient. In my opinion, this is a significant limitation.
- I personally remained unconvinced about the response to the question on number of adaptation steps, as well as on the lack of deeper models in the empirical studies.
I once again appreciate the authors for all the additional efforts, it may just be good for the work to be more comprehensive to be relevant and useful.