Published as a conference paper at ICLR 2018 META-LEARNING AND UNIVERSALITY: DEEP REPRESENTATIONS AND GRADIENT DESCENT CAN APPROXIMATE ANY LEARNING ALGORITHM
Learning to learn is a powerful paradigm for enabling models to learn from data more effectively and efficiently. A popular approach to meta-learning is to train a recurrent model to read in a training dataset as input and output the parameters of a learned model, or output predictions for new test inputs. Alternatively, a more recent approach to meta-learning aims to acquire deep representations that can be effectively fine-tuned, via standard gradient descent, to new tasks. In this paper, we consider the meta-learning problem from the perspective of universality, formalizing the notion of learning algorithm approximation and comparing the expressive power of the aforementioned recurrent models to the more recent approaches that embed gradient descent into the meta-learner. In particular, we seek to answer the following question: does deep representation combined with standard gradient descent have sufficient capacity to approximate any learning algorithm? We find that this is indeed true, and further find, in our experiments, that gradient-based meta-learning consistently leads to learning strategies that generalize more widely compared to those represented by recurrent models.
This paper studies the capacity of the model-agnostic meta-learning (MAML) framework as a universal learning algorithm approximator. Since a (supervised) learning algorithm can be interpreted as a map from a dataset and an input to an output, the authors define a universal learning algorithm approximator to be a universal function approximator over the set of functions that map a set of data points and an  input to an output. The authors show constructively that there exists a neural network architecture for which the model learned through MAML can approximate any learning algorithm. 
The paper is for the most part clear, and the main result seems original and technically interesting. At the same time, it is not clear to me that this result is also practically significant. This is because the universal approximation result relies on a particular architecture that is not necessarily the design one would always use in MAML. This implies that MAML as typically used (including in the original paper by Finn et al, 2017a) is not necessarily a universal learning algorithm approximator, and this paper does not actually justify its empirical efficacy theoretically. For instance, the authors do not even use the architecture proposed in their proof in their experiments. This is in contrast to the classical universal function approximator results for feedforward neural networks, as a single hidden layer feedforward network is often among the family of architectures considered in the course of hyperparameter tuning. This distinction should be explicitly discussed in the paper.  Moreover, the questions posed in the experimental results do not seem related to the theoretical result, which seems odd.
Specific comments and questions: 
Page 4: "\hat{f}(\cdot; \theta') approximates f_{\text{target}}(x, y, x^*) up to arbitrary position". There seems to be an abuse of notation here as the first expression is a function and the second expression is a value.
Page 4: "to show universality, we will construct a setting of the weight matrices that enables independent control of the information flow...". How does this differ from the classical UFA proofs? The relative technical merit of this paper would be more clear if this is properly discussed.
Page 4: "\prod_{i=1}^N (W_i - \alpha \nabla_{W_i})". There seems to be a typo here: \nabla_{W_i} should be \nabla_{W_i} L.
Page 7: "These error functions effectively lose information because simply looking at their gradient is insufficient to determine the label." It would be interesting the compare the efficacy of MAML on these error functions as compared to cross entropy and mean-squared error.
Page 7: "(1) can a learner trained with MAML further improve from additional gradient steps when learning new tasks at test time...? (2) does the inductive bias of gradient descent enable better few-shot learning performance on tasks outside of the training distribution...?". These questions seem unrelated to the universal learning algorithm approximator result that constitutes the main part of the paper. If you're going to study these question empirically, why didn't you also try to investigate them theoretically (e.g. sample complexity and convergence of MAML)? A systematic and comprehensive analysis of these questions from both a theoretical and empirical perspective would have constituted a compelling paper on its own.
Pages 7-8: Experiments. What are the architectures and hyperparameters used in the experiments, and how sensitive are the meta-learning algorithms to their choice?
Page 8: "our experiments show that learning strategies acquired with MAML are more successful when faced with out-of-domain tasks compared to recurrent learners....we show that the representations acquired with MAML are highly resilient to overfitting". I'm not sure that such general claims are justified based on the experimental results in this paper. Generalizing to out-of-domain tasks is heavily dependent on the specific level and type of drift between the old and new distributions. These properties aren't studied at all in this work.   
POST AUTHOR REBUTTAL: After reading the response from the authors and seeing the updated draft, I have decided to upgrade my rating of the manuscript to a 6. The universal learning algorithm approximator result is a nice result, although I do not agree with the other reviewer that it is a  "significant contribution to the theoretical understanding of meta-learning," which the authors have reinforced (although it can probably be considered a significant contribution to the theoretical understanding of MAML in particular). Expressivity of the model or algorithm is far from the main or most significant consideration in a machine learning problem, even in the standard supervised learning scenario. Questions pertaining to issues such as optimization and model selection are just as, if not more, important. These sorts of ideas are explored in the empirical part of the paper, but I did not find the actual experiments in this section to be very compelling. Still, I think the universal learning algorithm approximator result is sufficient on its own for the paper to be accepted.