Probabilistic Model-Agnostic Meta-Learning
Meta-learning for few-shot learning entails acquiring a prior over previous tasks and experiences, such that new tasks be learned from small amounts of data. However, a critical challenge in few-shot learning is task ambiguity: even when a powerful prior can be meta-learned from a large number of prior tasks, a small dataset for a new task can simply be too ambiguous to acquire a single model (e.g., a classifier) for that task that is accurate. In this paper, we propose a probabilistic meta-learning algorithm that can sample models for a new task from a model distribution. Our approach extends model-agnostic meta-learning, which adapts to new tasks via gradient descent, to incorporate a parameter distribution that is trained via a variational lower bound. At meta-test time, our algorithm adapts via a simple procedure that injects noise into gradient descent, and at meta-training time, the model is trained such that this stochastic adaptation procedure produces samples from the approximate model posterior. Our experimental results show that our method can sample plausible classifiers and regressors in ambiguous few-shot learning problems. We also show how reasoning about ambiguity can also be used for downstream active learning problems.
This paper presents an extension to the popular metalearning algorithm MAML, in which it is re-cast as inference in a graphical model.  This framing allows samples to be drawn from a model posterior, enabling reasoning about uncertainty and capturing multiple modes of ambiguous data, while MAML can only make a single point estimate of model parameters at test time.  This is shown in several experiments to better capture the characteristic of ambiguous, noisy data than MAML.  Strengths:  + The paper makes a strong point that few shot learning is often too ambiguous to confine to a single-model metalearning paradigm.  Especially with the high level of recent interest in topics such as safe learning, risk-aware learning, and active learning, this is a relevant area of work.  + The graphical model formulation logically follows from the stated goals of the paper, and the inference methods are built on a solid foundation of well-understood recent advances in variational inference and stochastic optimization and appear to be sound.  + The paper is well-written on the whole and easy to follow.  + The experiments do a nice job at showing intuitively various ways that ProMAML can represent uncertainty -- that it reduces uncertainty as more data is seen, that is can accurately predict uncertainty in a single-shot learning case, and that it captures multiple modes in ambiguous situations.   Weaknesses:  - The primary weakness of the paper is that the benefits of being able to represent and reason about model uncertainty are never shown or explored experimentally.  The experiments do a nice job of showing that uncertainty *can* be represented with some degree of fidelity, but it is never used directly for anything.  The paper hints at applications in active learning, risk-aware learning, etc., but never show any of these experimentally.  Not only would a concrete example make the paper more compelling, but having a more objective downstream success metric would make it much easier to interpret the results in the paper -- for example, in the final experiment, is coverage of 2.62 and accuracy of ~68% good enough?  It depends on the downstream use of the results of ProMAML to answer that question.  The authors' response addressed several of my concerns with the new experiments that were shown, particularly my concerns about a "downstream task" that utilizes the uncertainty measure.