META-LEARNING PROBABILISTIC INFERENCE FOR PREDICTION
This paper introduces a new framework for data efficient and versatile learning. Specifically: 1) We develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction. ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods. 2) We introduce VERSA, an instance of the framework employing a flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of shots, and outputs a distribution over task-specific parameters in a single forward pass. VERSA substitutes optimization at test time with forward passes through inference networks, amortizing the cost of inference and relieving the need for second derivatives during training. 3) We evaluate VERSA on benchmark datasets where the method sets new state-of-the-art results, handles arbitrary numbers of shots, and for classification, arbitrary numbers of classes at train and test time. The power of the approach is then demonstrated through a challenging few-shot ShapeNet view reconstruction task.
Summary:
This work tackles few-shot (or meta) learning from a probabilistic inference viewpoint. Compared to previous work, it uses a simpler setup, performing task-specific inference only for single-layer head models, and employs an objective based on predictive distributions on train/test splits for each task (rather than an approximation to log marginal likelihood). Inference is done amortized by a network, whose input is the task training split. The same network is used for parameters of each class (only feeding training points of that class), which allows an arbitrary number of classes per task. At test time, inference just requires forward passes through this network, attractive compared to non-amortized approaches which need optimization or gradients here.
It provides a clean, decision-theoretic derivation, and clarifies relationships to previous work. The experimental results are encouraging: the method achieves a new best on 5-way, 5-shot miniImageNet, despite the simple setup. In general, explanations in the main text could be more complete (see questions). I'd recommend shortening Section 4, which is pretty obvious.
- Quality: Several interesting differences to prior work. Well-done experiments
- Clarity: Clean derivation, easy to understand. Some details could be spelled out better
- Originality: Several important novelties (predictive criterion, simple model setup, amortized inference network). Closely related to "neural processes" work, but this happened roughly at the same time
- Significance: The few-shot learning results are competitive, in particular given they use a simpler model setup than most previous work. I am not an expert on these kind of experiments, but I found the comparisons fair and rather extensive
Interesting about this work:
- Clean Bayesian decision-theoretic viewpoint. Key question is of course whether
   an inference network of this simple structure (no correlations, sum combination
   of datapoints, same network for each class) can deliver a good approximation to
   the true posterior.
- Different to previous work, task-specific inference is done only on the weights of
   single-layer head models (logistic regression models, with shared features).
   Highly encouraging that this is sufficient for state-of-the-art few-shot classification
   performance. The authors could be more clear about this point.
- Simple and efficient amortized inference model, which along with the neural
   network features, is learned on all data jointly
- Optimization criterion is based on predictive distributions on train/test splits, not
   on the log marginal likelihood. Has some odd consequences (question below),
   but clearly works better for few-shot classification
Experiments:
- 5.1: Convincing results, in particular given the simplicity of the model setup and
   the inference network. But some important points are not explained:
   - Which of the competitors (if any) use the same restricted model setup (inference
      only on the top-layer weights)? Clearly, MAML does not, right? Please state this
      explicitly.
   - For Versa, you use k_c training and 15 test points per task update during
      training. Do competitors without train/test split also get k_c + 15 points, or
      only k_c points? The former would be fair, the latter not so much.
- 5.2: This seems a challenging problem, and both your numbers and reconstructions
   look better than the competitor. I cannot say more, based on the very brief
   explanations provided here.
   The main paper does not really state what the model or the likelihood is. From
   F.4 in the Appendix, this model does not have the form of your classification
   models, but psi is input at the bottom of the network. Also, the final layer has
   sigmoid activation. What likelihood do you use?
   One observation: If you used the same "inference on final layer weights" setup
   here, and Gaussian likelihood, you could compute the posterior over psi in closed
   form, no amortization needed. Would this setup apply to your problem?
Further questions:
- Confused about the input to the inference network. Real Bayesian inference would
   just see features h_theta(x) as inputs, not the x's. Why not simply feed features in
   then?
   Please do improve the description of the inference network, this is a major
   novelty of this paper, and even the appendix is only understandable by reading
   other work as well. Be clear how it depends on theta (I think nothing is lost by
   feeding in the h_theta(x)).
- The learning criterion based on predictive distributions on train/test splits seem
   to work better than ELBO-like criteria, for few-shot classification.
   But there are some worrying aspects. The marginal likelihood has an Occam's
   razor argument to prevent overfitting. Why would your criterion prevent overfitting?
   And it is quite worrying that the prior p(psi | theta) drops out of the method
   entirely. Can you comment more on that?
Small:
- p(psi_t | tilde{x}_t, D_t, theta) should be p(psi_t | D_t, theta). Please avoid a more
   general notation early on, if you do not do it later on. This is confusing