META-LEARNING WITH DIFFERENTIABLE CLOSED-FORM SOLVERS
Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.
This paper proposes a meta-learning approach for the problem of few-shot classification. Their method, based on parametrizing the learner for each task by a closed-form solver, strikes an interesting compromise between not performing any adaptation for each new task (as is the case in pure metric learning methods) and performing an expensive iterative procedure, such as MAML or Meta-Learner LSTM where there is no guarantee that after taking the few steps prescribed by the respective algorithms the learner has converged. For this reason, I find that leveraging existing solvers that admit closed-form solutions is an attractive and natural choice. 
Specifically, they propose ridge regression as their closed-form solver (R2-D2 variant). This is easily incorporated into the meta-learning loop with any hyperparameters of this solver being meta-learned, along with the embedding weights as is usually done. The use of the Woodbury equation allows to rewrite the closed form solution in a way that scales with the number of examples instead of the dimensionality of the features; therefore taking advantage of the fact that we are operating in a few-shot setting. While regression may seem to be a strange choice for eventually solving a classification task, it is used as far as I understand due to the availability of this widely-known closed-form solution. They treat the one-hot encoded labels of the support set as the regression targets, and additionally calibrate the output of the network (via a transformation by a scale and bias) in order to make it appropriate for classification. Based on the loss of ridge regression on the support set of a task, a parameter matrix is learned for that task that maps from the embedding dimensionality to the number of classes. This matrix can then be used directly to multiply the embedded (via the fixed for the purposes of the episode embedding function) query points, and for each query point, the entry with the maximum value in the corresponding row of the resulting matrix will constitute the predicted class label.
They also experimented with a logistic regression variant (LR-D2) that does not admit a closed-form solution but can be solved efficiently via Newton’s Method under the form of Iteratively Reweighted Least Squares. When using this variant they restrict to tackling the case of binary-classification.
A question that comes to mind about the LR-D2 variant: while I understand that a single logistic regression classifier is only capable of binary classification, there seems to be a straightforward extension to the case of multiple classes, where one classifier per class is learned, leading to a total of N one-vs-all classifiers (where N is the way of the episode). I’m curious how this would compare in terms of performance against the ridge regression variant which is naturally multi-class. This would allow to directly apply this variant in the common setting and would enable for example still oversampling classes at meta-training time as is done usually.
I would also be curious to see an ablation where for the LR-D2 variant SGD was used as the optimizer instead of Newton’s method. That variant may require more steps (similar to MAML), but I’m curious in practice how this performs.
A few other minor comments:
- In the related work section, the authors write: “On the other side of the spectrum, methods that optimize standard iterative learning algorithms, [...] are accurate but slow.” Note however that neither MAML nor MetaLearner LSTM have been showed to be as effective as Prototypical Networks for example. So I wouldn’t really present this as a trade-off between accuracy and speed.
- I find the term multinomial classification strange. Why not use multi-class classification?
- In page 8, there is a sentence that is not entirely grammatically correct: ‘Interestingly, increasing the capacity of the other method it is not particularly helpful’.
Overall, I think this is good work. The idea is natural and attractive. The writing is clear and comprehensive. I enjoyed how the explanation of meta learning and the usual episodic framework was presented. I found the related work section thorough and accurate too. The experiments are thorough as well, with appropriate ablations to account for different numbers of parameters used between different methods being compared. This approach is evidently effective for few-shot learning, as demonstrated on the common two benchmarks as well as on a newly-introduced variant of cifar that is tailored to few-shot classification. Notably, the ridge regression variant can reach results competitive with SNAIL that uses significantly more weights and is shown to suffer when its capacity is reduced. Interestingly, other models such as MAML actually suffer when given additional capacity, potentially due to overfitting.