An important research direction in machine learning has centered around developing meta-learning algorithms to tackle few-shot learning. An especially successful algorithm has been Model Agnostic Meta-Learning (MAML), a method that consists of two optimization loops, with the outer loop finding a meta-initialization, from which the inner loop can efficiently learn new tasks. Despite MAML's popularity, a fundamental open question remains -is the effectiveness of MAML due to the meta-initialization being primed for rapid learning (large, efficient changes in the representations) or due to feature reuse, with the meta initialization already containing high quality features? We investigate this question, via ablation studies and analysis of the latent representations, finding that feature reuse is the dominant factor. This leads to the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML where we remove the inner loop for all but the (task-specific) head of the underlying neural network. ANIL matches MAML's performance on benchmark few-shot image classification and RL and offers computational improvements over MAML. We further study the precise contributions of the head and body of the network, showing that performance on the test tasks is entirely determined by the quality of the learned features, and we can remove even the head of the network (the NIL algorithm). We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.
This paper is exploring the importance of the inner loop in MAML. It shows that using the inner loop only for the classifier head (ANIL) results are comparable to MAML. It also shows that using no inner loop at all (NIL) is okay for test time but not for training time.
It is indeed interesting to understand the effect of the inner loop. But, as the authors noted (“Our work is complementary to methods extending MAML, and our simplification and insights could be applied to such extensions also”), for it to be useful I’d like to see whether these insights can be extended to SOTA models. MAML is less than 50% accuracy on 1-shot mini-imagenet while current SOTS models achieve 60-65%.
The NIL experiment that shows low performance when no inner loop is used in training time doesn’t make sense. This is basically the same as the nearest-neighbour family of methods, e.g. ProtoNet (Snell et al., 2017), which have been shown to perform similarly to (or even better than) MAML.
After rebuttal:
I do think it's important to also have that kind of analysis works. My main concern is with how ANIL and NIL are introduced as new algorithms and not just an ablation of the MAML method. Presented as new algorithms I tend to compare them against the leaderboard where they are very far from the top. I am keeping my previous rating.