Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose META-DATASET: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on META-DATASET, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in META-DATASET. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.
The paper presents Meta-Dataset, a benchmark for few-shot classification that combines various image classification data sets, allows the number of classes and examples per class to vary, and considers the relationships between classes. It performs an empirical evaluation of six algorithms from the literature, k-NN, FineTune, Prototypical Networks, Matching Networks, Relation Networks, and MAML. A new approach combining Prototypical Networks and first-order MAML is shown to outperform those algorithms, but there is substantial room for improvement overall.
Many papers in the literature have observed that current algorithms achieve very high performance on existing few-shot classification data sets such as Omniglot, making it difficult to compare them. This paper fills the need for a more challenging data set. This data set is also more realistic in that
1. The training and test data may come from different data sets. In the real world, we often encounter locations, weather conditions, etc. that are unseen during training, and agents must be able to adapt quickly to these new situations.
2. Classes are selected based on their semantic structure. This allows control of the difficulty of the task, as it is easier to adapt to a semantically similar class, and thus enables a more nuanced comparison of few-shot classification algorithms.
Suggestions for clarifying the writing:
1. Section 3 should also discuss choosing the data set to sample from.
2. In figures 1c and 1d, it would be helpful to include standard error regions, like in the rest of figure 1.
3. Maybe the paragraph on Proto-MAML should be moved to section 3, as it is not from previous literature. In addition, steps 2b and 2c in section 3.2 overlap in content, and so it may be clearer to combine them.