The ability to learn new concepts with small amounts of data is a critical aspect of intelligence that has proven challenging for deep learning methods. Meta-learning has emerged as a promising technique for leveraging data from previous tasks to enable efficient learning of new tasks. However, most meta-learning algorithms implicitly require that the meta-training tasks be mutually-exclusive, such that no single model can solve all of the tasks at once. For example, when creating tasks for few-shot image classification, prior work uses a per-task random assignment of image classes to N-way classification labels. If this is not done, the meta-learner can ignore the task training data and learn a single model that performs all of the meta-training tasks zero-shot, but does not adapt effectively to new image classes. This requirement means that the user must take great care in designing the tasks, for example by shuffling labels or removing task identifying information from the inputs. In some domains, this makes meta-learning entirely inapplicable. In this paper, we address this challenge by designing a meta-regularization objective using information theory that places precedence on data-driven adaptation. This causes the meta-learner to decide what must be learned from the task training data and what should be inferred from the task testing input. By doing so, our algorithm can successfully use data from non-mutually-exclusive tasks to efficiently adapt to novel tasks. We demonstrate its applicability to both contextual and gradientbased meta-learning algorithms, and apply it in practical settings where applying standard meta-learning has been difficult. Our approach substantially outperforms standard meta-learning algorithms in these settings.
################################################################################
Summary:
This paper illustrates, identifies, and formally defines a memorization problem in meta-learning -- the model can simply memorize meta-training tasks and ignore meta-training train sets. The paper proposes to optimize the mutual information between testing predictions and the training data (given input and meta model), and upper bound it by imposing a information bottleneck between output and input+model. Unlike related work, this paper specifically is able to generalize to meta-test even when the meta-train dataset is not made confusing enough (i.e. even when model can learn well from test data in meta-train alone), making it applicable to use cases where it is hard to make the dataset confusing.
################################################################################
Decision
I vote for accepting this paper, since as far as I know this paper gives a novel insight to the overfitting problem in meta learning, and has formulated the problem formally with theoretical insight, and given a working solution with strong experiment results and clean comparative studies. Despite somewhat narrow experiments and sometimes confusing writing, the paper should provide new insight to meta-learning.
################################################################################
Pros:
! DISCLAIMER: I am not an expert in this field, so take my novelty judgements with a grain of salt.
+ Novel view into meta-learning's overfitting problem
(1) Large models can simply memorize which input data corresponds to which task, and memorize the meta-training tasks, without being able to generalize into meta-test tasks.
(2) Formulates this into a low mutual information between meta-train training data and predictions.
+ Easy to implement and quite widely applicable as a regularization loss addon to multiple existing meta-learning methods
+ Impact-wise, the paper takes meta-learning further from memorization, making methods more capable of operating on less-carefully designed, more natural datasets (rather than permutation of datasets)
+ Experiments are clean with ablation studies and hyperparameter sensitivity tests, and method performs well across real and toy datasets
+ Motivation part is easy to read
################################################################################
Cons:
- I'm still skeptical of the novelty of the paper, since the conclusions are, in hindsight, very straightforward.
- Sometimes sentences are very confusing to readers.
(1) The term "mutually-exclusive" is confusing because the view-point example the paper gives seems to be mutually exclusive (each task has its own kind of data, hence "exclusive"). It is unclear whether the task data is exclusive, or the task function is exclusive, and not straightforward to see its relationship with memorization. Consider renaming it to e.g. "mutually-confusing" or "mutually-contradictory mapping".
(2) Can you please rename "information-theoretic meta-regularizer" to "meta-regularizer using information theory"? It is hard to read for non-native speakers.
(3) Paragraph under definition 1 is confusing and has redundancies.
(4) Section 4.1, not very clear how the logic goes from the decomposition to adding the upper bound to the loss, and how the other term comes in.
- For the motivation, it is better to give examples of use cases when it is impossible to make meta-train "mutually-exclusive". I'm sure even in the patient example you can shuffle classes or input dimensions.
- An experiment comparing to others in mutually-exclusive datasets would be nice to have, in order to judge how much this compensates a badly-designed meta-learning dataset.
################################################################################
Improvements:
- Clarify each point in the "Cons" section.
- Please also clarify if all methods in all experiments are hyperparameter-tuned separately, i.e. that the experiments are not favoring the MR-* model in any way. (the paper only clarifies it in one of the experiments)
- For future work, does recent developments in mutual information modeling (e.g. MINE https://arxiv.org/abs/1801.04062) help this method in any way? e.g. try increasing mutual information between some representation of the meta-train training data and some feature vectors before the prediction?
- First parenthesis in Section 4.1 has a misplaced space
################################################################################
Post rebuttal
################################################################################
It seems that reviewers agree that the contributions are novel (regardless of whether each reviewer thinks it is trivial). So that addresses my main concern. I think the contributions are novel enough since it gives theoretical guidance as well. Other concerns are mostly addressed by the rebuttal. I will keep my rating.
Although I do urge the authors to reconsider the name choice "mutual-exclusive tasks" since it is not very informative and quite confusing to readers.