Meta-learning has achieved great success in few-shot learning. However, the existing meta-learning models have been evidenced to overfit on training tasks when using over-parameterized convolutional neural networks. This means that the generalization performance of the meta-learner cannot be improved by merely applying more complex networks. To remedy this deficiency, we propose a sparsity-constrained meta-learning approach to learn from training tasks a subnetwork from which first-order optimization methods can quickly converge towards the optimal network in testing tasks. Our theoretical analysis shows the benefit of sparsity for reducing the generalization gap of the estimated meta-learner. We have implemented our approach on top of Reptile assembled with varying network pruning routines including Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different over-parameterized deep networks demonstrate that our method not only effectively alleviates meta-overfitting but also in many cases improves the overall generalization performance when applied to few-shot classification tasks.
============= Post Rebuttal Assessment =============
I appreciate the authors thorough response: I believe the writing has improved considerably and the presentation is more convincing.
All in all, I think it's a good paper but not completely ready for publication based on the following final assessment.
The theoretical contribution of the paper is not entirely relevant to the proposed method as also mentioned by other reviewers. The empirical aspect of the work is incremental; by combining two prior works. In this situation, I would happily suggest acceptance if 1) the experimental setup is very thorough and 2) the results are consistent and conclusive 3) the improvements are significant.
Regarding these, the paper's most interesting result is 5-way 5-shot classification with IHT/DSD + Reptile. I think that makes the paper borderline since the results are not state of the art and the experiments are not thoroughly done, while on the other hand, it does show clear improvement for this case. For such a borderline scenario, and since I have to choose between weak reject and weak accept, I would lean towards weak reject as I believe the experimental setup can be significantly improved by following some of the items below for the next version:
- A thorough ablation study on all the components of IHT and DSD
- A study of the effect of the hyperparameters of IHT/DSD
- Additional datasets/architectures to show consistent improvement for various cases.
- ResNet results are not reliable in the current form. I understand that it would need a sizable computational buget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.
To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.
dget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.
To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.
============= Summary =============
 
The paper addresses the issue of overfitting the meta-parameters of optimization-based meta learning techniques. It uses an iterative training-pruning-retraining setup to learn the meta-parameters. The pruning step in each iteration is to limit the capacity of the meta-learner and thereby alleviate overfitting. Two pruning methods called DSD and IHT are employed on top of a first-order meta learning technique called Reptile. The combination is tested on 4 few-shot classification scenarios of two datasets; omniglot and miniImageNet. Results suggests improved accuracy on top of the Reptile meta-learning algorithm on miniImageNet.
 
 
============= Strengths and Weaknesses =============
+ overfitting the meta learner due to the small number of samples (shots) per task and the large number of meta-parameters and/or base-learner parameters is an important problem in meta learning which is the focus of this work.
+ the results suggest improvements over the Reptile baseline on miniImageNet.
 
 
General concerns:
- abstract: “the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks.” several methods such as CAVIA and MetaOptNet (among many others) address this issue by limiting the capacity of the learner.
- what is the formal definition of meta-generalization and meta-overfitting? This definition will be helpful for understanding the paper’s arguments, for instance, why “reducing meta-overfitting” and “improving meta-generalization” are two different matters (as suggested in the last part of the abstract).
- Figure 1.b: the green bars (proposed method) don’t seem to improve the generalization (testing accuracy). I can only speculate that the figure is to demonstrate that (in the rightmost plot) the difference between train and test accuracy for green bars is less than it is for the red bars. However, one should note that it seems to be due to the training accuracy being lower, which can be achieved, in the extreme case, by a random classifier (zero gap). So, it’s not necessarily useful when testing accuracy is not improved.
- eq (1) and eq (2): the left-most loss L should be on D^{query}
- page 4: “In view of the “lottery ticket hypothesis” (Frankle & Carbin, 2018), the model in equation 2 can be interpreted as a first-order meta-learner for estimating a subnetwork“ -> eq (2) in the current form still requires a 2nd-order derivative and I cannot see how (Frankle & Carbin, 2018) helps make it first-order optimization.
- the usefulness of the provided theory is in question since the final method does not enforce the sparsity in practice. That is, the output of the final meta-parameters are *dense*.
- page 7 mentions “top k_l entries”: what does “top” mean here? k_l dimensions of \theta with the highest absolute value, maybe?
- why do the two separate steps of only training the pruned subnetwork using Reptile and then retrain the whole network (with the updated subnetwork) again using Reptile? One can instead only train Reptile on the whole \theta^{(i)}_M (with L0 norm constrained using the M projection) in a single Reptile step. Doing the former should be justified over the latter since its twice as expensive. Also it should be shown empirically (as an ablation study) that the former works better. 
- reference list: arXiv versions of many papers are cited, it’s good to cite the peer-reviewed version of papers when applicable. For instance, IHT pruning method does not seem to be peer-reviewed which is a caveat for the current work.
- 10 pages seem excessive for the content of the paper. For instance, the experiments section can be shortened extensively and the theories can be put into the appendix.
 
 
Experimental concerns:
- Has the standard miniImageNet split been used? This split is especially important to be respected since CAVIA’s accuracy is taken from the original paper. 
- The reported number for CAVIA in table 2 is not the best number the original paper achieve. The best number is 51.82 ± 0.65% for one-shot, and 65.85 ± 0.55% for 5-shot. 
- There are quite a few new hyperparameters (3 iteration numbers for pretraining, pruning/finetuning, and retraining, and then the sparsity level k_l). It’s important to mention how the hyper-parameters are chosen, especially since they are different for different setups. 
- there seems to be no meta validation set for Omniglot.
- For a fair comparison, the network size of the baseline as well as the learning rate should be searched with the same budget as the hyperparameter search done for the proposed method.  
- ResNet experiment is especially concerning since 1) there is an even higher-level of hyperparameter tuning for ResNet: first conv layer is not pruned, different pruning rates for different residual blocks are used. 2) the baseline is only tried with one setting for the capacity. It is evident in Table 2 that there is a sweet spot for the capacity of the ConvNet baseline, there is no reason this does not apply to ResNet.
- Ablation study:  “For the purpose of fair comparison, only the retraining phase is removed and other settings are the same as proposed in Section 5.1”: this is not fair. I believe a fair comparison would be to repeat the hyperparameter search for the ablation studies with the same budget as the full version of the proposed method.
- Table 2 only compares with CAVIA while many other meta learning methods could be relevant here such as MetOptNet which also limits the number of base learner’s learnable parameters thereby helping with overfitting. Also, CAVIA has results on ResNet useful for the last experiment.
 
 
 
============= Final Decision =============
While the paper addresses an important problem and reports improvements, there are many concerns with it including the writing, method, and experimental setup.  My “weak reject” rating, however, is mainly based on the experimental concerns especially regarding the hyperparameters and the baselines.
 
 
============= Minor Points =============
- code is not provided. I think it’s in general very helpful to release the codes for reproducibility and replicability of the experiments, especially in the case of an incremental study.
- what meta learning task has been used for Figure 1? 
- There are different ways of including batchnorm in a meta learning setup. For instance, Reptile used two different strategies. How is batchnorm implemented for this paper’s experiments? Particularly, indicate if you are using transductive or non-transductive version of batchnorm at test time.
 
The text as well as the math notations require polishing on various occasions including the following (non-exhaustive) list:
- abstract: “sparsity constrained” -> sparsity-constrained
- abstract: “can not only” -> rephrase so that it’s not read as “cannot only”
- abstract and intro: “ease meta-overfitting” can be interpreted as facilitating overfitting -> maybe better to say “alleviate meta-overfitting”
- intro: “memorize the experience from previous meta-tasks for future meta-task learning with very few samples” → memorize the experience from previous tasks for a future task with very few samples
- intro: “As can be observed from Figure 1(a) that” -> It can be observed from Figure 1(a) that
- intro: “sparsity benefits considerably to the” → sparsity benefits considerably the
- intro: “along with two specifications to two widely used networking pruning methods”: rephrase
- related works: “most closely” -> closest
- page 4: “ultra goal -> maybe ultimate goal?
- page 4: \theta is not defined
- page 4: eq (2): i -> m or m -> i
- page 4: the definition of an unordered set (S) coming as a sample of the product of m \Tau spaces is not precise. Better to say each T_i \in \Tau.
- page 4: J_l is not defined,
- sec 3.3: the minimization are done over \theta, it’s better to put that under “min” and the conditions should be clearly indicated by “with or s.t. or similar”
- page 7: zero-one -> binary
- page 7: “ as the restriction of \theta^{(t)} over the mask M^{(t)}” -> rephrase
- page 7: “it deems to reduce” -> rephrase
- page 8: “this is consist” -> consistent
 
- it’s also good to settle on a single term for each concept to make it an easier read, for instance:
- “meta-training tasks” and “training tasks” have been used interchangeably. I think “training tasks” and “mata training dataset” are better choices since a “meta-training” task can refer to the whole task of meta learning.
- Meta-overfitting and meta-level overfitting
- meta-learner, meta-estimator, model
- meta-level training accuracy, meta-training accuracy
- meta training, meta learning
- base learner, learner, model → I think it’s better to emphasize on learner being “base learner”, avoid model as it is not clear what it refers to.
 
 
 
============= Points of improvements =============
The paper would significantly improve if 1) the text is polished and the equations revised. 2) hyperparameter optimization is done carefully and thoroughly described for both the proposed method as well as the baselines