While meta-learning approaches that utilize neural network representations have made progress in few-shot image classification, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model architectures have become increasingly specialized to the few-shot domain. A natural question that arises is how to develop learning systems that scale from few-shot to many-shot settings while yielding competitive performance in both. One scalable potential approach that does not require ensembling many models nor the computational costs of relation networks, is to meta-learn an initialization. In this work, we study first-order meta-learning of initializations for deep neural networks that must produce dense, structured predictions given an arbitrary amount of training data for a new task. Our primary contributions include (1), an extension and experimental analysis of first-order model agnostic meta-learning algorithms (including FOMAML and Reptile) to image segmentation, (2) a novel neural network architecture built for parameter efficiency and fast learning which we call EfficientLab, (3) a formalization of the generalization error of meta-learning algorithms, which we leverage to decrease error on unseen tasks, and (4) a small benchmark dataset, FP-k, for the empirical study of how meta-learning systems perform in both fewand many-shot settings. We show that meta-learned initializations for image segmentation provide value for both canonical few-shot learning problems and larger datasets, outperforming random and ImageNet-trained initializations for up to 400 densely labeled examples. Finally, we show both theoretically and empirically that a key limitation of MAML-type algorithms is that when adapting to new tasks, a single update routine is used that is not conditioned on the available data for a new task. We find that our network, with an empirically estimated optimal update procedure yields state of the art results on the FSS-1000 dataset, while only requiring one forward pass through a single model at evaluation time.
This paper proposes to apply MAML-style meta-learning to few-shot semantic segmentation in images. It argues that this type of algorithm may be more computationally-efficient than existing methods and may offer better performance with a higher number of examples. They further propose to perform hyper-parameter search to choose a new learning rate for the inner learning process after optimizing for the network parameters.
As far as I know, this is the first paper to apply gradient-based (i.e. MAML-style) meta-learning to this specific problem. Existing approaches to few-shot semantic segmentation have mostly used multi-branch conv-networks to condition the output on the training examples. This paper shows that (FO)MAML achieves similar accuracy to the FSS-1000 baseline. This is an empirical contribution in itself. The paper also demonstrates that the EfficientNet architecture can be applied to segmentation.
Major concerns:
(1.1) The improvement obtained by the hyper-parameter optimization seems quite marginal (79.0 - 81.4 and 73.3 - 73.9) and there is no study of the variance of the results. The fact that better performance is obtained by tuning the learning rate on the *training* set suggests to me that the improvement might not be significant. You could repeat the experiment by sampling multiple different training and testing sets (with different classes) to estimate (some of) the variance.
(1.2) The formalization in Section 4 is mathematically appealing but seems unnecessary. In the end, the paper is essentially arguing that it's better to use different learning rates (for the inner loop) during meta-training and meta-testing. This seems obvious, since this includes equal learning rates as a special case. The paper proposes to optimize the latter learning rate using a gradient-free method. This argument can be made without considering generalization bounds.
(1.3) One of the central claims of the paper is that "meta-learned representations smoothly transition as more data becomes available." It's not entirely clear what this means. I suppose it means that, with few shots, it should perform as well as existing few-shot methods, but with many shots, it should perform as well as a standard learning algorithm. The paper failed to present any evidence that existing algorithms for few-shot segmentation do not satisfy this property. It would strengthen the argument to include an existing few-shot segmentation algorithm in Figure 2.
(1.4) The details of the experiment in Figure 2 are not clear. Was a different number of iterations used when there are hundreds of examples? Were different hyper-parameters used when optimizing from a pre-trained checkpoint? It would be unfair to use the same hyper-parameters which had been optimized specifically for the meta-learned initialization.
Other issues:
(2.1) It's not clear what it means to achieve human-level performance in the few-shot task and the many-shot task. What is human-level performance at few-shot segmentation? It seems to me that humans are capable of segmenting novel objects (i.e. zero-shot) with almost perfect accuracy. Does this mean that your method should achieve the same accuracy with few- and many-shots?
(2.2) The use of early stopping was unclear. Do you use a fixed number of SGD iterations during training and a variable number of iterations (determined by a stopping criterion) during testing? However, this seems to be contradicted by the statement that the UHO algorithm determined an optimal number of iterations (8) for testing? On the other hand, this seems like too few iterations with hundreds of shots. Maybe the automatic stopping criterion was only used with many shots? Or maybe training proceeds until either the stopping criterion is satisfied or the maximum number of iterations is reached? Furthermore, the early stopping criterion was not specified.
(2.3) Missing reference: Meta-SGD (arxiv 2017) considers a different learning rate for every parameter and updates the learning rates during meta-training .
(2.4) There was no discussion of the running time of different methods. This would be particularly interesting in the many-shot regime. How slow are the RelationNet approaches?
(2.5) It is claimed that Figure 1 demonstrates that  "the estimated optimal hyperparameters for the update routine ... are not the same as those specified a priori during meta-training". However, it seems that the optimal learning rate is awfully close to the dotted blue line (within the variance of the results).
(2.6) For the IOU loss (equation 10), what are the predicted y values? Are they arbitrary real numbers? Do you use a sigmoid to constrain them to real numbers in [0, 1]?
Minor:
(3.1) It is not worth stating the optimal learning rate to more than 3 or 4 significant figures.
(3.2) Use 1 \times 1 instead of 1 x 1.
(3.3) Is there a reference for the Dice score? Where does the name come from?