META-LEARNING WITH LATENT EMBEDDING OPTIMIZATION
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this lowdimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.
This work presents an extension of the MAML framework for "learning to learn." This extension changes the space in which "inner-loop" gradient steps are taken to adapt the model to a new task, and also introduces stochasticity. The authors validate their proposed method with regression experiments in a toy setting and few-shot classification experiments on mini- and tiered-Imagenet. The latter are well known and competitive benchmarks in few-shot learning.
The primary innovations that distinguish this work from previous gradient-based approaches to meta-learning (namely MAML) are that (i) the initial set of parameters is data-dependent and drawn from a generative distribution; and (ii) the adaptation of model parameters proceeds in a lower-dimensional latent space rather than in the higher-dimensional parameter space. Specifically, model parameters are generated from a distribution parameterized by an adapted latent code at each adaptation step. I find both of these innovations novel.
The experimental results, in which LEO outperforms the state of the art on two benchmarks derived from ImageNet by "comfortable margins," and the ablation study demonstrate convincingly that these innovations are also significant. I also found the curvature analysis and embedding visualization illuminating of the model's function. My one suggestion would be to test the model on realistic data from beyond the image domain, perhaps on something sequential like language (consider the few-shot PTB setting from Vinyals et al. (2016)). I'm aware anecdotally that MAML struggles with adapting RNNs and I wonder if LEO overcomes that weakness.
The paper is clearly written and I had little difficulty in following the algorithmic details, although I'm sure it helped to be familiar with the convoluted meta-learning and inner-/outer- loop frameworks. I recommend it for publication.
Pros:
- Natural, novel extension to gradient-based meta-learning
- state of the art results on two competitive few-shot benchmarks
- good analysis
- clear writing
Cons:
- realistic, high-dim data is only from the image domain
Minor questions for the authors:
- Relation Networks are computationally intensive, although in few-shot learning the sets encoded are fairly small. Can you discuss the computational cost and training time of the full framework?
- What happens empirically when you generate parameters for more than just the output layer in, eg, your convolutional networks?
- What happens if you try to learn networks from scratch through the meta-learning process rather than pre-training and fine-tuning them? Some of the methods you compare against do so, to my understanding.