∆-encoder: an effective sample synthesis method for few-shot object recognition
Learning to classify new categories based on just one or a few examples is a long-standing challenge in modern computer vision. In this work, we propose a simple yet effective method for few-shot (and one-shot) object recognition. Our approach is based on a modified auto-encoder, denoted ∆-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it. The synthesized samples are then used to train a classifier. The proposed approach learns to both extract transferable intra-class deformations, or "deltas", between same-class pairs of training examples, and to apply those deltas to the few provided examples of a novel class (unseen during training) in order to efficiently synthesize samples from that new class. The proposed method improves the state-of-the-art of one-shot object-recognition and performs comparably in the few-shot case.
* The authors have contributed equally to this work Corresponding author: Leonid Karlinsky (leonidka@il.ibm.com) Figure 1: Visualization of two-way one-shot classification trained on synthesized examples. Correctly classified images are framed in magenta (Golden retriever) and yellow (African wild dog). The only two images seen at training time and used for sample synthesis are framed in blue. Note the non-trivial relative arrangement of examples belonging to different classes handled successfully by our approach. The figure is plotted using t-SNE applied to VGG features.
*** After rebuttal: Thank you for the additional experiment, this satisfies my curiosity and convinces me more of the inability of a linear offset to achieve similar results. It is also encouraging that the reported results compare favourably to the recent paper by Gidaris et al. Finally, the provided explanation fully clarify my understanding of the algorithm, and I feel it's a nice and simple idea. I've raised my score to 7 as a result of all the above.  One thing that would be important to clarify is that the feature extractor is trained exclusively on the meta-training set of each dataset considered. Another reviewer misunderstood this fact, and I remember also misunderstanding this during my first pass of the paper. I think the confusion might be due to using the word 'pretrained' which may be interpreted as reusing a feature extractor that was trained on another dataset(s), instead of merely describing a two-stage training phase where first the feature extractor is trained, and then held fixed while the encoder / decoder are trained. It would be useful to reword the relevant parts to avoid misunderstandings.  *** Original review: This paper proposes a method for few-shot classification of novel categories that belongs to the family of data augmentation. Specifically, the problem of few-shot classification requires constructing a classifier to distinguish between N new classes, of which only a few examples are available. The data augmentation approach to the problem aims to generate or synthesize additional ‘training’ examples for learning this N-way classifier, directly addressing the data insufficiency which is the cause of the difficulty of this setup. The proposed method, named Delta-encoder, can be used to generate additional examples of a new category given only a single ‘anchor’ example of that category. Intuitively, it works by creating a latent representation of a transformation that can be performed to the anchor example to transform it into something that still plausibly belongs to the same class as the anchor. In other words, it learns what transformations are ‘allowable’ for maintaining the ‘class-yness’ of the anchor example. This idea is the same as in [17], but the authors here propose a different method for achieving it. Notably, while [17] applied linear offsets to the anchor to generate new examples, this proposed method employs a non-linear function for learning these allowable transformations.  More specifically, Delta-encoder has an auto-encoder structure where the encoder takes a pair of examples that are known to belong to the same class and is encouraged by its training objective to use the latent space for encoding the ‘deformation’ between those two training examples. Specifically, this latent representation in addition to one of the initial two training examples are used by the decoder to reconstruct the other example. The dimensionality of the latent code is small enough to disallow this structure to function as a standard autoencoder where one of the two examples is merely reconstructed from itself, and the other is ignored. Therefore, what the latent code learns is how one of the two examples needs to be modified in order to be transformed into the other. The hope is that these ‘allowable class-preserving deformations’ that are learned between examples of training classes also are useful class-preserving deformations between instances of new classes.    Pros - Well-written, clearly conveys the problem setup and the proposed method - The proposed model is interesting and novel, to the best of my knowledge, and tackles an important problem - They show experimentally that this method performs very well on a wide range of standard datasets. - The section on ablation studies justifies the incremental changes performed on top of a Denoising Autoencoder model (for zero-shot learning) that was the inspiration for the Delta-Encoder.  Cons - While [17] and [43] seems to be the most related works, there is no experimental comparison to these methods. I feel this is quite important, and having this additional data point would significantly impact my opinion. - While a “linear offset” version is experimented with in the ablation studies section, I find that the particular linear offset version used there is a particular special case. Specifically, instead of computing the latent code as the vector difference between the anchor point and the input, this could be computed as the output of a linear layer that these two vectors as input (with a trainable weight and bias). I think it would be useful to quantify how well the best possible linear version fairs against Delta-Encoder. - While the paper overall is clearly written, I found lines 126-129 somewhat unclear. In particular, if I understand correctly these lines describe how the Delta-encoder is used at few-shot learning time. In particular, at this time, a single example of each new class is used as the anchor (if k > 1 examples are available of a class, they will in turn be used as the anchor) and the decoder produces additional examples that are hypothesized to belong to this new class, by taking as input this anchor and a Z (the latent code that encodes the deformation to be applied to the anchor). The part that is unclear to me, which is described by these lines, is how this Z is obtained. The quoted lines explain that a number of pairs of examples are encoded into Z’s, where each pair belongs to a training class (not one of the N new classes). This gives a set of { Z_i }’s. The part that is missing is how these are combined to form the Z that will condition the generation of examples of a new class. Is it an average that’s taken? In that case, would it make sense to just take the average over all encoded training pairs (which can be computed during the autoencoder training). - It should be noted that the representation-learning architecture used here is significantly larger than it is in many methods being compared to, casting doubts about the apples-to-apples-ness of these comparisons. Maybe an additional asterisk or note can be included in the table to indicate this (similar to how other aspects are indicated). In particular, Meta-Learner LSTM, MAML, Matching Networks, and Prototypical Networks all use a simple 4-layer CNN architecture.   Suggestions - I think that including a description of the overall algorithm in the section that explains the method would really aid the reader in understanding the process. For example something like the following: Given a dataset, the following phases are performed. Phase 1: Train a standard classifier on the set of training classes (often referred to as the meta-training set) of the dataset (all-way classification). These learned weights function as the pre-trained representation used in what follows. Phase 2: Train the encoder and decoder weights. As I understand it, this amounts to randomly sampling pairs of examples that have the same label, and training the autoencoder as described above. Phase 3: Few-shot learning phase. At this point, a ‘test episode’ is presented that contains k examples of each of N new classes. Each of these k examples of a given class will in turn be considered as the ‘anchor’ and the decoder will generate additional images of that class given this anchor and the latent code representing a deformation. Then, given all (real and synthesized) examples, an N-way classifier is trained for this episode. - Regarding Figure 2: A more descriptive and less confusing name for phase 2 (on the right of the figure) might be 'few-shot learning phase' instead of ‘sample synthesis phase’. In particular, as I understand it 'sample synthesis' is the process of decoding, which happens in both phases. I think the key part that distinguishes the two phases is that the former trains the encoder and decoder, while the latter applies the decoder to generate examples of new classes, given only an anchor for these classes.  - In related work, according to the proposed distinction between metric learning and meta-learning, Prototypical Networks should be in this meta-learning category too, instead of the metric learning one. Its training objective follows exactly the same structure as Matching Networks, the only difference being the choice of the conditional classifier's form.