A natural solution for one-shot learning is to augment training data to handle the data deficiency problem. However, directly augmenting in the image domain may not necessarily generate training data that sufficiently explore the intra-class space for one-shot classification. Inspired by the recent vocabulary-informed learning, we propose to generate synthetic training data with the guide of the semantic word space. Essentially, we train an auto-encoder as a bridge to enable the transformation between the image feature space and the semantic space. Besides directly augmenting image features, we transform the image features to semantic space using the encoder and perform the data augmentation. The decoder then synthesizes the image features for the augmented instances from the semantic space. Experiments on three datasets show that our data augmentation method effectively improves the performance of one-shot classification. Extensive study shows that data augmented from semantic space are complementary with those from the image space, and thus boost the classification accuracy dramatically. Source code and dataset will be available.
Summary:
This paper proposes a data augmentation method for one-shot learning of image classes. This is the problem where given just one labeled image of a class, the aim is to correctly identify other images as belonging to that class as well. 
The idea presented in this paper is that instead of performing data augmentation in the image space, it may be useful to perform data augmentation in a latent space whose features are more discriminative for classification. One candidate for this is the image feature space learned by a deep network. However they advocate that a better candidate is what they refer to as "semantic space" formed by embedding the (word) labels of the images according to pre-trained language models like word2vec. The reasoning here is that the image feature space may not be semantically organized so that we are not guaranteed that a small perturbation of an image vector will yield image vectors that correspond to semantically similar images (belonging to the same class). On the other hand, in this semantic space, by construction, we are guaranteed that similar concepts lie near by each other. Thus this space may constitute a better candidate for performing data augmentation by small perturbations or by nearest neighbour search around the given vector since 1) the augmented data is more likely to correspond to features of similar images as the original provided image and 2) it is more likely to thoroughly capture the intra-class variability in the augmented data.
The authors propose to first embed each image into a feature space, and then feed this learned representation into a auto-encoder that handles the projection to and from the semantic space with its encoder and decoder, respectively. Specifically, they propose to perform the augmentation on the semantic space representation, obtained from the encoder of this autoencoder. This involves producing some additional data points, either by adding noise to the projected semantic vector, or by choosing a number of that vector's nearest neighbours. The decoder then maps these new data points into feature space, obtaining in this way the image feature representations that, along with the feature representation of the original (real) image will form the batch that will be used to train the one-shot classifier.
They conduct experiments in 3 datasets where they experiment with augmentation in the image feature space by random noise, as well as the two aforementioned types of augmentation in the semantic space. They claim that these augmentation types provide orthogonal benefits and can be combined to yield superior results.
Overall I think this paper addresses an important problem in an interesting way, but there is a number of ways in which it can be improved, detailed in the comments below. 
Comments:
-- Since the authors are using a pre-trained VGG for to embed each image, I'm wondering to what extent they are actually doing one-shot learning here. In other words, the test set of a dataset that is used for evaluation might contain some classes that were also present in the training set that VGG was originally trained on. It would be useful to clarify whether this is happening. Can the VGG be instead trained from scratch in an end-to-end way in this model?
-- A number of things were unclear to me with respect to the details of the training process: the feature extractor (VGG) is pre-trained. Is this finetuned during training?  If so, is this done jointly with the training of the auto-encoder? Further, is the auto-encoder trained separately or jointly with the training of the one-shot learning classifier? 
-- While the authors have convinced me that data augmentation indeed significantly improves the performance in the domains considered (based on the results in Table 1 and Figure 5a), I am not convinced that augmentation in the proposed manner leads to a greater improvement than just augmenting in the image feature domain. In particular, in Table 2, where the different types of augmentation are compared against each other, we observe similar results between augmenting only in the image feature space versus augmenting only in the semantic feature space (ie we observe that "FeatG" performs similarly as "SemG" and as "SemN"). When combining multiple types of augmentation the results are better, but I'm wondering if this is because more augmented data is used overall. Specifically, the authors say that for each image they produce 5 additional "virtual" data points, but when multiple methods are combined, does this mean 5 from each method? Or 5 overall? If it's the former, the increased performance may merely be attributed to using more data. It is important to clarify this point.
-- Comparison with existing work: There has been a lot of work recently on one-shot and few-shot learning that would be interesting to compare against. In particular, mini-ImageNet is a commonly-used benchmark for this task that this approach can be applied to for comparison with recent methods that do not use data augmentation. Some examples are:
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. (Finn et al.)
- Prototypical Networks for Few-shot Learning (Snell et al.)
- Matching Networks for One-shot Learning (Vinyals et al.)
- Few-Shot Learning Through an Information Retrieval Lens (Triantafillou et al.)
-- A suggestion: As future work I would be very interested to see if this method can be incorporated into common few-shot learning models to on-the-fly generate additional training examples from the "support set" of each episode that these approaches use for training.