We consider visual domains in which a class label specifies the content of an image, and class-irrelevant properties that differentiate instances constitute the style. We present a domain-independent method that permits the open-ended recombination of style of one image with the content of another. Open ended simply means that the method generalizes to style and content not present in the training data. The method starts by constructing a content embedding using an existing deep metric-learning technique. This trained content encoder is incorporated into a variational autoencoder (VAE), paired with a to-be-trained style encoder. The VAE reconstruction loss alone is inadequate to ensure a decomposition of the latent representation into style and content. Our method thus includes an auxiliary loss, leakage filtering, which ensures that no style information remaining in the content representation is used for reconstruction and vice versa. We synthesize novel images by decoding the style representation obtained from one image with the content representation from another. Using this method for data-set augmentation, we obtain state-of-the-art performance on few-shot learning tasks.
In any domain involving classification, entities are distinguished not only by class label but also by attributes orthogonal to class label. For example, if faces are classified by identity, within-class variation is due to lighting, pose, expression, hairstyle; if masterworks of art are classified by the painter, within-class variation is due to choice of subject matter. Following tradition (Tenenbaum & Freeman, 2000), we refer to between-and within-class variation as content and style, respectively. What constitutes content is defined with respect to a task. For example, in a face-recognition task, identity is the content; in an emotion-recognition task, expression is the content. There has been a wealth of research focused on decomposing content and style, with the promise that decompositions might provide insight into a domain or improve classification performance. Decompositions also allow for the synthesis of novel entities by recombining the content of one entity with the style of another. Recombinations are interesting as a creative exercise (e.g., transforming the musical composition of one artist in the style of another) or for data set augmentation.
We propose an approach to content-style decomposition and recombination. We refer to the method as STOC, for Style Transfer onto Open-Ended Content. Our approach is differentiated from past work in the following ways. First, STOC can transfer to novel content. In contrast, most previous work assumes the content classes in testing are the same as those in training. Second, STOC is general purpose and can be applied to any domain. In contrast, previous work includes approaches that leverage specific domain knowledge (e.g., human body pose). Third, STOC has an explicit objective, leakage filtering, designed to isolate content and style. No such explicit objective is found in most previous work, and as a result, synthesized examples may fail to preserve content as style is varied and vice versa. Fourth, STOC requires a labeling of entities by content class, but explicit style labels are not required. In contrast, some previous work assumes supervised training of both style and content representations. Figure 1 shows examples of content-style recombination using STOC on the VGG-Face (Parkhi et al.,  2015)  data set. In each column, the content of the image in the top row is combined with the style of the image on the bottom row to synthesize a novel image, shown in the middle row. The images in the top and bottom rows are of identities (content) held out from the training set. Style is well maintained, and content is fairly well transferred, at least to the degree that the faces in the middle row are more similar to top-row than bottom-row faces. The training faces are labeled by identity, but style is induced by the training procedure. 
In any domain involving classification, entities are distinguished not only by class label but also by attributes orthogonal to class label. For example, if faces are classified by identity, within-class variation is due to lighting, pose, expression, hairstyle; if masterworks of art are classified by the painter, within-class variation is due to choice of subject matter. Following tradition (Tenenbaum & Freeman, 2000), we refer to between-and within-class variation as content and style, respectively. What constitutes content is defined with respect to a task. For example, in a face-recognition task, identity is the content; in an emotion-recognition task, expression is the content. There has been a wealth of research focused on decomposing content and style, with the promise that decompositions might provide insight into a domain or improve classification performance. Decompositions also allow for the synthesis of novel entities by recombining the content of one entity with the style of another. Recombinations are interesting as a creative exercise (e.g., transforming the musical composition of one artist in the style of another) or for data set augmentation.
We propose an approach to content-style decomposition and recombination. We refer to the method as STOC, for Style Transfer onto Open-Ended Content. Our approach is differentiated from past work in the following ways. First, STOC can transfer to novel content. In contrast, most previous work assumes the content classes in testing are the same as those in training. Second, STOC is general purpose and can be applied to any domain. In contrast, previous work includes approaches that leverage specific domain knowledge (e.g., human body pose). Third, STOC has an explicit objective, leakage filtering, designed to isolate content and style. No such explicit objective is found in most previous work, and as a result, synthesized examples may fail to preserve content as style is varied and vice versa. Fourth, STOC requires a labeling of entities by content class, but explicit style labels are not required. In contrast, some previous work assumes supervised training of both style and content representations. Figure 1 shows examples of content-style recombination using STOC on the VGG-Face (Parkhi et al., 2015) data set. In each column, the content of the image in the top row is combined with the style of the image on the bottom row to synthesize a novel image, shown in the middle row. The images in the top and bottom rows are of identities (content) held out from the training set. Style is well maintained, and content is fairly well transferred, at least to the degree that the faces in the middle row are more similar to top-row than bottom-row faces. The training faces are labeled by identity, but style is induced by the training procedure.
SUMMARY
The paper considers several methods for building generative models that disentangle image content (category label) and style (within-category variation). Experiments on MNIST, Omniglot, and VGG-Faces demonstrate that the proposed methods can learn to generate images combining the style of one image and the content of another. The proposed method is also used as a form of learned data augmentation, where it improves one-shot and low-shot learning on Omniglot.
Pros:
- The paper is well-written and easy to follow
- The proposed methods CC, CE, PM, and LF are all simple and intuitive
- Improving low-shot learning via generative models is an interesting and important direction
Cons:
- No comparison to prior work on generation results
- Limited discussion comparing the proposed methods to other published alternatives
- No ablations on Omniglot or VGG-Faces generation
- Low-shot results are not very convincing
COMPARISON WITH PRIOR WORK
There have been many methods that propose various forms of conditional image generation in generative models, such as conditional VAEs in [Sohn et al, 2015]; there have also been previous methods such as [Siddharth et al, 2017] which disentangle style and content using the same sort of supervision as in this paper. Given the extensive prior work on generative models I was a bit surprised to see no comparisons of images generated with the proposed method against those generated by previously proposed methods. Without such comparisons it is difficult to judge the significance of the qualitative results in Figures 3, 5, and 6. In Figure 3 I also find it difficult to tell whether there are any substantial differences between the four proposed methods.
The proposed predictiability minimization is very related to some recent approaches for domain transfer such as [Tzeng et al, 2017]; I would have liked to see a more detailed discussion of how the proposed methods relate to others.
OMNIGLOT / VGG-FACES ABLATIONS
The final model includes several components - the KL divergence term from the VAE, two terms from LF, and a WGAN-GP adversarial loss. How much do each of these terms contribute the quality of the generated results?
LOW-SHOT RESULTS
I appreciate low-shot learning as a testbed for this sort of disentangled image generation, but unfortunately the experimental results are not very convincing. For one-shot performance on Omniglot, the baseline Histogram Embedding methods achieves 0.974 accuracy which improves to 0.975 using STOC. Is such a small improvement significant, or can it be explained due to variance in other factors (random initializations, hyperparameters, etc)?
For low-shot learning on Omniglot, the proposed method is outperformed by [Antoniou et al, 2017] at all values of k. More importantly, I’m concerned that the comparison between the two methods is unfair due to the use of different dataset splits, as demonstrated by the drastically different baseline accuracies. Although it’s true that the proposed method achieves a proportionally larger improvement over the baseline compared with [Antoniou et al, 2017], the differences in experimental setup may be too large to draw a conclusion one way or the other about which method is better.
OVERALL
Although the paper is well-written and presents several intuitive methods for content/style decomposition with generative models, it’s hard to tell whether the results are significant due to incomplete comparison with prior work. On the generation side I would like to see a comparison especially with [Siddharth et al, 2017]. For low shot learning I think that the proposed method shows some promise, but it is difficult to draw hard conclusions from the experiments. For these reasons I lean slightly toward rejection.
MISSING REFERENCES
Siddharth et al, “Learning Disentangled Representations with Semi-Supervised Deep Generative Models”, NIPS 2017
Sohn, Lee, and Yan, “Learning structured output representation using deep conditional generative models”, NIPS 2015
Tzeng, Hoffman, Darrell, and Saenko, “Adversarial Discriminative Domain Adaptation”, CVPR 2017