Semi-supervised Learning with Deep Generative Models
The ever-increasing size of modern data sets combined with the difficulty of obtaining label information has made semi-supervised learning one of the problems of significant practical importance in modern data analysis. We revisit the approach to semi-supervised learning with generative models and develop new models that allow for effective generalisation from small labelled data sets to large unlabelled ones. Generative approaches have thus far been either inflexible, inefficient or non-scalable. We show that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.
This paper builds on the recently introduced variational autoencoder (VAE) framework of Kingma and Welling (2014) and of Rezende et al. (2014) to introduce two separate semi-supervised learning models. The first (called M1) is a VAE version of self-taught learning (Raina et al., 2007), while the second (M2) blends a standard discriminative model with a conditionalgenerative VAE that models the input data given the class label. I am  conflicted about this paper. On the positive side I find the VAE a very interesting and important modeling framework and I find the idea of extending this framework to the semi-supervised domain to be both natural and sensible. I would expect this to be the first in a long series of papers exploring this direction. On the negative side, the specific two proposed approaches seem fairly straightforward. This isn't a strong criticism but it does impact the novelty of the paper.  It would have been better if the experiments really focused on the differences between the two, in order to give the reader some idea of their differential abilities, but that wasn't really the case as they were only really compared directly in one set of experiments limited to the MNIST hand-written digits database. More seriously, there are a large number of similar methods thatare not discussed and not compared empirically. In particular there are many papers not cited discussing learning in the self-taught learning framework (including the original self-taught learning paper of Raina et al. (2007)) which is essentially identical to the proposed M1 approach. Other missing citations and comparisons are discussed below inmore specific comments on the experiments.The empirical evaluation of the proposed approach is based on three types of experiments:1 - Benchmark Classification: The first is a basic comparison of the proposed approach in a semi-supervised version of the MNIST hand-written digits database. These results are based entirely on the experiments of Rifai et al. (2011) where the authors have simply added the results for the proposed approach. This isn't really a problem, except that they neglect to provide all of the results from Rifai et al. (2011). While they include the results for 100, 600 and 1000 labeled examples. They do not provide results for either 3000 or 5000 (all) labeled examples. Why do they neglect to provide these 2 numbers? I think it's important to see how the proposed methods (M1 and M2) work in the larger labeled set regime. 2 - Conditional Generation: The second set of experiments explore the ability of the model to separately model label content and other more stylistic variations in two datasets: SVHN (street-view house numbers) and the NORB object dataset. Here the authors provide a set of qualitative results -- sampled images with variable "style" and "content" (label info). These results aremildly compelling in that they capture -- for SVHN -- the color information of both the background and foreground but do not seem to capture the font style. There are also several papers discussing the separation of style and content that are either cited nor compared. These include: Tenenbaum and Freeman (2000), "Separating Style and Content with Bilinear Models";  Grimes and Rao (2005) "Bilinear sparse coding for invariant vision"; Rifai et al (2012) "Disentangling factors of variation for facial expression recognition"; Desjardins et al. (2012) "Disentangling Factors of Variation via Generative Entangling" and Reed et al. (2014) "Learning to Disentangle Factors of Variation with Manifold Interaction"3 - Image Classification: The third set of experiments explore semi-supervised classification with SVHN and NORB image datasets. The authors state that there are no published results on these datasets in the semi-supervised setting so they compare to their own results using nearest neighbors and Transductive SVM. While I know of no results on these twodatasets in the semi-supervised regime, there are plenty of results on other image classification datasets with published results in the semi-supervised regime. Examples include: CIFAR-10 object recognitiondataset, Caltech-101, and the STL-10 dataset. Experiments with these dataset can be found in the following four papers (among others):Dai and Van Gool (2013) "Ensemble Projection for Semi-supervised Image Classiﬁcation" Goodfellow et al. (2012) "Large-Scale Feature Learning With Spike-and-Slab Sparse Coding"Coates et al. (2011) "An Analysis of Single-Layer Networks in Unsupervised Feature Learning"Coates and Ng (2011) "Selecting receptive fields in deep networks"