Variational Autoencoder for Deep Learning of Images, Labels and Captions
A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone.
This paper describes an approach to perform semi-supervised learning on images. The authors specifically consider classification and captioning tasks, although the method is applicable to other tasks that take single images as inputs.  The proposed approach consists of a type of variational autoencoder (VAE) where an encoder maps images to a latent feature space and a decoder then reconstructs images based on these latent features. The latent features are then used as inputs for a classification or text generation model appropriate for the task. This architecture enables semi-supervised training, either by pre-training the autoencoder in an unsupervised way and then training the task-specific model (two-step training) or by training end-to-end the autoencoder and the task-specific model as a single model on labeled examples and only the autoencoder on unlabeled examples (joint training).  For the autoencoder, the proposed approach uses a convolutional network (CNN) as the encoder, a deep generative deconvolutional network (DGDN) as the decoder, for the task-specific models it uses a Bayesian SVM for classification or a gated recurrent unit (GRU) for captioning.  The paper differs from similar approaches described in the literature for its use of stochastic pooling and unpooling in the encoder and the decoder, respectively. Also, it doesn't make use of non-linear activation functions after each layer, as it is usual, rather it uses tanh activation functions to compute stochastic pooling probabilities and in the final Gaussian sampling layer in the encoder. The use of a Bayesian SVM for classification is also unusual, although, together with stochastic unpooling, it was already found in the original DGDN paper. 
The paper is clearly written and the approach is sound, although it appears as a straightforward combination of existing approaches, specifically VAE, DGDN and image captioning by RNN.  The approach includes some architectural details, specifically the stochastic pooling/unpooling and the lack of non-linear activations between convolutions in the encoder and the decoder, which differ from standard approaches described in the literature. The authors don't motivate these choices. It would be advisable if these choices were motivated by theoretical arguments or individual experimental comparisons. On the other hand, the choice of the Bayesian SVM as a classifier, while unusual, is motivated by experimental comparison against the more common softmax classifier.  The experimental section evaluates the model on supervised classification, semi-supervised classification and captioning in comparison with existing methids.  For supervised classification, the paper compares the proposed approach to the DGDN approaches, but not to plain supervised approaches or other semi-supervised approaches that could have provided reasonable baselines. The authors report comparable test errors and faster test speeds to the DGDN methods on standard benchmark datasets. No significance estimates are provided. Since the test errors are close, significance tests could have helped to better assess these results.  For semi-supervised classification, the paper compares the proposed approach to transductive SVM, standard non-convolutional VAE-based semi-supervised classifiers and Ladder network. The MNIST dataset is used. They outperform TSVM and VAE and slightly underperform Ladder network (though still within the significance range, which is provided this time). This result is interesting, although a direct comparison with convolutional VAE should have been performed to assess the usefulness of the architectural variant introduced by this approach. They also compare their method against standard image classification CNNs on Imagenet.  As a side remark,it shall be noted that neither MNIST nor Imagenet are true semi-supervised classification datasets, as in more natural scenarios, such as domain adaptation, the distributions of the labeled and unlabeled examples are often different. Experiments on native semi-supervised datasets would allow a better evaluation of the method robustness to distribution difference.  For semi-supervised captioning, the paper reports experiments on standard dataset with the addition of source images from Imagenet as unlabeled examples. They compare against purely supervised approaches in the literature, obtaining good, though not SOTA results. No significance estimates are given and the table is difficult to read due to lack of bolding.  In conclusion the paper presents an interesting, though not particularly novel, approach that may be valuable and does not present any obvious flaws, but more theoretical motivation or experimental validation would be advisable to assess its worth.