Semi-Supervised Learning with Ladder Networks
We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on top of the Ladder network proposed by Valpola [1] which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification in addition to permutationinvariant MNIST classification with all labels.
The manuscript presents impressive semi-supervised results for MNIST using a simple idea related to denoising autoencoders.The paper proposes a layer-wise reconstruction error of pre-noised activations using a function that is dependent on top down connections and knowledge of the noisy activations at a given layer, i.e. \hat{z}^{l}= g(\noise{z}^{l},\noise{z}^{l+t}).The approach straightforwardly combines this errors into a scalar loss, combines this unsupervised loss with the supervised loss and trains end-to-end.the manuscript also presents good, though slightly less striking, results for the CIFAR dataset using the same approach.It would be nice to see the approach applied to one additional dataset (e.g. demonstrating that it is practical on at least a subset of ImageNet), but I don't believe this is important for acceptance.While the technique is simple the introduction makes it more difficult to follow than need be.The didactic Figure 1 (left panel) in particular could be improved for legibility.Could the choice of the g() function in equation (1) be better justified?Likely I'm missing something obvious, but why does the vector v have duplicate elements (line 190)?It may be helpful to other readers to make this clear.There are also some small miscellaneous grammatical and spelling issues throughout.The title of the paper might might work better if the work network was pluralized: e.g., "Semi-supervised learning with ladder networks".When comparing to past results, it should be made clear which past papers made an effort to "balance classes" for the small subsets used for training.Balancing classes almost certainly helps performance, so it would be good to note which past results had the benefit of balanced classes and which did not.As a control it would also be nice to compare the results presented with a more "vanilla" semi-supervised denoising autoencoder approach using the same architecture (e.g. batch normalization and relu activations).This would give us an idea about how much of the performance gain is due to the new ladder-network idea versus just using slightly more updated training methods.Finally, though the computational costs of the semi-supervised approach seem quite reasonable, but it would be nice to have some measure of the expected wall-clock time/performance for a well optimized version of the ladder network approach versus the purely supervised benchmark.Demonstration that this kind of semi-supervised approach only requires a modest amount of extra wall-time may help foster wider adoption of this kind of mechanism and help us push the through the tendency to use purely-supervised that currently dominates the field.