General Stochastic Networks for Classification
We extend generative stochastic networks to supervised learning of representations. In particular, we introduce a hybrid training objective considering a generative and discriminative cost function governed by a trade-off parameter λ. We use a new variant of network training involving noise injection, i.e. walkback training, to jointly optimize multiple network layers. Neither additional regularization constraints, such as 1, 2 norms or dropout variants, nor pooling-or convolutional layers were added. Nevertheless, we are able to obtain state-of-the-art performance on the MNIST dataset, without using permutation invariant digits and outperform baseline models on sub-variants of the MNIST and rectangles dataset significantly.
This paper extends the General Stochastic Network (GSN) models to perform simultaneous generative and discriminative tasks, via a hybrid training objective that balances a data reconstruction and label prediction. The model is trained using the recently introduced walkback algorithm, which yields a generative model by first injecting noise and then learning to remove it, and provides an alternative to other regularization techniques. The model is trained on several variants of MNIST, showing improvement with respect to previous methods. The paper has two main contributions. (i) extension of GSNs to perform simultaneous discriminative and generative task, (ii) introduce a hybrid cost function, with a trade-off parameter that controls the balance between discriminative and generative terms and an annealing scheme. I think that each contribution, taken individually, is relatively minor and incremental, but the overall model is interesting, with potential to drive further improvements, and shown to be effective in several tasks. Pros:- Good numerical results: state-of-the-art results on MNIST permutation invariant ( If I understood correctly, the performance of the network on that dataset would have been the same if the pixels were shuffled, hence it is not using the pixel geometry - if that is the case, I do not understand the terminology 'without permutation invariant digits' ). - Interesting alternative to the classical unsupervised pretraining + discriminative fine-tuning paradigm. Although the annealing scheme effectively performs something similar in spirit, the transition is smoother and the framework is easier to study. The reviewer is however not expert enough to say whether this is a novelty. - Potential for improvement: As the authors mention, the model seems robust to hyperparameters (number of walkback steps and noise level) and prone to generalization to more sophisticated tasks. Cons/Remarks:- Somehow the paper mixes two different aspects that might confuse the reader: (i) how to combine generative with discriminative tasks, and (ii) how to obtain efficient generative models. The paper considers the GSN framework, which is a valid choice, but I would have liked to see a more clear comparison (perhaps a section?) where this model is compared with traditional (noise-free) autoencoders which use a reconstruction term and a discriminative term. The authors mention in the numerical experiments section that the GSN model with no noise also performed well in experiments. Is that equivalent to a standard transforming autoencoder enriched with a discriminative cost? - Additive interactions in the Markov chains. Figures 2 and 3 show the basic architecture of the GSN, which generalizes the Denoising Autoencoder Markov chain. In this extended model, hidden random variables are modeled as sums of transition functions coming from different sources. I would have expected the influence of different sources to have a nonlinear effect in the state of hidden variables, such as multiplicative interactions.- Is the Gaussian additive noise (pre and post activation) sufficient to achieve multimodality? Could the authors describe how this mechanism scales to high-dimensional data (with potentially the number of 'modes' scaling exponentially with the dimension). Section 3.1 describes an analysis of the noise sigma, but sigma is not defined previously (I assumed it is the standard deviation of eta_out and eta_in). Is this noise isotropic? why? - Interpretation of section 3.1. The analysis of the role of walkback steps and the noise level is critical for the understanding of the model, and I wished the authors would give further, deeper insight on the results depicted in Figures 6 and 7. Why does the performance depend so little on K? Has the chain already converged? And also, from Figure 7 is seems as if the noise-free version performs nearly as well as the carefully chosen noisy version. Is this because the noise distributions are too simplistic (Isotropic gaussian, with same variance across the network?), or is there a more fundamental reason?