Generalized Denoising Auto-Encoders as Generative Models
Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued. This has led to various proposals for sampling from this implicitly learned density function, using Langevin and Metropolis-Hastings MCMC. However, it remained unclear how to connect the training procedure of regularized auto-encoders to the implicit estimation of the underlying datagenerating distribution when the data are discrete, or using other forms of corruption process and reconstruction errors. Another issue is the mathematical justification which is only valid in the limit of small corruption noise. We propose here a different attack on the problem, which deals with all these issues: arbitrary (but noisy enough) corruption, arbitrary reconstruction loss (seen as a log-likelihood), handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise (or non-infinitesimal contractive penalty).
This paper develops a step towards the first useful 
      formulation of a denoising autoencoder, a particular flavor of neural 
      network which implicitly captures the structure of the data by reproducing 
      the input at its output, as a generative model - i.e. as a probabilistic 
      model defining a distribution over the data. In particular, this paper 
      develops a methodology for drawing samples from the data distribution 
      implicitly captured by the denoising autoencoder. The paper does not 
      develop a formulation of denoising autoencoders as generative models 
      directly nor a method to extract the probability under the model of a 
      novel data case. Nevertheless, this is a step towards the solution to a 
      longstanding problem in the autoencoder and deep learning literature. 
      Restricted Boltzmann machines, have existed as generative models of the 
      data for many years but have remained difficult to use as such in 
      practice, particularly for non-binary valued data. Thus, this work may 
      offer a useful alternative for continuous valued data. From the 
      revelations of the paper, the authors develop a new training algorithm for 
      denoising autoencoders which they claim is better than existing methods - 
      this is however not empirically validated. The paper is well written, 
      structured and clear. The methodology and formulation appears to be 
      correct and the paper offers some interesting discussion and intuition. 
      The empirical validation is somewhat weak given the claims made in the 
      paper. For example, the empirical justification of the walkback algorithm 
      introduced in the paper is purely qualitative and only on one example. 
      Given the wide practical applicability of denoising autoencoders, and a 
      vast literature of corresponding quantitative empirical benchmarks, it 
      seems strange to not see a quantitative comparison between denoising 
      autoencoders trained with the walkback algorithm compared to those 
      without. Also, the only quantitative empirical result (line 404) would 
      suggest that the walkback algorithm in fact does not improve 
      (quantitatively) at all over the standard procedure despite some 
      additional cost. This is unfortunate as it seems that the walkback 
      algorithm should really be empirically validated for the claim that it 
      reduces spurious modes to be justified. Thus it seems unclear whether the 
      sampling algorithm introduced in the paper can be used to actually improve 
      autoencoder training. On line 350, the authors state that they 
      observe in the empirical evaluation that the walkback algorithm is more 
      efficient than the standard training algorithm - however, this is nowhere 
      to be found in the empirical evaluation section. Quality: The 
      overall quality of ideas, writing and concepts is high. The empirical 
      evaluation could be significantly more thorough. Clarity: The 
      paper is extremely clear, well written and structured. 
      Originality: The work is certainly original and attempts to 
      address a heretofore unsolved problem with denoising autoencoders. The 
      algorithm for sampling from the data density captured by the autoencoder 
      is neat and novel. Significance: This is a possibly 
      significant step towards solving a longstanding problem in the autoencoder 
      literature. However, some of the theoretical aspects of the paper are not 
      sufficiently empirically validated. It is especially not clear from the 
      empirical validation whether the 'walkback algorithm' is actually 
      practically useful. The ability to draw samples from the data distribution 
      is very neat but it would be useful to have some significant empirical 
      justification of the practical utility of this. Much remains to be solved 
      to use denoising autoencoders as generative models - thus the title may be 
      somewhat misleading. It seems the paper claims much more than is actually 
      justified by empirical evaluation.