Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of the VAE. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without having to force it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data points, we introduce an ex-post density estimation step that can be readily applied to the proposed framework as well as existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules.
This paper propose an extension to deterministic autoencoders. Motivated from VAEs, the authors propose RAEs, which replace the noise injection in the encoders of VAEs with an explicit regularization term on the latent representations. As a result, the model becomes a deterministic autoencoder with a L_2 regularization on the latent representation z. To make the model generalize well, the authors also add a decoder regularization term L_REG. In addition, due to the encoder in RAE is deterministic, the authors propose several ex-post density estimation techniques for generating samples.
The idea of transferring the variational to deterministic autoencoders is interesting. Also, this paper is well-written and easy to understand. However, in my opinion, this paper needs to consider more cases for autoencoders and needs more rigorous empirical and theoretical study before it can be accepted. Details are as follow:
1. The RAEs are motivated from VAEs, or actually CV-VAEs as in this paper. More precisely, the authors focus on VAEs with a constant covariance Gaussian distribution as the variational distribution and a Gaussian distribution with the identity matrix as the covariance matrix as the model likelihood. However, there might be many other settings for VAEs. For example, the model likelihood can be a Gaussian distribution with non-constant covariance, or even some other distributions (e.g. Multinomial, Bernoulli, etc). Similarly, the variational distribution can be a Gaussian distribution with non-constant covariance, or even some more complicated distributions that do not follow the mean-field assumption. Any of these more complex models may not be easily transferred to the RAE models that are mentioned in this paper. Perhaps it is better if the authors can consider RAEs for some more general VAE settings.
2. Perhaps the authors needs more empirical study, especially on the gain of RAE over CV-VAE and AE. 
a) As the motivated model (CV-VAE) and the most related model in the objective (AE), they are not appearing in the structured input experiment (Section 6.2). It will be great if they can be compared with in this experiment.
b) The authors did not show us clearly whether the performance gain of RAE over VAE, AE and CV-VAE is due to the regularization on z (the term L_z^RAE) or the decoder regularization (the term L_REG) in the experiments. In table 1, the authors only compare the standard RAE with RAE without decoder regularization, but did not compare with RAE without the regularization on z (i.e. equivalent to AE + decoder regularization) and CV-VAE + decoder regularization. The authors would like to show that the explicit regularization on z is better than injecting the noise, hence the decoder regularization term should appear also in the baseline methods. It is totally possible that perhaps AE + decoder regularization or CV-VAE + decoder regularization perform better than RAE. 
c) The authors did not show how they tune the parameter \sigma for CV-VAE. Since the parameter \beta in the objective of RAE is tunable, for fair comparison, the authors needs to find the best \sigma for CV-VAE in order to get the conclusion that explicit regularization is better than CV-VAE.
d) Although the authors mention that the 3 regularization techniques perform similarly, from Table 1, it is still hard to decide which one should we use in practice in order to get a performance at least not too much worse compared to the baseline methods. RAE-GP and RAE-L2 perform not well on CelebA while RAE-SN perform not well on MNIST, compared to the baseline methods. We know that the best performance over the 3 methods is always comparable to or better than the baselines, but not none of the single methods do. It is better if the authors can provide more suggestions on the choice for decoder regularization for different datasets.
3. The authors provided a theoretical derivation for the objective L_RAE (Equation 11), but this is only for the L_GP regularization. Besides, this derivation (in Appendix B) has multiple technique issues. For example, in the constraints in Equation 12, the authors wrote ||D_\theta(z1) - D_\theta(z2)|| < epsilon for all z1, z2 ~ q_\phi(z | x), this is impossible for CV-VAE since this constraint requires D_theta() to be bounded while q_\phi(z | x) in CV-VAE has an unbounded domain. Moreover, in the part  (||D_\theta(z1) - D_\theta(z2)||_p=\nabla D_\theta(\tilde z)\cdot ||z_1-z_2||_p) of Equation 13, \nabla D_\theta(\tilde z) is a vector well the other two terms are scalars, which does not make sense. There are many other issues as well. Please go through the proof again and solve these issues.
Questions and additional feedback:
1. Can the authors provide more intuitions why do you think the explicit regularization works better compared to the noise injection? Can you provide a theoretical analysis on that?
2. Can the authors provide some additional experiments as mentioned above? Also, can the authors provide more details about how do they tune the parameters \beta and \lambda?
========================================================================================================
After the rebuttal:
Thanks the authors for the detailed response and the additional experiments. I agree that the additional experiment results help to support the claims from the authors, especially for the CV-VAE for the structured data experiments and the AE + L2 experiment. So I think now the authors have more facts to support that RAE is performing better compared to the baseline methods. 
Therefore, I agree that after the revision, the proposed method RAE is supported better empirically. So I am changing my score from "weak reject" to "weak accept". But I still think the baseline CV-VAE + regularization is important for Table 1 and the technical issues in the theoretical analysis needs to be solved. Hope the authors can edit them in the later version.