Learning generative probabilistic models that can estimate the continuous density given a set of samples, and that can sample from that density, is one of the fundamental challenges in unsupervised machine learning. In this paper we introduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by a neural network, that is efficiently trained to represent a kernel density estimator of the data. In addition, we show how to leverage DDEs to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not require specific network architectures like in normalizing flows, ODE solvers as in continuous normalizing flows, nor do we require adversarial training as in generative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.
This paper presents an approach for learning density estimates of a data distribution convolved with noise through “denoising density estimators” and shows how to leverage these density estimates to train generative samplers. The methods are evaluated on toy low-d datasets, MNIST, and fashion MNIST where they show reasonable density estimates and OK sample quality. 
My major concern with this paper is that the “denoising density estimator” proposed here is identical to denoising score matching (which is not cited or discussed). The second contribution of learning a sampler given a density estimate is interesting but likely suffers from all the instabilities of GAN training, and does not compare to related work on distilling energy-based models. Unless both these concerns are addressed, I cannot recommend this paper for acceptance.
Major comments:
* The idea of learning a generative model whose energy gradient (score) matches the gradient of the data distribution has been explored extensively under the name “score matching”. The proposed approach of “Deep Denoising Density Estimation” is *identical* to denoising score matching (Vincent et al., 2011, http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf). There’s no discussion of any prior work on score matching in this paper, or comparison with recent approaches in this space (e.g. sliced score matching https://arxiv.org/abs/1905.07088, and https://arxiv.org/abs/1907.05600 that uses denoising score matching).
* The algorithm presented for learning a generator given an energy function is not compared to other implicit sampling approaches like MCMC. Additionally, the algorithm requires alternating density estimation with updating the generative model, which is quite similar to GAN-training alternating density ratio estimation with updating the generative model. Thus the proposed algorithm likely experiences similar instabilities and challenges in how to partially solve the density estimation step. There are no comparisons to any GAN-based approaches anywhere in the paper. That said, mixing DSMs and explicit generators is a neat area of research.
* The experiments are too limited, and do not compute any quantitative metrics on the image datasets.
Minor comments:
* “Defining property of PGMs is that they provide functionality to sample” ->  what about density estimates? Likelihood ratios? etc.
* Boltzmann machines don’t allow for efficient inference
* Intro could should spend more time discussing relation to energy-based models, score matching, noise-contrastive estimation
* Eqn 2 only holds for sigma^2 -> 0, please add and discuss this limitation
* Divergenc -> divergence, above eqn 12
* I found the math in section 4 very confusing. What’s the <> notation in this context? Do you need to introduce \Delta? Alternating density (ratio) estimation and generative model updates is super common in GAN literature and is not discussed here.
* Table 1: are the estimates of log-likelihood upper or lower bounds? Unlike other approaches, you don’t have exact likelihoods so I’m not sure your #s are comparable.
* 2048 samples per iteration -> batch size?
* “For faster convergence, we take 10 DDE gradient steps…”  -> please add an experiment showing how results are impacted by # of gradient descent steps. Is larger # steps always better?
* MNIST/Fashion MNIST samples aren’t that good to my eye and there are no quantitative metrics (e.g. KID, FID, IS)