Modelling statistical relationships beyond the conditional mean is crucial in many settings. Conditional density estimation (CDE) aims to learn the full conditional probability density from data. Though highly expressive, neural network based CDE models can suffer from severe over-fitting when trained with the maximum likelihood objective. Due to the inherent structure of such models, classical regularization approaches in the parameter space are rendered ineffective. To address this issue, we develop a model-agnostic noise regularization method for CDE that adds random perturbations to the data during training. We demonstrate that the proposed approach corresponds to a smoothness regularization and prove its asymptotic consistency. In our experiments, noise regularization significantly and consistently outperforms other regularization methods across seven data sets and three CDE models. The effectiveness of noise regularization makes neural network based CDE the preferable method over previous non-and semi-parametric approaches, even when training data is scarce.
The paper considers the problem of parametric conditional density estimation, i.e. given a set of points {(x_n, y_n)} drawn from a distribution $p(x,y)$, the task is to estimate the conditional distribution p(x|y). The paper considers parametric estimation where in given a parametrized family of distributions f_{theta} we wish to minimize the likelihood of seeing the given data over theta. The parametric family in a lot of applications consists of highly expressive families like neural networks, which leads to the issue of overfitting in small data regimes. This has been tackled via regularization over the parameter space which might be hard to interpret as the associated inductive bias is not well understood and depends on the parametric family under consideration. On the other hand the paper proposes to add explicit noise in the examples used during training, i.e. irrespective of the optimization procedure (which could be mini-bath sgd) the paper proposes to draw examples from the data set, explicitly add noise onto the examples and create a proxy objective over the augmented data set. 
The paper establishes two theoretical results. First is a simple taylor approximation based analysis to highlight the effect of the variance of noise. The conclusion is that higher variance penalizes the high curvature areas of the resultant density and hence this kind of noise addition could be seen as making resulting density smoother. The second contribution is to show that this procedure is asymptotically consistent, i.e. as n goes to infinity and the number of augmented data points go to infinity, the resulting density converges to the target density. This of course requires the noise variance to follow a decreasing schedule to 0. 
The main merit of the idea is in its agnostic nature, as it can be applied to any parametric family and the experiements show that it seems to uniform improvement across models,  The basic idea proposed by the error, has existed in the space of deep learning based methods forever. This is the same idea behind image augmentation which forms a crucial part of training supervised models in vision. The authors claim that this idea is novel in the space of parametric density estimation, however I do not know enough about the area to verify the claim. It would surprise that this very natural idea has not been tried before. 
I have gone through the theoretical derivations in the paper and they look sound to me. However the results are all asymptotic in nature without establishing explicit rates which is a little bit of disappointment. Since I am not completely familiar in nature, but I guess such asymptotic consistency might be achievable using other forms of regularization under suitable assumptions. In that light, the theoretical contributions while being sound did not lend much intuition about why such a method might outperform others. Intuition does arise from the derivation for the effect of noise on the objective which helps understand the nature of the noise, but one wonders if similar intuitions could be derived for other forms of regularization as well. It would be great to see this derivation being extended to some concrete scenarios under well understood parametric families and seeing the effect explicitly. 
Regarding the experiments - The experiments definitely look promising as the improvments seem uniformly good across the cases considered. I am not an expert however in this setting so it is hard for me to judge the quality and significance of the benchmarks. The experiment methodology nevertheless looks sound.