RNADE: The real-valued neural autoregressive density-estimator
We introduce RNADE, a new model for joint density estimation of real-valued vectors. Our model calculates the density of a datapoint as the product of onedimensional conditionals modeled using mixture density networks with shared parameters. RNADE learns a distributed representation of the data, while having a tractable expression for the calculation of densities. A tractable likelihood allows direct comparison with other methods and training by standard gradientbased optimizers. We compare the performance of RNADE on several datasets of heterogeneous and perceptual data, finding it outperforms mixture models in all but one case.
Summary ======= The authors present a directed 
      model for estimating the density of continuous random variables by 
      exploiting the chain rule of probability theory. They propose to use 
      particular weight-sharing constraints which have proven useful for 
      modeling discrete data and combine it with mixture density networks. 
      They show that their model can generally outperform large mixtures of 
      Gaussians when applied to image patches, speech signals, and several 
      smaller datasets. Comments ======== Weight sharing 
      -------------- Since the main difference to related work appears 
      to be in the RBM-inspired weight-sharing, it would be interesting to see a 
      more thorough investigation of its effects. While it is clear that it can 
      reduce computational costs, its effects on performance have not been fully 
      explored. One would expect the weight-sharing to reduce overfitting where 
      data is scarce, and to hurt performance where data is plenty. It would 
      therefore be interesting to see some results on the extent to which weight 
      sharing can help and to learn more about regimes where weight sharing 
      might actually hinder performance. Natural image patches 
      --------------------- In Section 4.2, RNADE is shown to perform 
      similarly to a large mixtures of Gaussians when applied to natural image 
      patches. While this already represents a competitive result, I believe 
      that RNADE could fare even better here. Because of a mixture model's 
      inability to represent independencies efficiently, it scales poorly to 
      high dimensional data. I therefore suspect that RNADE would outperform a 
      mixture model when tested on slightly larger patches. The number 
      of training and validation points used for training RNADE also appears to 
      be quite small. 25,000 training points (1,000 batches with 25 data points 
      each) in my experience is not a lot, even for smaller image patches. If 
      the performance does in fact not improve with more data (or, in other 
      words, there is no overfitting with 25,000 training points), then this 
      would also speak for the advantages of using the proposed weight-sharing 
      constraints – which could be a point worth mentioning. 
      Related work ------------ Since several directed 
      models for real valued data already exist which all use Gaussian mixtures 
      to represent conditional distributions and – which may not be obvious 
      – have similar gating mechanisms to predict the mixing weights (Domke 
      et al., 2008; Hosseini et al., 2009; Theis et al., 2012), a comparison 
      with at least one of these models would have been nice. Minor 
      comments -------------- It would be easier to judge the size of 
      the differences in performance in Table 2 and 3 if additional models were 
      included in the comparison, as in Table 1. There appears to be a 
      $\rho_d$ missing in Equation 4. Quality ======= The 
      methods used in the paper are technically sound. The authors provide 
      extensive comparisons of their model and mixture models on several 
      datasets and go to great lengths to ensure that the results are 
      representative by performing 10-fold cross-validation or using very large 
      test sets where possible. Clarity ======= The paper is 
      well written and easy to follow. It also appears to include enough detail 
      to reproduce the results. Originality =========== The 
      paper explores the nontrivial extension of NADE – a model for discrete 
      data – to the continuous case. While related models exist which 
      also model continuous data and which share some similarities with the 
      proposed model, there are also plenty of differences which make this an 
      interesting and original contribution. Significance 
      ============ Density estimation is an important problem underlying 
      many applications. Just like the superior density estimation performance 
      of NADE has proven useful in solving applications involving discrete data 
      (Larochelle & Lauly, 2012), RNADE has the potential to be useful in 
      tackling applications involving continuous 
      signals.