On the Representational Efficiency of Restricted Boltzmann Machines
This paper examines the question: What kinds of distributions can be efficiently represented by Restricted Boltzmann Machines (RBMs)? We characterize the RBM's unnormalized log-likelihood function as a type of neural network, and through a series of simulation results relate these networks to ones whose representational properties are better understood. We show the surprising result that RBMs can efficiently capture any distribution whose density depends on the number of 1's in their input. We also provide the first known example of a particular type of distribution that provably cannot be efficiently represented by an RBM, assuming a realistic exponential upper bound on the weights. By formally demonstrating that a relatively simple distribution cannot be represented efficiently by an RBM our results provide a new rigorous justification for the use of potentially more expressive generative models, such as deeper ones.
The paper provides a theoretical analysis of the types 
      of distributions that can efficiently be represented by restricted 
      Boltzmann machines (RBMs). The analysis is based on a representation of 
      the unnormalized log probability (free energy) of RBMs as a special form 
      of neural network (NN). The paper relates these RBM networks to more 
      common types of NNs whose properties have been studied in the literature. 
      This approach allows the authors to identify two non-trivial examples of 
      functions that can and cannot be represented efficiently by RBM networks - 
      and hence related distributions can / cannot be modeled efficiently by 
      RBMs. Specifically they show that RBM networks can efficiently represent 
      any function that only depends on the number of non-zero visible units, 
      such as parity, but that they are unable to represent the only somewhat 
      more difficult example of inner product parity. I think the 
      paper addresses an interesting and important problem: RBMs have received a 
      lot of attention in the recent literature but their properties are still 
      not well understood. Many applications of RBMs refer to the result that 
      RBMs are universal approximators of binary distributions, but these 
      results give no indication what distributions can be *efficiently* 
      represented. This question is largely unanswered. Against this 
      background, the paper makes two interesting contributions: 
      Firstly, although the paper clearly does not "solve" the problem 
      of characterizing the efficiently representable distributions I think it 
      nevertheless adds a useful piece to the puzzle: It provides two concrete 
      examples of meaningful and non-trivial distributions that can and cannot 
      be represented efficiently by RBMs. These results are, in my eyes, 
      different and complementary to previous works on this topic, which have 
      either focused on the universal approximator properties of RBMs, or have 
      characterized the set of distributions that can be represented by RBMs in 
      a very abstract manner. Secondly, the method of the proof, which 
      is based on an analysis of the free energy function as a special type of 
      NN, is quite different from the approaches taken in previous works that I 
      have seen: By establishing connections between the special type of RBM 
      networks and NN more widely used in the literature the paper makes it 
      possible to apply established results from the analysis of such networks. 
      This (to my knowledge) new perspective is interesting and might enable 
      further progress in understanding RBMs and related architectures in the 
      future. Despite its theoretical nature I find the paper to be well 
      written. It provides a good overview of previous work on the 
      representational capabilities of RBMs. The results of the paper are well 
      structured and explained, and should be accessible to most of the NIPS 
      community. One concern is, however, that all proofs had to be deferred to 
      the appendix, so maybe it'd be more appropriate to turn this into a longer 
      journal paper -- where the technical aspects can be properly checked (I 
      only skimmed some of the supplemental material). Other points: 
      > Maybe, a few more words could be spent on explaining why the 
      particular functions being analyzed in sections 3 and 4 are interesting 
      examples to study. For IP you have a sentence dedicated to this question 
      (lines 382-384), but I think it would be useful to expand on this. 
      > I believe the universality results, for which you cite Le 
      Roux & Bengio (2008) have already been established earlier in the tech 
      report by Freund & Haussler (1994): "Unsupervised learning of 
      distributions of binary vectors using two layer networks" (UCSC-CRL-94-25) 
      > I am wondering to what extent it is going to be possible to 
      adapt the analysis technique in the paper to deeper architectures (e.g. 
      Deep Boltzmann Machines). Here, the latent variable cannot be marginalized 
      analytically so it is not immediately obvious to me how they could be 
      translated into easy-to-analyze networks.