Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs
Sparse high-dimensional data vectors are common in many application domains where a very large number of rarely non-zero features can be devised. Unfortunately, this creates a computational bottleneck for unsupervised feature learning algorithms such as those based on auto-encoders and RBMs, because they involve a reconstruction step where the whole input vector is predicted from the current feature values. An algorithm was recently developed to successfully handle the case of auto-encoders, based on an importance sampling scheme stochastically selecting which input elements to actually reconstruct during training for each particular example. To generalize this idea to RBMs, we propose a stochastic ratio-matching algorithm that inherits all the computational advantages and unbiasedness of the importance sampling scheme. We show that stochastic ratio matching is a good estimator, allowing the approach to beat the state-of-the-art on two bag-of-word text classification benchmarks (20 Newsgroups and RCV1), while keeping computational cost linear in the number of non-zeros.
Summary: This paper develops an algorithm that 
      can successfully train RBMs on very high dimensional but sparse input 
      data, such as often arises in NLP problems. The algorithm adapts a 
      previous method developed for denoising autoencoders for use with RBMs. 
      The authors present extensive experimental results verifying that their 
      method learns a good generative model; provides unbiased gradient 
      estimates; attains a two order of magnitude speed up on large sparse 
      problems relative to the standard implementation; and yields state of the 
      art performance on a number of NLP tasks. They also document the curious 
      result that using a biased version of their estimator in fact leads to 
      better performance on the classification tasks they tested. Pros: 
      While the underlying idea is present in previous work with 
      autoencoders, transferring this idea to the RBM setting required some 
      non-trivial insights, particularly into the ratio-matching objective for 
      training RBMs. Every claim is supported by an appropriate figure 
      (learns good generative models; gradient estimates are unbiased; 
      importance sampler proposal distribution selects large-gradient samples; 
      attains a large speed up relative to standard implementation; yields state 
      of the art performance on certain NLP tasks). The experiments have been 
      done to a high standard. The paper includes some useful other 
      results, including a highly intuitive form for the ratio matching 
      objective, and a method for saving intermediate variables that allows more 
      computationally efficient computation of the free energy terms required by 
      the ratio matching objective. The paper is very clearly written. 
      Cons: While the proposed method does achieve 
      state-of-the-art results, its performance is highly similar to that of the 
      denoising autoencoder. There is no real case made in the paper that the 
      RBM algorithm is particularly more useful than the already developed DAE 
      version of the algorithm. The experimental results do not give 
      quantitative classification performance results for the unbiased SRM 
      algorithm (e.g. Tables 2-3). While it is stated in the text that the 
      unbiased method was not the top performer, it would still be informative 
      to get a sense of the magnitude of the difference. It would also be 
      valuable to clarify if the denoising autoencoders were trained with or 
      without biased gradient estimates. Since biased gradients work better for 
      the RBM, it seems likely this is true of the autoencoder as 
      well.