Wasserstein Training of Restricted Boltzmann Machines
Boltzmann machines are able to learn highly complex, multimodal, structured and multiscale real-world data distributions. Parameters of the model are usually learned by minimizing the Kullback-Leibler (KL) divergence from training samples to the learned model. We propose in this work a novel approach for Boltzmann machine training which assumes that a meaningful metric between observations is known. This metric between observations can then be used to define the Wasserstein distance between the distribution induced by the Boltzmann machine on the one hand, and that given by the training sample on the other hand. We derive a gradient of that distance with respect to the model parameters. Minimization of this new objective leads to generative models with different statistical properties. We demonstrate their practical potential on data completion and denoising, for which the metric between observations plays a crucial role.
The paper proposes training RBMs using the gamma-smoothed Wasserstein distance W_{gamma}(\hat{p}, p_{theta}) (eqs 3 and 4), where  \hat{p} is the data distribution and p_{theta} is the model.  They  obtain derivatives of W_{gamma}(\hat{p}, p_{theta}) and train  with gradient descent.  However, the shrinkage of the Gaussian example in sec 4.3 (and the results in sec 4.2) gives serious concern wrt the consistency of Wasserstein training and this issue MUST be investigated more carefully before the paper is suitable for publication.
The paper refers to [2] and says that those authors proved statistical consistency. However, I am then surprised to see in section 4.3 that non-zero shrinkage is obtained (including for gamma=0) for the  very simple case of modelling a N(0,I) distribution with  N(0, sigma^2 I). What is going on here?? A failure of consistency  would be a serious flaw in the formulation of a statistical learning criterion.  Also in sec 3 (Stability and KL regularization) the authors say that at least for learning based on samples (\hat{p}_{theta}) that some regularization wrt the KL divergence is required. This clearly weakens the "purity" of the smoothed Wasserstein objective fn.  Experiments (sec 4) are carried out on MNIST-small (the 0s from MNIST), a subset of the UCI PLANTS dataset, and 28-d binary codes (MNIST code). The results (as reported in lines 168-172 and Figs 3 and 4) seem to produce "compact and contiguous regions that are prototypical  of real spread, but are less diverse than the data". This reinforces  my belief above (relating to the Gaussian example insec 4.3) that  minimizing the (smoothed) Wasserstein does not lead to a consistent density estimator.  In sec 4.4 the authors try to "spin" these weakness into strenghts wrt  data completion or denoising.   Overall: The authors are to be congratulated on investigating an interesting alternative to standard KL-based training of RBMs. However, the shrinkage of the Gaussian example in sec 4.3 gives serious concern wrt the consistency of Wasserstein training and this issue MUST be investigated more carefully before the paper is suitable for  publication.  Other points:  Title: the title over claims. The paper only discusses training *restricted* BMs.  Intro l 7: RBMs do not need to be restricted to binary x's, see e.g. Exponential Family Harmoniums  https://papers.nips.cc/paper/2672-exponential-family-harmoniums-with-an-application-to-information-retrieval.pdf  Fig 5 would be much more readable using different colors for the different gammas. Also last entry should be gamma=0 (not OpenCV). You can explain in the text that you used OpenCV to compute this.   === COMMENTS POST REBUTTAL =====  I have read the other reviewers comments and the rebuttal.  I will remove my charge of a "fatal flaw" based on the point that the sample size in the experiments in sec 4.3 is small. However, the KL version of the experiment (where we fit a Gaussian with cov matrix theta^2 I to n = 100 samples drawn from a 10-d Gaussian) the values of theta I get from the max likelihood estimator are all pretty much in the range 0.95-1.05 (for about 10 repetitions). This should be compared  with something around theta = 0.65 (estimated by eye from Fig 5(left))  for the Wasserstein (gamma=0) estimator.  The convergence rate given by the authors of O(n^-1/(10+1)) for  this simple problem is truly appalling, and presumably is  in general O(n^-1/(D+1)) for D dimensions. This gives rise to  serious concerns about the practicality of such a method.   It would be helpful to see a study in this simple Gaussian case, showing e.g. how the theta that minimizes W_0 varies with n, although the rate given above suggests convergence will be  painfully slow. Can you understand in such a case why there is  a shrinkage bias?   I note the response to R5 wrt lines 177-178 i.e.  "the entropy regularizer contributes to the clustering effect [...]. It also appears for the true W distance in Fig 5."  One could use say a KL matching criterion KL(\hat{p}||p_theta) along with an entropy penalty (no Wasserstein), and my understanding from the quote above is that this will cause clumping, whose virtues are extolled in sec 4.4 wrt data completion and denoising.  (Note that due to the KL regularization in section 3 this is partly happening anyway in many of the experiments).  I also agree with other reviwers who found the clarity of technical explanation poor, and suggested use of supp mat to help with this.  Overall, I am not really in favour of accepting this paper (although I could live with acceptance). There is quite a lot of interesting work, but the issue where the clumping comes from (Wasserstein or entropic prior) needs to be better articulated and dissected. I believe some of claims wrt Wasserstein training are actually to do with the entropic prior, and this should be teased out by comparing against a KL + entropic prior construction.