Empirical Localization of Homogeneous Divergences on Discrete Sample Spaces
In this paper, we propose a novel parameter estimator for probabilistic models on discrete space. The proposed estimator is derived from minimization of homogeneous divergence and can be constructed without calculation of the normalization constant, which is frequently infeasible for models in the discrete space. We investigate statistical properties of the proposed estimator such as consistency and asymptotic normality, and reveal a relationship with the information geometry. Some experiments show that the proposed estimator attains comparable performance to the maximum likelihood estimator with drastically lower computational cost.
This paper proposes a new method to derive an estimator for discrete value parameters in probabilistic models, which does not require the calculation of the normalizing constant. This avoids a computational bottle neck, which is often of exponential order and makes many problems infeasible. This new estimator minimizes a risk function defined as the localized pseudo-spherical (PS) divergence between the empirical distribution of the dataset and the unnormalized probabilistic model. This risk function can be made to be convex with properly chosen parameters. The paper shows that its proposed estimator is asymptotically efficient and additionally gives an intuitive explanation of this by showing the localized PS divergence can be interpreted as a mixture of alpha-divergences. The method is applied to fully visible and hidden variable Boltzmann machines and the estimator is compared with the MLE.The quality of this paper is very good. It is well written, organized, and technically strong. I think there is a typo in Remark 9 on line 337, which should be just alpha > 0 in the parentheses so that sgn(alpha) doesn't matter.Additionally it would be nice if the simulations compared (both of RMSE and computation time) the proposed estimator to estimators from other methods [5], [6], and [10] instead of just the MLE. There are several issues that should be dealt with.In line 352, the experiment on fully observed Boltzmann machine is done with data dimension d=10. This is not very practical. Note that the claim of this paper is the efficiency, so it should tested on a larger dimensional example, e.g. d greater than 100.Furthermore, depending on the modeling of Boltzmann machine in line 82 and line 88, it is not fully convincing that this method is ready to be applied in deep neural network with large dimension of hidden layer. Furthermore, the author's result only applies to exponentially families, and may not necessarily hold in a mixture family like the Boltzmann machine with hidden layers.The critical assumption for the model to work is that the empirical distribution is restricted on a finite subregion $\cZ$, while the model is defined on domain $\set{0,1}^{n} \supset \cZ$.The authors should make this clear in the statement of their theory. It is noted that in the last line of page 3, the author introduces a trick to compute the weighted normalization factor as\sum_{x}\ind{x\in \cZ}\frac{n_{x}}{n}q(x). It should be mentioned that in generalBoltzmann machine is a mixture model that does not satisfy the constraint that there exists a domain Z \subset \set{0,1}^{n}, such that \frac{n_{x}}{n} = 0, x\not\in Z.   There are some other confusing typos:1. In line 225, r_{\alpha, \theta_{0}}(x) = \bar{q}_{\theta_{0}} (x) should be r_{0}, \theta_{0}}(x) = \bar{q}_{\theta_{0}} (x) as confusing typo. Same typo is in supplementary material. 2. In line 225, "By expanding an equilibrium condition" is not clearly explained in either main text and supplementary material. Could it just be the Taylor expansion?This should be clarified.3. Supplementary material line 92, a typo, \langle \phi \epsilon\rangle should be \langle \psi'_{\theta_{0}} \epsilon\rangle.