Legendre Decomposition for Tensors
We present a novel nonnegative tensor decomposition method, called Legendre decomposition, which factorizes an input tensor into a multiplicative combination of parameters. Thanks to the well-developed theory of information geometry, the reconstructed tensor is unique and always minimizes the KL divergence from an input tensor. We empirically show that Legendre decomposition can more accurately reconstruct tensors than other nonnegative tensor decomposition methods.
Main ideas of the submission  The manuscript presents an approximation of nonnegative multi-way tensorial data (or high-order probability mass functions) based on structured energy function form that minimizes the Kullback-Leibler divergence. Comparing against other multilinear decomposition methods of nonnegative tensors, the proposal approach operates on multiplicative parameters under convex objective function and converges to a globally optimal solution. It also shows interesting connections with graphical models such as the high-order Boltzmann machines. Two optimization algorithms are developed, based upon gradient and natural gradient, respectively. The experiment shows that under the same number of parameters, the proposed approach yields smaller RMSEs than the other two baseline non-negative tensor decomposition methods.      Although the authors frame the proposed approach under tensor decomposition and draw a connection to statistical models, I found that it is easier to interpret it from the perspective of graphical models. Basically, the main technical point seems to be the incorporation of partial ordering constraints on the inclusion of interaction terms such that the model complexity of higher-order Boltzmann machines is under control. It seems the tensor decomposition view tends to overcomplicate the idea, and I’m not quite convinced by its benefits.   Meanwhile, the experiments seem problematic and not compelling.    First, the capacity of tensor models not only depends on the number of parameters but also on the operations on them. Usually, the low-dimensional components could also be interpretable, and able to capture certain underlying structure of data (e.g., multi-linear). Moreover, we do not always want an accurate reconstruction of the observed tensor (e.g., denoising). In this manuscript, the possible interactions are restricted by a partial order that looks quite artificial. In practice, would this ordering naturally arise? What kind of characteristic of the multi-variate density do we expect the Legendre decomposition to capture?    Second, it seems the two baseline methods considered in the experiments are both based on least-square (LS) objective functions and multiplicative updates (Lee and Seung, 1999), while this manuscript adopts KL divergence. The LS objective may not be appropriate for non-negative smaller values. In addition, those algorithms could converge to a non-KKT point (Gonzalez and Zhang, 2015). Therefore, the worse performance of competitors could either be attributed to the objective function or the multiplicative updates. Without additional experimental evidence, it is hard to say whether the proposed method truly outperforms the competitors in terms of model capacity. To make a fair comparison, there exists KL divergence based nonnegative tensor decomposition methods, such as CP-APR (Chi and Kolda, 2012), an important baseline missing in this paper.   The faster convergence of the natural gradient algorithm than its gradient counterparts is not too surprising.   On page 3 line 83, what is the main reason for using the forward KL divergence KL(P, Q)? How would the zero-avoiding property affect the results in tensor reconstruction? How does it compare to the variational Bayes methods for Boltzmann machines?    The is certainly room for improvements in the clarity and self-containess of this paper. I found the notations in Section 2.1 and 2.2 particularly hard to follow. Also on page 8 line 267-268, the author claims that the “Legendre decomposition effectively uses the index order information”. I’m having trouble understanding what is exactly index order information and how does it improve the reconstruction.       Reference   Lee, Daniel D., and H. Sebastian Seung. "Learning the parts of objects by non-negative matrix factorization." Nature401.6755 (1999): 788.  Chi, Eric C., and Tamara G. Kolda. "On tensors, sparsity, and nonnegative factorizations." SIAM Journal on Matrix Analysis and Applications 33.4 (2012): 1272-1299.  Gonzalez, Edward F., and Yin Zhang. Accelerating the Lee-Seung algorithm for nonnegative matrix factorization. 2005.  [After Rebuttal] The CP-APR comparison included in the rebuttal serves as a more genuine competitor than the other two. The experimental results provide additional evidence on the margin of improvements (despite small) of the proposed approach. Accordingly, I will raise my rating to “weak accept”. The connection between tensor formulation and graphical models looks quite interesting, and now we know that the partial ordering constraint is at least useful for image data.