Distributed Flexible Nonlinear Tensor Factorization
Tensor factorization is a powerful tool to analyse multi-way data. Recently proposed nonlinear factorization methods, although capable of capturing complex relationships, are computationally quite expensive and may suffer a severe learning bias in case of extreme data sparsity. Therefore, we propose a distributed, flexible nonlinear tensor factorization model, which avoids the expensive computations and structural restrictions of the Kronecker-product in the existing TGP formulations, allowing an arbitrary subset of tensorial entries to be selected for training. Meanwhile, we derive a tractable and tight variational evidence lower bound (ELBO) that enables highly decoupled, parallel computations and high-quality inference. Based on the new bound, we develop a distributed, key-value-free inference algorithm in the MAPREDUCE framework, which can fully exploit the memory cache mechanism in fast MAPREDUCE systems such as SPARK. Experiments demonstrate the advantages of our method over several state-of-the-art approaches, in terms of both predictive performance and computational efficiency.
This paper proposes a tensor factorization algorithm that can be viewed as a tensor completion problem, in which a ``low-rank'' decomposition of the tensor is constructed using partial observations of the tensor.   It is based on the Tucker decomposition with appropriate priors for the Tucker factors $U$.  It considers both binary and continuous tensors. For continuous tensors the formulation is based on the model of equation 2, where the kernelized prior is based defined  in lines 113--120. Furthermore because the covariance matrix $K$ that appears in EQ 2 is very large, an approximation by using the scheme in 18.   A fixed-step gradient descent scheme is used (combined with L-BFGS). Additional optimizations are discussed for use on a distributed architecture using the MapReduce and the SPARK system.   The paper concludes with numerical experiments that demonstrate the performance of the algorithm in terms of accuracy of the approximation and AUC values for several datasets.  
REVIEW  The results in Figure 2 look quite good, but why? Is it because kernelization or is it because of the simpler covariance structure? Overall no explanation on why the method performs better is given.    Timing comparisons with the other factorizations are not presented, so is the method faster? If not, what happens if we increase the ranks for the other methods. If there are so many fudging factors, where the other methods similarly optimized?  Not sure that the bounds are useful. How are they used in the algorithm?  There are numerous parameters: prior for W core tensor, kernel form and kernel parameters, $\beta$ for observations, the number of inducing points, regularization for $U$ (first term in Eq 2), the size $p$ for the B matrix. This seems an almost impossible method to use, how do you choose all these parameters? This formulation introduces more non-convexity with having to solve for the inducing points B.   Also, how is the algorithm terminated? In the experimental session the splitting between testing and non-testing is arbitrary.    SECONDARY COMMENTS   Are the inducing points solved for or are there a subset of the original ones?   How exactly is EQ 2 combined with the equation after line 148?  Question on Line 38: ``First, it can capture highly nonlinear interactions in the tensor, and is flexible enough to incorporate arbitrary subset of (meaningful) tensorial entries for the training.'' How is this different from other tensor factorizations? What does ``Highly nonlinear'' mean? How Gaussian priors allow more nonlinear interactions? These statements seem quite ad hoc to me.  Line 45: ``For example, one can choose a combination of balanced zero and nonzero elements to overcome the learning bias.'' But in the first paragraph of the introduction it is mentioned that zero elements are meaningless. Not sure what this sentence means.  The algorithmic complexity in 4.4 does not involve the number of iterations. Also, how do you invert K_BB, that should have at least p^3 complexity, right? I guess you just ignore this term?  There is way too much material in this paper. If it ends up resubmitted, I recommend that first the focus is on the formulation and detailed numerical experiments, and then on the distributed runs.   The comparison with linear logistic and svm is interesting, but one should compare with kernelized versions of those methods.