Low-Rank Regression with Tensor Responses
This paper proposes an efficient algorithm (HOLRR) to handle regression tasks where the outputs have a tensor structure. We formulate the regression problem as the minimization of a least square criterion under a multilinear rank constraint, a difficult non convex problem. HOLRR computes efficiently an approximate solution of this problem, with solid theoretical guarantees. A kernel extension is also presented. Experiments on synthetic and real data show that HOLRR computes accurate solutions while being computationally very competitive.
This paper proposes a tensor-variate multilinear regression model called HOLRR for the prediction of a tensor output from vector input. They do this by formulating the problem as the minimization of a least squares subject to a multilinear rank constraint on the regression tensor. A kernelized extension of HOLRR for nonlinear setting is also introduced.
Strength: --The paper provides the theoretical analysis of approximation guarantees and a generalization bound for the class of tensor-valued regression functions.  --The paper is clearly written for readers to understand and appreciate the proposed algorithm.   Weakness: --A major drawback is that the novelty and contribution is rather limited.   The key idea and the model of this paper is actually equivalent to the HOPLS in the following paper: [Zhao et. al 2013a] "Higher order partial least squares (HOPLS): a generalized multilinear regression method. PAMI".  In HOPLS, it assumes the tensor input has low-rank structure and also the tensor output has low-rank structure, and the link of them is established in the common latent space. Specifically, HOPLS projects both input tensor and output tensor simultaneously onto a common latent space by using Tucker models. And then follows a regression step against the projected latent variables.   In HOLRR, this equivalence can be seen by taking look at the final regression function f^star in the paragraph "low-rank regression function" of Section 3.  For the input vector x: the term x^T*U_0 in f^star is in fact the latent variables, where the factor matrix U_0 projects the input vector x onto a common latent space vector.  For the output tensor Y: the authors assume tensor Y has a low-rank Tucker model structure, where G is in fact the core tensor of Y and U_1 to U_p are the loadings.  The regression from x to Y is linked by the latent vector as follows: f^star = (x^T*U_0)*G_(1) (U_p kron ... kron U_1), which is essentially the same as HOPLS, the correspondence can be seen clearly in Section 3.4 in the paper [Zhao et. al 2013a].  Thus, the model HOLRR is just a special case of model HOPLS where the input takes form of matrix (stacked vector samples) and the output assume to be a low-rank tensor.  The kernelized version is very similar to the Kernel-based tensor PLS regression which can be found in the paper: [Zhao et. al 2013b] "Kernelization of tensor-based models for multiway data analysis: Processing of multidimensional structured data".    --This paper only considers the correlations in the output tensor, and hence ignores (destroys) the input-side tensor structure by the vectorization of input data into a vector form, which may suffer the similar limitations as those ignoring the output-side structure.   In practice, most real-world tensor-variate regression applications involve tensor covariates input, which may limit the applicability of this model.   --For the algorithms, the authors simply apply the truncated HOSVD to find the projection matrices U_p for each mode, and HOSVD is usually used as a initialization point for other algorithms. I doubt that HOSVD is not as accurate as ALS-based HOOI that is used in algorithms such as HOPLS.   --For real-data experiments, the HOLRR is mainly tested on the spatio-temporal task against ADMM and GREEDY. The spatio-temporal data is a special case of general tensor data. I think some tasks involving general tonsorial input and output data should be tested.   In order to show the superiority of the proposed HOLRR, I think HOPLS [Zhao et. al 2013a] and the corresponding Kernel-based tensor PLS regression [Zhao et. al 2013b] should also be included in comparison on the general data.