Boosted Sparse and Low-Rank Tensor Regression
We propose a sparse and low-rank tensor regression model to relate a univariate outcome to a feature tensor, in which each unit-rank tensor from the CP decomposition of the coefficient tensor is assumed to be sparse. This structure is both parsimonious and highly interpretable, as it implies that the outcome is related to the features through a few distinct pathways, each of which may only involve subsets of feature dimensions. We take a divide-and-conquer strategy to simplify the task into a set of sparse unit-rank tensor regression problems. To make the computation efficient and scalable, for the unit-rank tensor regression, we propose a stagewise estimation procedure to efficiently trace out its entire solution path. We show that as the step size goes to zero, the stagewise solution paths converge exactly to those of the corresponding regularized regression. The superior performance of our approach is demonstrated on various real-world and synthetic examples.
In this paper, the authors propose an algorithm for tensor regression. The problem of tensor regression with low-rank models have been approached in several recent works. The novelty in the present paper is that, besides the low-rank CP (PARAFAC) property, sparsity in the factor matrices are also imposed, thus providing interpretability to the results. In order to estimate the CP model factors with sparsity constraints, the authors propose a deflation approach which means to estimate one rank-1 term of the decomposition at a time, giving that the other terms are fixed. For the rank-1 term estimation step, the authors use a stagewise technique inspired in the algorithm for Lasso (Zhow & Yu, 2007). Finally, the experimental results are presented using simulated as well as real-world datasets. I found the paper somewhat relevant to NIPS, although its contribution can be seen as an improvement of previously proposed regression techniques. Particularly, I found the following issues in the current version of the paper: - The authors made use of the term “boosted” in the title but it is not well justified in the paper. In particular, the explanation in lines 158 – 160 is not clear. Does it mean that using a combination of a forward plus a backward step provides a boosting effect? The Authors have clarified this in their responses. - Line 98: The sentence “3) It leads to a more flexible and parsimonious model, thus makes the model generalizability better” is kind of vague and should be more precise. What does it mean to have a more flexible model? What does it mean to have a more parsimonious model? What do the authors mean by better generalizability? Are the authors referring to the fact that the model is less prone to overfitting? If so, this must be clearly stated and demonstrated, at least, with experiments. The Authors have clarified this in their responses. - Line 114-115: The sentence “Here for improving the convexity of the problem and its numerical stability …” is not clear enough. These concepts must be discussed more precisely. Is the model not convex without using the elastic-net? What does it mean to be numerically unstable and why the elastic-net model helps to avoid it?The Authors have clarified this in their responses and will provide further clarification in the final version. - Line 116 – 117: The authors say that “we fix alpha>0 as a small constant”. What does it mean a "small value"? What is the range of this parameter and how should we judge if a value is small or large? The Authors have clarified this in their responses and will make more specific recommendations on the choice of alpha in the final version of the paper. - Algorithm 1: This algorithm is called “Fast” by the authors. The authors should provide some complexity analysis to justify the term “fast” in the algorithm name. - I found the experimental results based on real-world signals somewhat confusing. The authors state that the goal in this case is to predict the Montreal Cognitive Assessment scores. The authors evaluate the results in terms of root mean squares error (RMSE), sparsity and computation time. This error (RMSE) is not defined in the paper. Is it computed by calculating the error between the ground truth score and the one predicted by the model? The Authors have clarified this in their responses and will provide a definition in the final version of the paper. - Line 66, the index in the lower end in the last sum must be changed i_1 -> i_N - Line 35 in supplemental information: “Some Experimental Results” -> “Additional Experimental Results”