Multitask learning meets tensor factorization: task imputation via convex optimization
We study a multitask learning problem in which each task is parametrized by a weight vector and indexed by a pair of indices, which can be e.g, (consumer, time). The weight vectors can be collected into a tensor and the (multilinear-)rank of the tensor controls the amount of sharing of information among tasks. Two types of convex relaxations have recently been proposed for the tensor multilinear rank. However, we argue that both of them are not optimal in the context of multitask learning in which the dimensions or multilinear rank are typically heterogeneous. We propose a new norm, which we call the scaled latent trace norm and analyze the excess risk of all the three norms. The results apply to various settings including matrix and tensor completion, multitask learning, and multilinear multitask learning. Both the theory and experiments support the advantage of the new norm when the tensor is not equal-sized and we do not a priori know which mode is low rank.
After reading the authors feedback, I think I have misunderstood the "mode" in the experiment, which is exactly what I was requested. I have modified my score correspondingly. ===In this paper the authors studied an approach for multi-task learning, which represent the (linear) models of the tasks into a tensor, and enforce a low-rank structure in the tensor to model the task relatedness, in which the knowledge is transferred among the tasks. The existing approaches such as latent trace norm failed to address the problem where the task dimensions are heterogeneous, and this paper proposed an improve version of latent trace norm, which normalizes along each dimension. The authors derived error bounds for the two existing tensor norms and the proposed one, relating them to the expected dual norm and showing the expected dual norm for the three norms. It is really nice to have a unified comparison over different norms as shown in the paper. Finally, the authors showed experimental results on synthetic data as well as two real-world data. Overall, this is a good paper with interesting theoretical analysis, and below are my comments of the paper. My main concern of this paper is the usefulness of the tensor multi-task learning. Indeed we can have some “features X aspects X customers” model to explore the relatedness, and the question is that: is it necessary to go beyond two-dimensional case. For any tensor model, we can collapse the model into a flat “features X task” matrix and enforce a low rank to enforce the task relatedness. As one may argue this cannot capture some part of the information as did in tensor formulations, it rare to see how tensor can indeed increase the performance. Even if it helps a little bit, will people pay a large amount of additional time cost for such increase in performance? The lacking of the comparison to simple matrix trace norm in the experiments somehow exaggerated my concern. The core novelty of this paper is the scaled version of a traditional latent trace norm, by adding a scaling factor to the unfolding along each mode. This suggests that we should treat each task (and the unfolding) differently due to some reasons. The idea is somehow related to another paper trying to address a similar problem, the “Multivariate Regression with Calibration” by Liu et. al. The authors may want to see if this paper can be related to that one.  It is interesting if the authors can show some connection between the two. What is the algorithm used in this paper to solve the formulation with the new norm? Is it the same as the one used in the original latent tensor norm? What is the complexity of algorithm? In the case where scaled latent trace norm performs better than the original version, how much time does it take to converge? As mentioned before, for multi-task learning, everything use tensor can also be used by flat trace norm. The authors should at least compare to these types of “flat” multi-task learning methods, e.g., the trace norm, in terms of performance and efficiency.  Also, it is weird to see that the authors use different evaluation (explained variance) metric for the school data. There are many multi-task learning papers that report MSE/NMSE for school data. The authors can use MSE on school data as well to make the paper consistent.Some minor comments:1. Notation “W_{(k)}^{(k)}” in (2) and later on is very confusing. It has not been defined before its use. 2. Typo “Turing”=>“Turning” in page 5, 10 lines from the bottom.