On Multiplicative Multitask Feature Learning
We investigate a general framework of multiplicative multitask feature learning which decomposes each task's model parameters into a multiplication of two components. One of the components is used across all tasks and the other component is task-specific. Several previous methods have been proposed as special cases of our framework. We study the theoretical properties of this framework when different regularization conditions are applied to the two decomposed components. We prove that this framework is mathematically equivalent to the widely used multitask feature learning methods that are based on a joint regularization of all model parameters, but with a more general form of regularizers. Further, an analytical formula is derived for the across-task component as related to the taskspecific component for all these regularizers, leading to a better understanding of the shrinkage effect. Study of this framework motivates new multitask learning algorithms. We propose two new learning formulations by varying the parameters in the proposed framework. Empirical studies have revealed the relative advantages of the two new formulations by comparing with the state of the art, which provides instructive insights into the feature learning problem with multiple tasks.
UPDATE:After reading the feedback, I now understand that Lemma 1 should be correct. Since the proof also confuses the other reviewer, I suggest the authors to improve the presentation of Lemma 1 to avoid such confusion. Also, may the paper accepted, please emphasize the impact of the equivalence.  =========== This paper studies a particular type of multi-task learning, in which the models are decomposed into the product of two components. The authors have established these class of models with group Lasso models. This idea is really interesting and gives many insights into the two sets of formulations. The authors extend the models based on these insights with new parameters and designed an alternating optimization algorithm to solve the problem. The theoretical contribution of the paper is interesting. However, I have some concerns on this paper. First of all, the Lemma does not seem to be right. In Lemma1, to satisfy the equality of Cauchy-Schwarz, we need a specific relationship between sigma_j and alpha. However, in (3), we note that both alpha and sigma are variables to be optimized, and in the optimal solution of (3), it is not necessary the case that this relationship holds, and thus the equality of Cauchy-Schwarz may not hold. As such, the causality here does not seem correct to me.If authors can show that Lemma 1 is correct, then the equivalence of the two types of formulations is novel. Besides the novelty, the authors should further elaborate on why the equivalence is important. The value of such equivalence should be the insights given by it. In this paper, since this equivalence does not have much implication on the efficiency (one is an efficient alternative to the other in some scenario), the new formulations derived based on the equivalence should be the most important part to convince reader its importance. Unfortunately, the paper does not say much about the new formulations, neither does it provide a comprehensive empirical study on them.  Why p and k are chosen to be only 1 or 2? Is there any justification on choices? If there is no theoretical justification, it is also interesting to provide empirical observations on different p, k combinations other than fixing them to be 1 and 2 (e.g., fraction numbers). In the Table 2, we see that in the real datasets, the proposed two formulations MMTFL typically cannot improve much from the existing approaches. This give me concern of how really useful are these newly proposed methods. I would encourage the authors to explore on more real multi-task learning datasets and try to explore what are those datasets that the proposed method works well and why this is the case.