Sparse Overlapping Sets Lasso for Multitask Learning and its Application to fMRI Analysis
Multitask learning can be effective when features useful in one task are also useful for other tasks, and the group lasso is a standard method for selecting a common subset of features. In this paper, we are interested in a less restrictive form of multitask learning, wherein (1) the available features can be organized into subsets according to a notion of similarity and (2) features useful in one task are similar, but not necessarily identical, to the features best suited for other tasks. The main contribution of this paper is a new procedure called Sparse Overlapping Sets (SOS) lasso, a convex optimization that automatically selects similar features for related learning tasks. Error bounds are derived for SOSlasso and its consistency is established for squared error loss. In particular, SOSlasso is motivated by multisubject fMRI studies in which functional activity is classified using brain voxels as features. Experiments with real and synthetic data demonstrate the advantages of SOSlasso compared to the lasso and group lasso.
Comments: This paper proposes a new method for 
      multi-task learning where the features in different tasks are similar but 
      not identical. The main contribution is the proposed regularizer for 
      Sparse Overlapping Sets lasso, where the group configuration can be 
      automatically learned. Experiments with real and synthetic data 
      demonstrate the advantages of SOSlasso compared to the lasso and group 
      lasso. Overall, this paper is interesting. However, compared to 
      earlier works, the novelty and improvements are not very clear enough. 
      Several key problems need further clarified in this submission. In 
      addition, connection between proposed framework and more practical 
      applications is unclear. The paper presentation should be further 
      improved. Some additional comments: 1. The equation (3) is 
      interesting point in this paper. However, it is not an easy optimization 
      problem. It seems that w_G is already non-overlapping. Why do the variable 
      need to be duplicated as in [5]? Another issue is how to fix the group 
      size of each group G, for the cases with overlapping and non-overlapping 
      respectively? The readers need know more details on how to exactly solve 
      this problem efficiently. Without it, it is hard to evaluate its 
      advantage. 2. The reviewer has the feeling that Section 3 contains 
      a significant amount of existing results, while the new things offered in 
      this section seem to be simply extended from those existing works. It is 
      suggested to present this section in a more efficient way to be 
      differentiated clearly from prior works. 3. The experiments are 
      not convincing. a) The synthetic data in Section 5.1 seems to be 
      typical overlapping group sparse data, why does the Glasso method perform 
      worse? b) In Fig.2(b), when alpha = 0, all coefficients are zeros. Why 
      the MSE of Glasso is not zero as the other two algorithms? c) In 
      Section 5.2, the proposed SOSlasso is compared with standard solvers lasso 
      and Glasso. However, there is no comparison against the state-of-the-art 
      methods of fMRI analysis and overlapped group lasso methods. Without such 
      experiments, it is hard to validate the effectiveness of the proposed 
      method on this application. 4. The connection between the method 
      and fMRI application is missing. It is very difficult for readers know how 
      to apply the proposed method on fMRI analysis, especially for those 
      without any fMRI background. It is suggested to describe the problem in 
      fMRI analysis briefly. The author should clarify which data is sparse or 
      group sparse, and what is the data matrix $Phi$. 5. In table 1, 
      the norm values of some examples are provided. Why not present the 
      mechanism of the proposed regularizer, which can automatically select the 
      best groups? 6. In p2 line 84, there should be a period at the 
      end. 7. In p3 line 110-111, “Uppercase and lowercase” changes to 
      “Lowercase and uppercase”. 8. In p3 line 119, “non zero” changes 
      to “non-zeros”. 9. In p6 line 317, “$G \in G$” is a typo. 
      10. Please define MSE, AWGN etc.