We investigate multi-task learning approaches which use a shared feature representation for all tasks. To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task. We study the theory of this setting on linear and ReLU-activated models. Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtain a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT LARGE using our alignment method. We also design an SVD-based task reweighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.
This paper analyzed the principles for a successful transfer in the hard-parameter sharing multitask learning model. They analyzed three key factors of multi-task learning on linear model and relu linear model: model capacity (output dimension after common transformation), task covariance (similarity between tasks) and optimization strategy (influence of re-weighting algorithm), with theoretical guarantees. Finally they evaluated their assumptions on the state-of-the-art multi-task framework (e.g GLUE,CheXNet), showing the benefits of the proposed algorithm.
Main comments:
This paper is highly interesting and strong. The author systematically analyzed the factors to ensure a good multi-task learning. The discovering is coherent with with previous works, and it also brings new theoretical insights (e.g. sufficient conditions to induce a positive transfer in Theorem 2). The proof is non-trivial and seems technically sound.
Moreover, they validated their theoretical assumptions on the large scale and diverse datasets (e.g NLP tasks, medical tasks) with state-of-the-art baselines, which verified the correctness of the theory and indicated strong practical implications.
Minor comments:
The main message of the paper is clear but some parts still confuse me:
1. I suggest the author to merge the Figure 3 and Data generation (Page 4) part for a better presentation. e,g which “diff.covariance” is task 3 or 4 ?  And why we use different rotation matrix Q_i ? 
2. In algorithm 1 (Page 5) , I suggest the author use a formal equation (like algorithm 2) instead of descriptive words.
     -- Step 2, I have trouble in understading this step.
     -- Step 3, how to jointly minimize R_1,\dots, R_k, A_1, \dots, A_k ? we use loss (3)  or other losses ?
     -- I suggest that the author release the code for a better understanding.
3. For theorem 2, can we find some “optimal” c to optimize the right part ? Since 6c + \frac{1}{1-3c}\frac{\epsilon}{\X_2\theta_2} might be further optimized 
4. In section 3.3. (Figure 6)  of the real neural network, the model capacity is the dimension of Z or simply the dimension before last fc-layer ?
5. Some parts in the appendix can be better illustrated:
    (a) I am not clear how proposition 4 can derive proposition 1.
    (b) Page 15, proving fact 8: last line \frac{1}{k^4}sin(a^{prime},b^{prime}) should be \frac{1}{k^4}sin^{2}(a^{prime},b^{prime}). 
Overall I think it is a good work with interesting discoverings for the multi-task learning. I think it will potentially inspire the community to have more thoughts about the transfer learning.