Deep neural networks are a promising approach towards multi-task learning because of their capability to leverage knowledge across domains and learn general purpose representations. Nevertheless, they can fail to live up to these promises as tasks often compete for a model's limited resources, potentially leading to lower overall performance. In this work we tackle the issue of interfering tasks through a comprehensive analysis of their training, derived from looking at the interaction between gradients within their shared parameters. Our empirical results show that well-performing models have low variance in the angles between task gradients and that popular regularization methods implicitly reduce this measure. Based on this observation, we propose a novel gradient regularization term that minimizes task interference by enforcing near orthogonal gradients. Updating the shared parameters using this property encourages task specific decoders to optimize different parts of the feature extractor, thus reducing competition. We evaluate our method with classification and regression tasks on the multiDigitMNIST, NYUv2 and SUN RGB-D datasets where we obtain competitive results.
The submission argues that when training the multiple objectives in a multi task learning framework, orthogonalizing gradients is beneficial toward reducing task competitions and efficient allocation of the learning capacity in the parameters.  This is demonstrated experimentally and subsequently a second order method is used for incorporating this in training. 
Positives: 
1. The concept is sensible, intuitive, and simple. There has a been quite few works (Sener and Koltun 2018) that similarly argue analyzing/regularizing the gradient direction when training multiple task objectives is beneficial. This submission reinforces those. 
2. The implementation of the concept using a second order method is sensible and simple.  
Weaknesses: 
1. Missing comparisons: As reiterated above, several existing works have augmented training of neural networks with terms that are based on directions of gradients of multiple objectives. Some of them, e.g. Sener and Koltun 2018 or Du et al 2018, are very related. Though the submission cites them, no experimental comparison or convincing verbal critique of the differences are provided. The experiments could have included baselines that correspond to the concepts proposed in those prior papers to support the novelty of this submission. If the authors believe the concept of those papers are basically the same as theirs, then this submission should change to an analysis paper rather than appearing to pitch a new regularization term. I would have found an analysis paper as valuable as one with a novel method, but the stance should be clear. 
2. I found the experiments too toy to be convincing. The MultiDigitMNIST is artificial and with limited benchmarking value as its construct doesn't really reflect the construct of multi task learning in the real world (e.g. as in the multi task vision datasets). NYUv2 dataset is more realistic, but the reported results are not clear to show significant and meaningful differences (see table 2). Particularly since the NYUv2 dataset has certain biases with imperfect ground truth from the kinect sensor which questions if differences in 0.001 range are meaningful. The authors can consider more recent and comparatively more reliable multi label datasets like Taskonomy for benchmarking. 
3. Inline with the above comment, qualitative results or any other method for convincing that the achieved results (on datasets other than MultiDigitMNIST) is meaningful seems crucial. 
Further comments: 
4. The single task baselines in table 2 often perform inferior to the multi task baselines. This suggests the NYUv2 dataset could be too small to learn individual networks, whereas many other multi task papers often find single task baselines are hard to beat if they're not starved of parameters (e.g. see Standley 2018 "Which Tasks Should Be Learned Together in Multi-task Learning?"). Please clarify and/or consider using large enough datasets that allow you to benchmark under both high data and low data regimes. 
5. why there is no single task baseline in table 1? 
6. why CosReg is not included in figure 3 plot? 
7. A closer look at the recent works on analyzing task competitions in multi task learning could be useful, and probably supportive of the concept of this submission. For instance standley 2018 Which Tasks Should Be Learned Together in Multi-task Learning seem to suggest that tasks that are related under transfer learning setting did not help each other in multi task setting. Their observation seems related to the pitch of this paper that orthogonal gradients (ie tasks with dissimilar updates) can be optimized better. 
8. In page 4 authors state that computing the regularization term for all layers is complex, hence they do that only for the last layer. This seems okay to me, though I would have found an experiment demonstrating the consequences of this simplification useful (e.g. am experiment showing which layer to pick and a one-time expensive experiment demonstrating that picking one layer vs more layers is not too damaging). 
-------
Comments after rebuttal: 
I read the rebuttal and appreciate authors responses and the attempted improvement. It helped. I think the submission has the potential to be ultimately a useful read for the community, but at this stage more work/revision that wouldn't fit rebuttal time constraints would be needed to achieve that. I still think an experimental comparison with the related methods would be more convincing than verbal. 
The added experiment on SUN RGB-D dataset is appreciated, but all results appear too close (table 2) to suggest one should adopt the proposed method. Also SUN RGB-D limits the tasks that could be learned together in a multi-task experiment given the limited number of labels per datapoint. I would have found using more recent  datasets with higher number of labels (rather than forcing to use depth+semantics which doesn't have to be a good mix, see Standley 2018) and more significant quantitate improvements more convincing. 
Overall I lean toward not accepting the current submission but acknowledge the potential value in resubmission after further work.