An unintended consequence of feature sharing is the model fitting to correlated tasks within the dataset, termed negative transfer. In this paper, we revisit the problem of negative transfer in multitask setting and find that its corrosive effects are applicable to a wide range of linear and non-linear models, including neural networks. We first study the effects of negative transfer in a principled way and show that previously proposed counter-measures are insufficient, particularly for trainable features. We propose a adversarial training approach to mitigate the effects of negative transfer by viewing the problem in a domain adaptation setting. Finally, empirical results on attribute prediction multi-task on AWA and CUB datasets further validate the need for correcting negative sharing in an end-to-end manner.
- The authors study the problem of negative transfer in representation learning, and propose to use the formulation proposed by Ganin & Lempitsky '15 for domain adaptation to reduce negative transfer. Instead of defining the domain classification as the adversarial task to learn a domain-independent representation, they collect a set of classification problems irrelevant to the main task as the adversarial tasks, and aim to learn a representation that focuses only on the primary task. There are very little changes compared to the proposal by Ganin & Lempitsky '15, but the application to solve the problem of negative transfer is interesting.  
- My main concern on the whole argument of the paper is whether the benefits we see in the experiments come from the elimination of negative transfer, or just come from having more training labels from different tasks available. In the main formulation of the approach (equation 7), the authors try to learn a feature representation that works well for the primary task but works poorly for the auxiliary(irrelevant) tasks. If we switch the sign for lambda, then it becomes very similar to traditional multi-task learning. I wonder how the multi-task formulation would compare against the adversarial formulation proposed by the authors. There are reasons to suspect the multi-task formulation will also work better than the logistic regression baseline, since more labels from different tasks are available to learn a better joint representation. It is not clear whether the improvements come from modeling the auxiliary tasks using negative transfer (where the adversarial approach should beat the baseline and multi-task approach), or just come from having more information (where both the adversarial approach and the multi-task approach beat the baseline, but have similar performance). 
- From a practical point of view, it is not easy to decide what prediction tasks are irrelevant. For example, in the birds dataset, I would expect the color and patterns in the body parts to have some correlations (primary_color, upperparts_color, underparts_color, wing_color, etc). In the case of occlusion of the relevant body parts, I could make a guess on the color based on the colors on other parts of the bird. In the ideal case for the current method, I would expect the adversarial approach proposed to learn a representation that mask out all the irrelevant parts of the animal or irrelevant contextual information. Apart from showing improved prediction performance, have the authors perform analysis on the image activation patterns similar to the motivation example in Figure 1 to see if the new approach actually focus on the relevant body parts of the animals? 
- The definition of auxiliary tasks are described in the second last paragraph of 3.3, but it would be clearer if it is also mentioned how they are defined in the experiments section. I went through the whole experiments section having trouble interpreting the results because I could not find the definition of adversarial tasks.  
- Overall I like this paper since it attempts to solve an interesting problem in computer vision, but I would like to see the above question on comparison with multi-task learning answered, or some image activation pattern analysis to provide a more solid argument that the improvements come from elimination of negative transfer.