While deep learning and deep reinforcement learning systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge, particularly as these algorithms learn individual tasks from scratch. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single task learning are not fully understood. Motivated by the insight that gradient interference causes optimization challenges, we develop a simple and general approach for avoiding interference between gradients from different tasks, by altering the gradients through a technique we refer to as "gradient surgery". We propose a form of gradient surgery that projects the gradient of a task onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task reinforcement learning problems, we find that this approach leads to substantial gains in efficiency and performance. Further, it can be effectively combined with previously-proposed multi-task architectures for enhanced performance in a model-agnostic way.
The paper presents a method to boost multi-task learning performance by editing gradient to remove conflicts between tasks. The main idea is to use cosine similarity to 1) determine if two task gradients conflict and 2) to project one conflicting gradient to the normal plane of the other, thereby removing the conflict at the expense of disturbing the other gradient to some extent. Experiments are presented for classification and other computer vision tasks along with reinforcement learning problems.
Overall, I really liked this paper. The explications are clear, the visualizations provided help the understanding (especially Fig.1), and results are compelling. I definitely value the way the method is straightforwardly presented: the underlying idea is simple yet strong. There are, however, a few elements precluding me to pick a higher rating, which I describe in details here.
First, there are a lot of similarities with the MTL method of Sener and Koltun 2018. In particular, I do not agree with the statement that "[this] work, in contrast to many of these optimization schemes [incl. Sener and Koltun], suggests that the challenge in multi-task learning can be attributed to the problem of gradient trashing, which we address directly by introducing a practical algorithm that de-conflicts gradients from different tasks." MTL has the concept of "common descent direction" and, as Fig.1 of relevant paper suggests, it does "de-conflict" the gradients. Sure, the wording is not the same, but the idea is there nonetheless.
To be clear, this is not to say that PCGrad has no merits. I do find it simpler and more elegant (although the latter is a subjective assessment). But I think the similarities should have been discussed in greater details, and the performance compared with Sener and Koltun on at least one problem.
Second, the experiments make it difficult to see the performances of PCGrad alone. Indeed, it is always combined with another multi-task approach/algorithm (MTAN, WPL, SAC+PA, etc.). Providing these results is not incorrect in itself, but it makes it difficult grasp what PCGrad can do alone. Is it worth using only in conjunction with other approaches, or could someone consider using it in a standalone manner? Is PCGrad more a "gradient fine-tuner" than a comprehensive solution for multi-task leearning? The experiments presented, although significant, do not answer these questions.
Third, I am unsure about the value of Theorem 1 and its proof in Sec. 3.2. It assumes too much (e.g., 2 convex tasks) to be of any use in practice. Also, leading to a minimizer of L does not forcibly mean leading to a good solution, depending on the definition of L1 and L2.
There is also one element I'm unsure about. Looking at Algorithm 1, it looks like the order of the tasks may have an effect. Indeed, since it is g_i that is modified at line 8 and not g_j, the last element of B will always remain unaltered (because all other task gradients will have previously been modified to avoid gradient conflict with it). The second to last element of B will be altered, but only due to potential conflicts with the last, and so on, up to the first which can potentially be altered by all others. As an effect, some gradients will always be significantly more altered than others, which can have an impact (at least theoretical) on the learning process. Algorithm 1 thus looks like a greedy approach. Nothing bad in itself (and some ad hoc adjustments, like shuffling B at each update, could very well fix this), but I think this deserves more discussion.
Finally, a more generic comment: the naming of the method should remain the same through the paper. The abstract/intro/Fig.1 refer to the technique as "gradient surgery", while the explanations and experiments talk about PCGrad. Also, in the introduction "plateuas" -> "plateaus".
In summary, I think this is a good paper, presenting a straightforward and useful idea for multi-task learning. However, the related work is not always well described, the experiments lack important comparisons, and the practical effects of PCGrad should be explained in more details instead of focusing on a proof for a convex case.