Multi-Task Learning as Multi-Objective Optimization
In multi-task learning, multiple tasks are solved jointly, sharing inductive bias between them. Multi-task learning is inherently a multi-objective problem because different tasks may conflict, necessitating a trade-off. A common compromise is to optimize a proxy objective that minimizes a weighted linear combination of pertask losses. However, this workaround is only valid when the tasks do not compete, which is rarely the case. In this paper, we explicitly cast multi-task learning as multi-objective optimization, with the overall objective of finding a Pareto optimal solution. To this end, we use algorithms developed in the gradient-based multiobjective optimization literature. These algorithms are not directly applicable to large-scale learning problems since they scale poorly with the dimensionality of the gradients and the number of tasks. We therefore propose an upper bound for the multi-objective loss and show that it can be optimized efficiently. We further prove that optimizing this upper bound yields a Pareto optimal solution under realistic assumptions. We apply our method to a variety of multi-task deep learning problems including digit classification, scene understanding (joint semantic segmentation, instance segmentation, and depth estimation), and multilabel classification. Our method produces higher-performing models than recent multi-task learning formulations or per-task training.
Overall summary of the paper:  This paper proposed a multi-task learning algorithm from multi-objective optimization perspective and the authors provided an approximation algorithm, which could accelerate the training process. The authors claim that existing MTL algorithms used linear combinations (uniform weight) of the loss from different tasks, which is hard to achieve the Pareto optimality. Unlike the uniform weight strategy, the authors use the MGDA algorithm to solve the optimal weight, which would increase the performance for all the tasks to achieve the Pareto optimality. Moreover, when solving the sub-problem for shared parameters, the author gave an upper bound of the loss function, and this upper bound optimization only requires one-time back propagation regardless of the number of tasks. The results show that the approximation strategy not only can accelerate the training process but also improve the performance.  Strengths: 1. The proposed method can solve the optimal weight for the loss of different tasks. 2. The approximation strategy can efficiently accelerate the training process. 3. The proposed algorithm is easy to implement.  Weaknesses: 1. The motivation is not convincing enough. The decrease of training loss does not mean the generalization performance is also good. Pareto optimality is for the training loss decrease, which has no guarantee that it will also improve the generalization performance. 2. The optimization treatment is not novel.  3. The approximation provides a good and easy way to accelerate the algorithm, which is the spotlight of this paper. However, in the experiment, the approximation one is even better than the original one in prediction, which does not make sense. Although the authors provide some qualitative reasons at the end of the paper, it is hard to convince me why this approximation can even improve the prediction result. I noticed that in Table 2, the training time for without approximation is longer than 10 days. Is the baseline without approximation carefully tuned?  4. The author does not include the uniform scaling + approximation strategy. I am wondering how this baseline perform compared to other methods in the paper. This could help readers to understand which part is really effective.