Consistent Multitask Learning with Nonlinear Output Relations
Key to multitask learning is exploiting the relationships between different tasks in order to improve prediction performance. Most previous methods have focused on the case where tasks relations can be modeled as linear operators and regularization approaches can be used successfully. However, in practice assuming the tasks to be linearly related is often restrictive, and allowing for nonlinear structures is a challenge. In this paper, we tackle this issue by casting the problem within the framework of structured prediction. Our main contribution is a novel algorithm for learning multiple tasks which are related by a system of nonlinear equations that their joint outputs need to satisfy. We show that our algorithm can be efficiently implemented and study its generalization properties, proving universal consistency and learning rates. Our theoretical analysis highlights the benefits of non-linear multitask learning over learning the tasks independently. Encouraging experimental results show the benefits of the proposed method in practice.
The paper tackles multi-task learning problems where there are
non-linear relationships between tasks. The relationships between
tasks is encoded as a set of non-linear constraints that the outputs
of each task must satisfy (e.g . y1^2 + y2^2 = 1). In a nutshell, he
proposed technique can be summarized as: use kernel regression to make
predictions for each task independently, then project the prediction
vector onto the constrained set.
Overall, I like the idea of being able to take advantage of non-linear
relationships between tasks. However, I am not sure how practical it
is to specify the non-linear constraints between tasks in practice.
The weakest point of the paper is the empirical evaluation.
It would be good if the authors can discuss real applications where
one can specify a reasonable set non-linear relationships between
tasks. The examples provided in the paper are not very
satisfactory. One imposes deterministic relationships between tasks
(i.e. if the output for task 1 is known then the output for task 2 can
be inferred exactly), and the other approximates the relationship
through a point cloud (which raises the question why was the data in
the point cloud not used as training data. Also to obtain a point
cloud one needs to be in a vector-valued regression setting).
The paper could be significantly improved by providing a more thorough
empirical evaluation section that analyses the performance of the
proposed method on multiple real problems.  One of the main
contributions of the paper, as stated by the authors, is the extension
of the SELF approach beyond vector-valued regression. But evaluation
is done only on vector-valued regression problems.
The technique depends on using a kernel K for the kernel
regression. Is there any guidance on how to select such a kernel? What
kernel was used in the experimental section?
It would also be good if the authors analyze the effect of various
design and parameter choices. How sensitive are the results to the
kernel K, to the regularization parameter \lambda, to the parameters
\delta and \mu?
How were all the above parameters set in the experimental section?
Looking at figure 1, I am very surprised that the performance of STL
seems to converge to a point significantly worse than that of
NL-MTL. Given a large amount of data I would expect STL to perform
just as well or better than any MTL technique (information transfer
between tasks is no longer relevant if there is enough training
data). This does not seem to be the case in figure 1. Why is that?