Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data. Applications of neural networks often consider learning in the context of a single task. However, in many scenarios what we hope to learn is not just a single task, but a model that can be used to solve multiple different tasks. Such multi-task learning settings have the potential to improve data efficiency and generalization by sharing data and representations across tasks. However, in some challenging multi-task learning settings, particularly in reinforcement learning, it is difficult to learn a single model that can solve all the tasks while realizing data efficiency and performance benefits. Learning each of the tasks independently from scratch can actually perform better in such settings, but it does not benefit from the representation sharing that multi-task learning can potentially provide. In this work, we develop an approach that endows a single model with the ability to represent both extremes: joint training and independent training. To this end, we introduce matrix-interleaving (Mint), a modification to standard neural network models that projects the activations for each task into a different learned subspace, represented by a per-task and per-layer matrix. By learning these matrices jointly with the other model parameters, the optimizer itself can decide how much to share representations between tasks. On three challenging multi-task reinforcement learning problems with varying degrees of shared task structure, we find that this model consistently matches or outperforms joint training and independent training, combining the best elements of both.
The paper propose a single-network approach to multi-task learning by adding a task-specific linear transformation layer after each fully connected layer. The authors prove that the addition of such a layer keeps the expressive power of the network for each task. They also discuss how the linear transformation (parameterized by a transformation matrix and a bias vector) can be represented in a discrete manner in usual multi-task supervised learning and in a continuous manner (by two other neural networks) in goal-conditioned reinforcement learning. Experiments demonstrate the superiority of the proposed single-network approach.
The proposed approach is single and elegant. It is recommended to weak-reject the paper because of the following key reasons.
(1) Problem formulation is far from clear, perhaps because of the lack of clarity in writing. In particular, the super short Section 2 did not clearly illustrate what's being trained and what's being tested, and whether we care about the generalization performance of each task, or the generalization performance to new tasks (generated from P(T)). Some notations are confusing there---for instance, i seems to be indicating tasks, but then there is a z_k as task indicator. Even for the main proposed approach in Section 3.1, the notations are loosely used in nature. For instance, it is hard to understand what the authors mean by "train separate neural networks to output the Mint matrices and biases"---there is no information about the "training data" for learning those neural networks.
(2) Theoretical justification is at best shallow, or at least in the context that the authors have put it. While having an universal expressive power is good, it is easily achieved by adding an indicator variable (z_k) per layer (similar to task-specific-all-fc in the experiments). So the guarantee does not seem to be closely related to explaining the proposed approach (though the guarantee is nice to have). The authors contrast the guarantee with what FiLM (a competitor approach) can do, but in the experiments FiLM is not taken as a competitor in multi-task supervised learning, leaving a big gap between theory and practice. In the flow presented by the authors, it is strongly suggested to introduce FiLM in more detail and compare it with the proposed approach more clearly in design, theory and experiments.
(3) It is hard to understand whether the experiments are reasonably designed. In particular, the two settings take different sets of competitors, and there is little information on why those competitors are selected, whether they represent state-of-the-art, etc.. The authors highlight that the proposed approach uses much fewer parameters but other than that it is hard to infer why the proposed approach is better. Is it better because there is more overfitting for the competitor's approaches given more parameters? Is it better because it is easier to tune? The task-specific-all-fc (which is of similar # parameters to the proposed approach) result particularly looks suspicious to me but there is no other information to double-check on why the proposed approach is better. In particular, I believe the authors have *not* answered their first proposed question "does our method enable effective multi-task learning both in settings where there is substantial overlap in tasks and where there is little overlap" properly---their best evidence may have been MT10 and MT50 experiments, but even in those experiments, I am not sure whether the authors want to take the results as suggesting there are "substantial overlap" or "little overlap." 
Some other suggestions:
(4) It is suggested to analyze the matrices learned by the proposed approach. Do the matrices contain reasonable task correlations (i.e. for two similar tasks, are the matrices somewhat similar) to understand more about the proposed approach.
(5) It looks a bit strange to me that there is no discussion on regularizing the linear transformation matrices, as it seems possible to embed the task relations through the regularization. Have the authors considered the possibility?
(6) The authors are overly-emphasizing what they want to do (interpolating between independent networks and shared network). This occupies multiple redundant paragraphs in the early sections. It is better to remove some of those and use the space for more solid results, such as clarifying the notations.
(7) One baseline that could have been considered is to just train a fully-shared network (without z_k), and a fully-independent one. Then, use validation to select the better network and compare with the proposed approach.