Multi-task learning promises to use less data, parameters, and time than training separate single-task models. But realizing these benefits in practice is challenging. In particular, it is difficult to define a suitable architecture that has enough capacity to support many tasks while not requiring excessive compute for each individual task. There are difficult trade-offs when deciding how to allocate parameters and layers across a large set of tasks. To address this, we propose a method for automatically searching over multi-task architectures that accounts for resource constraints. We define a parameterization of feature sharing strategies for effective coverage and sampling of architectures. We also present a method for quick evaluation of such architectures with feature distillation. Together these contributions allow us to quickly optimize for parameter-efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify architectures that effectively make trade-offs between task resource requirements while maintaining a high level of final performance.
This paper proposes a framework for learning multi-task convolutional neural networks. For each layer of the network, the proposed algorithm assigns a subset of the layer's channels to each of the tasks. This is in contrast to existing methods that assign whole layers to tasks. There are two key ideas here: (1) instead of searching in the space of binary assignments of layers to tasks, search in the continuous space of fractions of channels assigned to each layer, subject to some consistency constraints; this allows for using finite differences for gradient estimation which can be fed into a black-box optimization procedure; (2) the use of distillation to estimate the performance of a given assignment, rather than retraining many models. Experimentally, the proposed framework performs relatively well on the Visual Decathlon benchmark.
Overall, I do like the paper's key ideas. However, I am not convinced by the presentation of the paper and by the lack of multi-task learning baselines in the experiments. I will consider changing my score if the following comments are addressed.
- Testing your main premise: "we instead partition out individual feature channels within a layer. This offers a greater degree of control over both the computation required by each task and the sharing that takes place between tasks." 
Actually, based on the paper, it is not clear that assigning channels rather than layers brings about additional gains. To prove that hypothesis, you should carry out experiments comparing your method to existing multi-task learning methods that you surveyed in the paper.
- "share all" performance: this baseline seems to do quite well in Table 2, beating es in 5/10 w.r.t. the average accuracy. Why does this happen?
- Too many hacks hidden in the appendix and it is not clear what works why. Have these values been found by cross-validation? The reader should be able to understand how your method works *exactly*. Examples of these magical values:
-- Number of samples for the ES
-- Number of parameter directions
-- "Distillation training is done for a brief 3000 iterations with a batch size of 4 and a learning rate of 1 which is dropped by a factor of 10 at iteration 2000."
Clarification questions:
- I don't understand what "strategy" means here: "Fixed vs Learned Strategies: Is a uniform strategy applied across tasks or is a task-specific solution learned?"
- Section 4.2 needs to be rewritten to give a more structured exposition of the distillation process. Please add pseudocode if needed. The Figure is not very informative.
- MIP: Generally, this is a very important component of your method, yet it is lacking in detail and left to the appendix. Some questions:
-- how long does it take to solve? Which commercial solver do you use? Please provide more details.
-- constraint (8) is for which k? Is it for all k? Please fix. Also for (10), it should be for all i and j as well as k.
-- what is the "non-linear constraint" that (8-9) are linearizing? Please provide the original MIP formulation (without slack or linearization), and explain the final formulation accordingly.
-- MIP: are you adding slack variables to allow for slightly infeasible solutions in case no M can be derived from \tilde{P}?
Minor:
- "cut down average feature dramatically": unclear, rephrase as needed.
- Figure 6 is low-resolution.