This paper proposes a novel per-task routing method for multi-task applications. Multi-task neural networks can learn to transfer knowledge across different tasks by using parameter sharing. However, sharing parameters between unrelated tasks can hurt performance. To address this issue, routing networks can be applied to learn to share each group of parameters with a different subset of tasks to better leverage tasks relatedness. However, this use of routing methods requires to address the challenge of learning the routing jointly with the parameters of a modular multi-task neural network. We propose the Gumbel-Matrix routing, a novel multi-task routing method based on the Gumbel-Softmax, that is designed to learn fine-grained parameter sharing. When applied to the Omniglot benchmark, the proposed method improves the state-of-the-art error rate by 17%.
In many ways this work is well presented. However, I have major concerns regarding the novelty of the proposed method and the theoretical rationale for the key design choices. Although the authors do cite and discuss (Rosenbaum et al., 2019), what is very much not clear to me is how the Gumbel-Matrix Routing proposed in this work differs from past work using the Gumbel Softmax within routing networks. It seems like past work even focused on using only the task for routing, so it is not clear to me how the approach here is really novel in comparison. Even if there is some distinction I am missing, the high level idea is clearly not that new. Additionally, there is not much theoretical discussion about what the Gumbel Softmax adds to routing networks. 
The bias/variance tradeoff of Gumbel Softmax / RELAX / REINFORCE was already highlighted in (Rosenbaum et al., 2019). Can the performance of the model on the settings tested be attributed to this tradeoff? If so, would a RELAX model perform even better? Moreover, there is not much discussion of important implications of using the Gumbel Softmax trick in the context of routing. First, as the authors acknowledge, but don't really elaborate on, using the Gumbel Softmax means we must backprop through every possible routing choice in each layer. As a result, the Gumbel approach results in a large scaling of computation with the number of modules, limiting the applicability to more ambitious settings. Moreover, while a clear motivation of this work is eliminating interference between tasks, it is not really explained how Gumbel Softmax does this and how it compares to hard routing decisions in this respect. During backprop, the computation it very similar to mixtures of experts models, and should contain more interference than hard routing. Can you explicitly show that the shape of the Gumbel distribution results in less interference between modules during learning than the standard mixtures of experts softmax approach? 
Furthermore, (Rosenbaum et al., 2019) found that a number of RL based models outperform Gumbel Softmax when routing on multi-task settings of CIFAR-100 and the Stanford Corpus of Implicatives. The authors do not provide any explanation for why this approach did not succeed in their settings. This also leads me to doubt how impressive the results presented here are as there is really not any apples to apples comparison with the same architecture and different routing decisions. In Tables 1 and 2 the best baseline is full sharing. This indicates to me that the performance difference with other cited baselines has to do with different architecture choices and not changes in the routing policy itself. The experiments can be much improved by discussing why past approaches to Gumbel based routing have failed and by thoroughly comparing to other methods for just the routing decisions with the same base architecture as done in prior work.  Unfortunately, in its current form, there is not enough context provided for the community to understand the implications of the proposed approach in the submitted draft even though it achieves good performance.