Learning multiple visual domains with residual adapters
There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to perform well uniformly.
Paper address the problem of learning a joint recognition model across variety of domains and tasks. It develops a tunable deep network architecture that, by means of adapter residual modules, can be steered for a variety of visual domains. Methods archives good performance across variety of domains (10 in total) while having high degree of parameter sharing. Paper also introduces a corresponding challenge benchmark - Decathlon Challenge. The benchmark evaluates the ability of a representation to be applied across 10 very different visual domains. 
Generally the paper is very well written and presents an interesting approach and insights. I particularly applaud drawing references to related works and putting the approach in true context. Additional comments are below. 
> References
Generally the references are good and adequate. The following references are thematically related and should probably be added:
Learning Deep Feature Representations with Domain Guided Dropout for Person Re-identification,
Tong Xiao, Hongsheng Li, Wanli, Ouyang, Xiaogang Wang,
CVPR, 2016.
Expanding Object Detector's Horizon: Incremental Learning Framework for Object Detection in Videos, 
Alina Kuznetsova, Sung Ju Hwang, Bodo Rosenhahn, and Leonid Sigal, 
CVPR, 2015.
> Clarifications
Lines 201-202 talk about shrinking regularization. I think this is very interesting and could be a major benefit of the model. However, it is unclear if this is actually tested in practice in experiments. Can you please clarify.
Also, authors claim that the proposed model ensures that performance does not degrade in the original domain. This does not actually appears to be the case (or supported experimentally). For example, performance on ImageNet in Table 1 goes from 59.9 to 59.2. This should be discussed and explained.
> Experiments
In light of the comments above, Table 2 should include the proposed model (or models), e.g., Res. adapt. decay and Res. adapt. finetune all.
Finally, explanation of Scratch+ is unclear and should be clarified in Lines 303-305. I read the sentence multiple times and still unsure what was actually done.