K FOR THE PRICE OF 1: PARAMETER-EFFICIENT MULTI-TASK AND TRANSFER LEARNING
We introduce a novel method that enables parameter-efficient transfer and multitask learning with deep neural networks. The basic approach is to learn a model patch -a small set of parameters -that will specialize to each task, instead of finetuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match singletask performance.
Summary: the paper introduces a new way of fine-tuning neural networks. Instead of re-training the whole model or fine-tuning the last few layers, the authors propose to fine-tune a small set of model patches that affect the network at different layers. The results show that this way of fine-tuning is superior to above mentioned typical ways either in accuracy or in the number of tuned parameters in three different settings: transfer learning, multi-task learning and domain adaptation.
Quality: the introduced way of fine-tuning is interesting alternative to the typical last layer re-training. I like that the authors present an intuition behind their approach and justify it by an illustrative example. The experiments are fair, assuming the authors explain the choice of hyper-parameters during the revision.
Clarity: in general the paper is well-written. The discussion of multi-task and domain adaptation parts can be improved though.
Originality: the contributions are novel to my best knowledge.
Significance: high, I believe the paper may facilitate a further developments in the area.
I ask the authors to address the following during the rebuttal stage:
* explain the choice of the hyper-parameters of RMSProp (paragraph under Table 1).
* fix Figure 3, it's impossible to read in the paper-printed version
* explain how the average number of parameters per model in computed in Tables 4 and 5. E.g. 700K params/model in the first column of Table 4 is misleading - I suppose the shared parameters are not taken into account. The same holds for 0 in the second column, etc.
* add a proper discussion for domain adaptation part. The simple "The results are shown in Table 5" is not enough. 
* consider leaving the discussion of cost-efficient model cascades out. The presented details are too condensed and do not add value to the paper.
* explain how different resolutions are managed by the same model in the domain adaptation experiments.