K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.
 Summary: the paper introduces a new way of fine-tuning neural networks. Instead of re-training the whole model or fine-tuning the last few layers, the authors propose to fine-tune a small set of model patches that affect the network at different layers. The results show that this way of fine-tuning is superior to above mentioned typical ways either in accuracy or in the number of tuned parameters in three different settings: transfer learning, multi-task learning and domain adaptation. Quality: the introduced way of fine-tuning is interesting alternative to the typical last layer re-training. I like that the authors present an intuition behind their approach and justify it by an illustrative example. The experiments are fair, assuming the authors explain the choice of hyper-parameters during the revision. Clarity: in general the paper is well-written. The discussion of multi-task and domain adaptation parts can be improved though. Originality: the contributions are novel to my best knowledge. Significance: high, I believe the paper may facilitate a further developments in the area. I ask the authors to address the following during the rebuttal stage: * explain the choice of the hyper-parameters of RMSProp (paragraph under Table 1). * fix Figure 3, it*s impossible to read in the paper-printed version * explain how the average number of parameters per model in computed in Tables 4 and 5. E.g. 700K params/model in the first column of Table 4 is misleading - I suppose the shared parameters are not taken into account. The same holds for 0 in the second column, etc. * add a proper discussion for domain adaptation part. The simple "The results are shown in Table 5" is not enough. * consider leaving the discussion of cost-efficient model cascades out. The presented details are too condensed and do not add value to the paper. * explain how different resolutions are managed by the same model in the domain adaptation experiments.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary: the paper introduces a new way of fine-tuning neural networks. Instead of re-training the whole model or fine-tuning the last few layers, the authors propose to fine-tune a small set of model patches that affect the network at different layers. The results show that this way of fine-tuning is superior to above mentioned typical ways either in accuracy or in the number of tuned parameters in three different settings: transfer learning, multi-task learning and domain adaptation. Quality: the introduced way of fine-tuning is interesting alternative to the typical last layer re-training. I like that the authors present an intuition behind their approach and justify it by an illustrative example. The experiments are fair, assuming the authors explain the choice of hyper-parameters during the revision. Clarity: in general the paper is well-written. The discussion of multi-task and domain adaptation parts can be improved though. Originality: the contributions are novel to my best knowledge. Significance: high, I believe the paper may facilitate a further developments in the area. I ask the authors to address the following during the rebuttal stage: * explain the choice of the hyper-parameters of RMSProp (paragraph under Table 1). * fix Figure 3, it*s impossible to read in the paper-printed version * explain how the average number of parameters per model in computed in Tables 4 and 5. E.g. 700K params/model in the first column of Table 4 is misleading - I suppose the shared parameters are not taken into account. The same holds for 0 in the second column, etc. * add a proper discussion for domain adaptation part. The simple "The results are shown in Table 5" is not enough. * consider leaving the discussion of cost-efficient model cascades out. The presented details are too condensed and do not add value to the paper. * explain how different resolutions are managed by the same model in the domain adaptation experiments.
assistant
 Summary: the paper introduces a new way of fine-tuning neural networks. Instead of re-training the whole model or fine-tuning the last few layers, the authors propose to fine-tune a small set of model patches that affect the network at different layers. The results show that this way of fine-tuning is superior to above mentioned typical ways either in accuracy or in the number of tuned parameters in three different settings: transfer learning, multi-task learning and domain adaptation. Quality: the introduced way of fine-tuning is interesting alternative to the typical last layer re-training. I like that the authors present an intuition behind their approach and justify it by an illustrative example. The experiments are fair, assuming the authors explain the choice of hyper-parameters during the revision. Clarity: in general the paper is well-written. The discussion of multi-task and domain adaptation parts can be improved though. Originality: the contributions are novel to my best knowledge. Significance: high, I believe the paper may facilitate a further developments in the area. I ask the authors to address the following during the rebuttal stage: * explain the choice of the hyper-parameters of RMSProp (paragraph under Table 1). * fix Figure 3, it*s impossible to read in the paper-printed version * explain how the average number of parameters per model in computed in Tables 4 and 5. E.g. 700K params/model in the first column of Table 4 is misleading - I suppose the shared parameters are not taken into account. The same holds for 0 in the second column, etc. * add a proper discussion for domain adaptation part. The simple "The results are shown in Table 5" is not enough. * consider leaving the discussion of cost-efficient model cascades out. The presented details are too condensed and do not add value to the paper. * explain how different resolutions are managed by the same model in the domain adaptation experiments.