Explicit Induction Bias for Transfer Learning with Convolutional Networks
In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We eventually recommend a simple penalty using the pre-trained model as a reference, and we show that this approach behaves much better than the standard scheme using weight decay on a partially frozen network.
 The paper proposes an analysis on different adaptive regularization techniques for deep transfer learning. Specifically it focuses on the use of an L2-SP condition that constraints the new parameters to be close to the ones previously learned when solving a source task. + The paper is easy to read and well organized + The advantage of the proposed regularization against the more standard L2 regularization is clearly visible from the experiments - The idea per se is not new: there is a list of shallow learning methods for transfer learning based on the same L2 regularization choice [Cross-Domain Video Concept Detection using Adaptive SVMs, ACM Multimedia 2007] [Learning categories from few examples with multi model knowledge transfer, PAMI 2014] [From n to n+ 1: Multiclass transfer incremental learning, CVPR 2013] I believe this literature should be discussed in the related work section - It is true that the L2-SP-Fisher regularization was designed for life-long learning cases with a fixed task, however, this solution seems to work quite well in the proposed experimental settings. From my understanding L2-SP-Fisher can be considered the best competitor of L2-SP so I think the paper should dedicate more space to the analysis of their difference and similarities both from the theoretical and experimental point of view. For instance: -- adding the L2-SP-Fisher results in table 2 -- repeating the experiments of figure 2 and figure 3 with L2-SP-Fisher
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Explicit Induction Bias for Transfer Learning with Convolutional Networks
In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We eventually recommend a simple penalty using the pre-trained model as a reference, and we show that this approach behaves much better than the standard scheme using weight decay on a partially frozen network.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper proposes an analysis on different adaptive regularization techniques for deep transfer learning. Specifically it focuses on the use of an L2-SP condition that constraints the new parameters to be close to the ones previously learned when solving a source task. + The paper is easy to read and well organized + The advantage of the proposed regularization against the more standard L2 regularization is clearly visible from the experiments - The idea per se is not new: there is a list of shallow learning methods for transfer learning based on the same L2 regularization choice [Cross-Domain Video Concept Detection using Adaptive SVMs, ACM Multimedia 2007] [Learning categories from few examples with multi model knowledge transfer, PAMI 2014] [From n to n+ 1: Multiclass transfer incremental learning, CVPR 2013] I believe this literature should be discussed in the related work section - It is true that the L2-SP-Fisher regularization was designed for life-long learning cases with a fixed task, however, this solution seems to work quite well in the proposed experimental settings. From my understanding L2-SP-Fisher can be considered the best competitor of L2-SP so I think the paper should dedicate more space to the analysis of their difference and similarities both from the theoretical and experimental point of view. For instance: -- adding the L2-SP-Fisher results in table 2 -- repeating the experiments of figure 2 and figure 3 with L2-SP-Fisher
assistant
 The paper proposes an analysis on different adaptive regularization techniques for deep transfer learning. Specifically it focuses on the use of an L2-SP condition that constraints the new parameters to be close to the ones previously learned when solving a source task. + The paper is easy to read and well organized + The advantage of the proposed regularization against the more standard L2 regularization is clearly visible from the experiments - The idea per se is not new: there is a list of shallow learning methods for transfer learning based on the same L2 regularization choice [Cross-Domain Video Concept Detection using Adaptive SVMs, ACM Multimedia 2007] [Learning categories from few examples with multi model knowledge transfer, PAMI 2014] [From n to n+ 1: Multiclass transfer incremental learning, CVPR 2013] I believe this literature should be discussed in the related work section - It is true that the L2-SP-Fisher regularization was designed for life-long learning cases with a fixed task, however, this solution seems to work quite well in the proposed experimental settings. From my understanding L2-SP-Fisher can be considered the best competitor of L2-SP so I think the paper should dedicate more space to the analysis of their difference and similarities both from the theoretical and experimental point of view. For instance: -- adding the L2-SP-Fisher results in table 2 -- repeating the experiments of figure 2 and figure 3 with L2-SP-Fisher