Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyper-parameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several common practices of setting hyper-parameters for fine-tuning. Our findings are based on extensive empirical evaluation for fine-tuning on various transfer learning benchmarks.
(1) While prior works have thoroughly investigated learning rate and batch size, momentum for fine-tuning is a relatively unexplored parameter. We find that the value of momentum also affects fine-tuning performance and connect it with previous theoretical findings. (2) Optimal hyper-parameters for fine-tuning in particular the effective learning rate are not only dataset dependent but also sensitive to the similarity between the source domain and target domain. This is in contrast to hyper-parameters for training from scratch. (3) Reference-based regularization that keeps models close to the initial model does not necessarily apply for "dissimilar" datasets. Our findings challenge common practices of finetuning and encourages deep learning practitioners to rethink the hyper-parameters for fine-tuning.
This submission studies the problem of transfer learning and fine tuning. This submission proposes four insights: Momentum hyperparameters are essential for fine-tuning; When the hyperparameters satisfy some certain relationships, the results of fine-tuning are optimal; The similarity between source and target datasets influences the optimal choice of the hyperparameters; Existing regularization methods for DNN is not effective when the datasets are dissimilar. This submission provides multiple experiments to support their opinion.
Pros:
+  This submission provides interesting facts that are omitted in previous research works.
+  This submission examines the previous theoretical results in empirical setting and finds some optimal hyperparameter selection strategies.
+  This submission provides many experiment results of fine-tuning along with its choice of hyperparameters that could be taken as baselines in future researches.
Cons:
-	All experiments results are based on same backbone, which makes all discoveries much less reliable. More experiments on other backbones are necessary. Furthermore, this submission claims that the regularization methods such as L2-SP may not work on networks with Batch Normalization module. But there is no comparison on networks without BN.
-	Providing a complete hyperparameter selecting strategy for fine-tuning could be an important contribution of this submission. I suggest authors to think about it.
-	This submission claim that the choice of hyperparameters should depend on similarity of different domains. But this submission does not propose a proper method for measure the similarity or provide detailed experiments on previous measurements. 
-	It seems that the MITIndoors Dataset is not similar with ImageNet from the semantic view. This submission does not provide similarity measurement between these datasets. Why the optimal momentum is 0? 
-	The effective learning rate and ‘effective’ weight decay are not first given in this submission. This makes the novelty of this submission relatively weak. Authors only test these strategies in fine-tuning setting and find that they also work with a different initialization.
-	It seems that merely searching for learning rate and weight decay hyperparameters (as Kornblith et al. (2018) did) on a fixed momentum is Ok if there is a most effective relationship between learning rate and momentum. So the discoveries in the first part that a 0 momentum can be better is based on a careless search of learning rates?
-	This submission omits that Kornblith et al. (2018) also referred to the fact that the momentum parameter of BN is essential for fine-tuning and provided a strategy in section A.5. Discussion about this strategy will make this submission more complete.
This submission gives important discoveries about the hyperparameter choice in the fine-tuning setting. But there are several flaws in this submission. I vote for rejecting this submission now but I expect authors to improve the submission in the future version.