While the accuracy of image classification achieves significant improvement with deep Convolutional Neural Networks (CNN), training a deep CNN is a timeconsuming task because it requires a large amount of labeled data and takes a long time to converge even with high performance computing resources. Finetuning, one of the transfer learning methods, is effective in decreasing time and the amount of data necessary for CNN training. It is known that fine-tuning can be performed efficiently if the source and the target tasks have high relativity. However, the technique to evaluate the relativity or transferability of trained models quantitatively from their parameters has not been established. In this paper, we propose and evaluate several metrics to estimate the transferability of pre-trained CNN models for a given target task by featuremaps of the last convolutional layer. We found that some of the proposed metrics are good predictors of fine-tuned accuracy, but their effectiveness depends on the structure of the network. Therefore, we also propose to combine two metrics to get a generally applicable indicator. The experimental results reveal that one of the combined metrics is well correlated with fine-tuned accuracy in a variety of network structures and our method has a good potential to reduce the burden of CNN training.
This paper aims to speed up finetuning of pretrained deep image classification networks by predicting the success rate of the process without running it. The authors suggest running samples from the target task on the (trained) source model, and computing a few sensible measures from the final output layer which indicate how well the trained features are separating the target images. Many experiments are presented, and some of them seem promising. 
Overall the idea is, while simple, interesting and potentially promising: DL Training costs have been rising significantly in the past few years, and being able to make sensible predictions about which source dataset/task combination best fits a target task would be a great contribution. Nonetheless, there are some methodological concerns that cast doubt about the presented results, which would need to be addressed before making this paper ready for publication.
Comments:
1. The authors present multiple experimental results, many of which indicate a somewhat noisy signal. The only method that works on most tasks, combining S5 and S6, is somewhat underreported. How many different \alpha values did the authors consider? how were they selected? Was the same alpha values used for each all experiments? Given this large set of measures and combinations of measures experimented with in this paper, I am left wondering whether this approach would generalize to new target datasets.
2. A major assumption the authors are making is that there is a single number that determines whether a given trained architecture would transfer well to a new target task. But the finetuning process is affected by many factors (e.g, hyperparameters, random seeds), and thus it might be that with a different hyperparameter selection, the observed correlations would look completely different. I would have liked to see at the very least an analysis of the correlation between multiple runs of finetuning that show that they are well correlated before computing correlations with external measures.
3. The authors point out that transfer learning works poorly on medical imaging (Sato et al., 2018). It would be interesting to experiment with such datasets and observe whether the proposed method is able to make accurate predictions in this domain.
4. The experiments regarding truncating CNNs seem interesting, and I was disappointed to find out they are not part of the main paper.
Other comments:
1. Moons et al. (2016) and Molchanov et al. (2017) speed up *inference* and not training.
2. The authors argue that Mahajan et al. (2018) "did not apply it (their method) for detection or segmentation tasks.". However, neither did this paper.
3. Writing: 
-- Several typos and grammatical errors across the paper. For instance:
- "*In* many previous research efforts suggested" ("in" should be dropped)
- Hypothesis H1 is ungrammatical 
-- Many of the citations were in the wrong format (intro: Canziani et al. 2016), repetitive (3.5: Yaguchi et al.), etc.