Deep convolutional neural networks are now widely deployed in vision applications, but the size of training data can bottleneck their performance. Transfer learning offers the chance for CNNs to learn with limited data samples by transferring knowledge from weights pre-trained on large datasets. On the other hand, blindly transferring all learned features from the source dataset brings unnecessary computation to CNNs on the target task. In this paper, we propose attentive feature distillation and selection (AFDS) that not only adjusts the strength of regularization introduced by transfer learning but also dynamically determines which are the important features to transfer. When deploying AFDS on ResNet-101, we achieve state-of-theart computation reduction at the same accuracy budget, outperforming all existing transfer learning methods. On a 10× MACs reduction budget, transfer learned from ImageNet to Stanford Dogs 120, AFDS achieves an accuracy that is 12.51% higher than its best competitor.
This paper proposes a method called attentive feature distillation and selection (AFDS) to improve the performance of transfer learning for CNNs. The authors argue that the regularization should constrain the proximity of feature maps, instead of pre-trained model weights. Specifically, the authors proposes two modifications of loss functions: 1) Attentive feature distillation (AFD), which modifies the regularization term to learn different weights for each channel and 2) Attentive feature selection (AFS), which modifies the ConvBN layers by predicts unimportant channels and suppress them. 
Overall, this is a good work in terms of theory and experimentation, thus I would recommend to accept it. The approach is well motivated, and the literature is complete and relevant. The author's argument is validated by experiments comparing the proposed AFDS method and other existing transfer learning methods. 
To improve this paper, the authors are suggested to address the following issues:
1.  Section 3.5 is not well organized. Besides, it is not mentioned what value the threshold hyper-parameter delta_m is set. 
2. In page 9, "MACs" is missing in the sentence "In Figure 3, ...the number of vs. the target..."