DELTA: DEEP LEARNING TRANSFER USING FEATURE MAP WITH ATTENTION FOR CONVOLUTIONAL NET- WORKS
Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA aligns the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in a supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L 2 and L 2 -SP . The experiment results show that our method outperforms these baselines with higher accuracy for new tasks.
This is a reasonable paper based on a simple intuition. The authors have noticed that some of the state of the art methods (they use Li et al - ICML18 as the main reference) are using only some simple normalization for improving the transfer learning and as such they propose preserving the outer layer output of the target network and aligning it with the one of the source network. On top of that they also propose modeling the difference of feature maps considering an attention mechanism obtain through supervised learning. 
The idea in itself is interesting and valuable. However, I have had some difficulty in understanding precisely how the "behavior" is really regularized. While I understand what is depicted in Figure 1 I'm not completely sure this really means that the network behavior is regularized rather than simply correlating the two outputs. In the evaluation, the authors present in Figure 4 some qualitative examples but I would have expected to see some quantitative evaluation of this. I would have liked to see experiments on some larger datasets that are commonly used in computer vision (e.g., Caltech 256 is rather old even if it has been used in Li et al.). The quantitative results in Table 1 and 2 indicate some slight improvement but I'm not completely convinced that this is really significant in the end. The results in Figure 4 tend to show that with the attention mechanism there is a central bias and most of the results tend to be concentrated on the center of the image (in this case the result might also be correct but the examples presented are not too eloquent).