How transferable are features in deep neural networks?
Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.
This paper systematically studies the transferrability of deep convnet features across a few experimental settings.      The experiments are focused on the "AlexNet" architecture and the ILSVRC-2012 dataset, with the main set of results on transfer learning problems induced by dividing the 1000 ILSVRC-2012 object classes into random 500/500 splits.      The N-layer network is trained on split A, then layers 1 through k are copied to a new network and layers k+1 through N are randomly initialized and retrained on split B.      Results are reported across both choices of allowing layers 1 through k to be fine-tuned vs. frozen, and across all choices of k.      Another set of results shows these in the setting of a manually chosen split of natural vs. man-made classes, and yet another set of results explores using random, un-trained weights in some layers.      Many interesting observations are made which were not previously reported, particularly results #3 and #5 in section 4.1.      Despite the large number of papers that either extract features from or fine-tune the ImageNet-pretrained "AlexNet" CNN, this is the first work I know of to do a clean, thorough exploration of the space of choices one has to make when using this class of methods.      While the paper offers little in the way of technical or mathematical novelty, it does not strive to -- it provides an extensive, clean set of numerical answers to many commonly asked questions, and I believe its results give valuable practical insight to researchers and practitioners of modern deep learning methods.      The authors also promise to release the code they use to run these experiments, allowing other researchers to explore how these experiments generalize to other architectures, etc.