Transfer learning for feature extraction can be used to exploit deep representations in contexts where there is very few training data, where there are limited computational resources, or when tuning the hyper-parameters needed for training is not an option. While previous contributions to feature extraction propose embeddings based on a single layer of the network, in this paper we propose a full-network embedding which successfully integrates convolutional and fully connected features, coming from all layers of a deep convolutional neural network. To do so, the embedding normalizes features in the context of the problem, and discretizes their values to reduce noise and regularize the embedding space. Significantly, this also reduces the computational cost of processing the resultant representations. The proposed method is shown to outperform single layer embeddings on several image classification tasks, while also being more robust to the choice of the pre-trained model used for obtaining the initial features. The performance gap in classification accuracy between thoroughly tuned solutions and the full-network embedding is also reduced, which makes of the proposed approach a competitive solution for a large set of applications.
The paper addresses the scenario when using a pretrained deep network as learnt feature representation for another (small) task where retraining is not an option or not desired. In this situation it proposes to use all layers of the network to extract feature from, instead of only one layer. 
Then it proposes to standardize different dimensions of the features based on their response on the original task. Finally, it discretize each dimension into {-1, 0, 1} to compress the final concatenated feature representation. 
Doing this, it shows improvements over using a single layer for 9 target image classification datasets including object, scene, texture, material, and animals.
The reviewer does not find the paper suitable for publication at ICLR due to the following reasons:
- The paper is incremental with limited novelty.
- the results are not encouraging
- the pipeline of standardization, discretization is relatively costly, the final feature vector still large. 
- combining different layers, as the only contribution of the paper, has been done in the literature before,  for instance:
“The Treasure beneath Convolutional Layers: Cross-convolutional-layer Pooling
for Image Classification” CVPR 2016