Pre-trained Deep Convolutional Neural Network (CNN) features have popularly been used as full-reference perceptual quality features for CNN based image quality assessment, super-resolution, image restoration and a variety of imageto-image translation problems. In this paper, to get more insight, we link basic human visual perception to characteristics of learned deep CNN representations as a novel and first attempt to interpret them. We characterize the frequency and orientation tuning of channels in trained image classification deep CNNs (e.g., VGG-16) by applying grating stimuli of different spatial frequencies and orientations as input. We observe that the behavior of CNN channels as spatial frequency and orientation selective filters can be used to link basic human visual perception models to their characteristics. Doing so, we develop a theory to get more insight into deep CNN representations as perceptual quality features. We conclude that sensitivity to spatial frequencies that have lower contrast masking thresholds in human visual perception and a definite and strong orientation selectivity are important attributes of deep CNN channels that deliver better perceptual quality features.
I thank the authors for their detailed response. Some of my questions have been addressed in the rebuttal, but as far as I can tell very few modifications have been made to the paper: mainly an additional super-resolution experiment in the appendix, which goes in a good direction, but currently comes across as quite preliminary. So I think the paper still has most of the weaknesses I mentioned and thus it is not quite fit for publication. But I do encourage the authors to strengthen the experiments (for instance by addressing the points I raised, or by some other means) and resubmit elsewhere.
---
The paper proposes an approach to analyzing the properties of "perceptual metrics" used in deep learning image generation methods. "Perceptual metrics" are computed by measuring distances between images not in the pixel space, but i na feature space of a pre-trained cNN. The proposed analysis method, inspired by studies of human perception, is based on measuring the response of these features to sinusoidal gratings of varying frequency or orientation. Based on these responses, the paper proposes a "Perceptual Efficacy Score" that should measure the importance of certain feature in the feature maps for the performance of a perceptual metric. Experiments show that indeed distances measured between features with high score better correlate with human judgement of image similarity than distances between features with a lower score.
I find the paper quite interesting, but lean towards rejection at this point. This i smainly because the experiments seem somewhat anecdotal and incomplete, see below for further details.
Pros:
1) Application of methods from psychology/neuroscience to artificial neural networks is an interesting avenue of work. Moreover, better unsderstanding of "perceptual metrics" is of wide interest for various image processing applications.
2) The proposed score seems to indeed correlate quite well with the importance of features for human judgement of image similarity.
3) Presentation is mainly clear.
Cons:
1) Experiments are not very exhaustive and at times a bit confusing. For instance:
1a) Results are sometimes presented in a confusing way. In Figure 4 first of all it is not quite clear what points correspond to I guess each point is an image) and, second, it is not very obvious that the correlation is higher i none of the plots. In tables 1 and 2 it is confusing that different percentiles for H and L are used for different networks/layers. Is this based on some tuning? Then the tuning process should be clearly explained. Moreover, it might be useful to report the full curves of performance as a function of the percentage of features used.
1b) There are no baselines and there is not much justification of computing the "Perceptual Efficacy" score the way it is computed. What if one uses only the orientation-based score? Or only the frequency-based? What if one selects the most relevant features in a data-driven way (based on correlation on a training set)? What if one selects subsets of features randomly? 
1c) While the method is inspired by methods used for studying natural vision systems, there is no connection to human experiments. It would be interesting to see a comparison of frequency and orientation tuning of features in a CNN to human cells (as I understand, the latter should be available in prior works?).
1d) It would be great to see the selected features be used not only for offline image similarity assessment, but also for training image processing models - in the end, this has been the main use of "perceptual metrics". Do they lead to improved results?
1e) Since the paper is about (subjective) image quality, it might be useful to show some qualitative results, potentially in the appendix if space is an issue.
2) There are some issues with the presentation:
2a) I had a hard time understanding what exactly "Contrast Sensitivity Function" and "contrast masking" are.
2b) Minor issues:
- In the abstract: "trained object detection deep CNNs" - I guess image classification is meant
- Beginning of Section 2: "Section. 2", "convolution layer as collection channels"
- Section 4: "corresponds the the peak:
- Section 5.2 "Berkeley-Adobpe"