Classification images and spike triggered analysis have been widely used in psychophysics and neurophysiology to understand underlying mechanisms of sensory systems in humans and monkeys. In this paper, we leverage these techniques to investigate the inherent biases of deep neural networks and to obtain a first-order approximation of their functionality. We emphasize on convolutional neural networks (CNNs) since they are currently the state of the art methods in computer vision and are a good model of human visual processing. In addition, we also study multi-layer perceptrons, logistic regression and recurrent neural networks. Experimenting over three classic datasets, MNIST, Fashion-MNIST and CIFAR-10, we show that the computed bias maps resemble the target classes and when used for classification lead to an over two-fold performance than the chance. Further, we show that classification images can be used to attack a black box classifier and to detect adversarial patch attacks. Finally, we utilize spike triggered averaging to derive filters of CNNs and explore how the behavior of a network changes when neurons in different layers are modulated. Our effort illustrates a successful example of borrowing from neuroscience to study artificial neural networks and highlights the importance of cross-fertilization and synergy across machine learning, deep learning, and computational neuroscience.
This work inspects the bias existing in the neural network classifiers through two techniques from computational neuroscience, the classification image techniques and spike-triggered covariance (STA) analysis. The classification image is generated as the difference map between all the averaged images with predicted classes under noise perturbation. STA is used to generate the eigenvector as the input and response, as a visualization. Both of the tools are the standard psychophysics tools. 
The authors use the two tools to visualize the bias learned by various classifiers, CNN, MLP, and logistic regression, trained on three datasets MNIST, Fashion-MNIST and CIFAR 10. There are interesting patterns that emerged from the classifiers trained on MNIST and Fashion-MNIST by the classification image technique, but the result from the CIFAR 10 is not explainable. The classification image is further used to analyze the adversarial attack and defense while STA is used to visualize the internal units. 
Strengths:
The paper is well-written and easy to follow. With detailed and complete psychophysics experiments, several interesting properties of the filters and the biases of the neural network classifiers are revealed. It is great to see that more tools from psychophysics can be applied to understand neural network classifiers.
Weakness:
1. One concern I have is that the result on CIFAR-10 is not interpretable. See figure 3. On the other hand, the visualization of filters in higher layers (conv4 and conv6) from the net trained on CIFAR-10 in Figure 8 still looks like edge detectors. There are not so many discussions and explanations on what possibly happened here. 
2. The inferior result on CIFAR-10 might indicate that the techniques cannot be generalized to interpret large-scale networks such as AlexNet or resnet, trained on ImageNet or Places. It is 2019 now, I expect to see more experiments on at least AlexNet or VGG, rather than tiny sets MNIST or CIFAR. I will suggest the authors run experiments on AlexNet trained on ImageNet/Places, then we can compare the visualization with the other filter visualization to see the difference.
3. There is no comparison for other bias visualization methods. Following the idea of mean images, I think you can also show the mean images conditional on the predicted labels (average all the testing images with certain predicted labels). How the result from the psychophysics is different from that? How well the proposed methods are able to identify the bias? What other bias cannot be identified? Some discussions on the failure cases would be appreciated