Various methods of measuring unit selectivity have been developed in order to understand the representations learned by neural networks (NNs). Here we undertake a comparison of four such measures on AlexNet, namely, localist selectivity Bowers et al. (2014), precision (Zhou et al., 2015, class-conditional mean activity selectivity CCMAS; Morcos et al. (2018), and a new measure called top-class selectivity. In contrast with previous work on recurrent neural networks (RNNs), we fail to find any 100% selective 'localist units' in AlexNet, and demonstrate that the precision and CCMAS measures provide a much higher level of selectivity than is warranted, with the most selective hidden units only responding strongly to a small minority of images from within a category. We also generated activation maximization (AM) images that maximally activated individual units and found that under (5%) of units in fc6 and conv5 produced interpretable images of objects, whereas fc8 produced over 50% interpretable images. Furthermore, the interpretable images in the hidden layers were not associated with highly selective units. These findings highlight the problem with current selectivity measures and show that new measures are required in order to provide a better assessment of learned representations in NNs. We also consider why localist representations are learned in RNNs and not AlexNet.
Summary - This paper analyzes the selectivity of individual units in CNNs. The authors analyze existing techniques such as precision selectivity, class-conditional mean activity selection and localist sensitivity. These methods are analyzed in the context of AlexNet and ImageNet. The authors also use Activation Maximization (AM) techniques for visualizing single-unit representations in CNNs.
Paper strengths
- The authors have minutely examined each of the metrics and the underlying assumptions they make. 
    - Example - Number of images used for computing the precision threshold in Zhou et al., 2014; The wrongly stated range of CCMAS [0, 1]. Considering the second highest CCMAS class is a good way of handling multiple classes that activate a single unit.
- The results of this paper are surprising compared to existing work. The authors have made a surprising discovery and done a good job of both presenting it well and experimentally validating it. The paper raises interesting questions and this should inspire future work in understanding networks.
- Figure 2 is insightful - It compares the various different interpretations of selectivity for a single unit in fc6. It shows how the mean activating class and the maximally activating class can be semantically very different. It also shows that despite the high precision and CCMAS score, the unit cannot be labelled as a detector for the single concept "custard apple". More such results are presented in the Appendix (e.g. Fig A6)
- The human study in Section 3.3 is a good way to evaluate the generated AM images.
Paper weaknesses
- One of the major weaknesses of this paper is that it uses only ImageNet images to evaluate the units. As this is limited to 1000 classes, the authors cannot probe other visual concepts such as color, texture, materials for the units. As an example, Network dissection (Zhou et al., 2017) proposes a dataset called Broden which has many diverse sets of visual concepts labeled. This paper focuses only on one definition of selectivity - selecting objects. This should be made explicit and the authors have not done a good job of clarifying this assumption or showing that it exists.
- All of the analysis is limited to AlexNet. With modern architectures that use residual/skip connections, it is not clear how well this analysis will generalize. It is an open question if the authors work overfits to AlexNet.
- The jitterplots are hard to understand especially if there are many overlapping "dots" (samples). Since the y-axis values are not really meaningful anyway, using a histogram to see how many samples have a particular activation value is easier. A possible suggestion for Figure 2(a): split into two parts - 1) histogram of all samples; 2) histogram of the highest mean activating class.
- The organization of the paper could be improved. The sections in the paper are not well connected.