Selectivity metrics can overestimate the selectivity of units: a case study on AlexNet
Various methods of measuring unit selectivity have been developed in order to understand the representations learned by neural networks (NNs). Here we undertake a comparison of four such measures on AlexNet, namely, localist selectivity, precision (Zhou et al, ICLR 2015), class-conditional mean activity selectivity CCMAS; (Morcos et al, ICLR 2018), and a new measure called top-class selectivity. In contrast with previous work on recurrent neural networks (RNNs), we fail to find any 100\% selective `localist units* in AlexNet, and demonstrate that the precision and CCMAS measures provide a much higher level of selectivity than is warranted, with the most selective hidden units only responding strongly to a small minority of images from within a category. We also generated activation maximization (AM) images that maximally activated individual units and found that under (5\%) of units in fc6 and conv5 produced interpretable images of objects, whereas fc8 produced over 50\% interpretable images. Furthermore, the interpretable images in the hidden layers were not associated with highly selective units. These findings highlight the problem with current selectivity measures and show that new measures are required in order to provide a better assessment of learned representations in NNs. We also consider why localist representations are learned in RNNs and not AlexNet.
 This is a paper with scattered potentially interesting ideas. But the execution is limited and the writing poor with critical details lacking. A major limitation of the paper is that it is not clear what contribution it makes. Some of the analyses are indeed interesting but 1) these analyses are mostly descriptive and 2) they are limited to one particular (outdated) architecture. How would batch norm or residual connections or any of the developments that have happened since AlexNet affect these results? As a side note, the references/comparisons between AlexNet and recurrent nets (see abstract, etc) are misleading. This is based on the claim that Bowers et al (2014) qualitatively different results but this is for entirely different domains (words). Indeed what could have made potentially the work more relevant would have been to show some kind of benchmarking between AlexNet and alternative architectures (possibly RNNs). As such the current study does not contribute much except for comparing different semi-arbitrary measures of selectivity for one specific (outdated) network architecture trained on a particular problem (ILSVRC). **** Minor points: The study is limited to correctly classified images as stated on page 3. This seems like a major confound in a study aimed at understanding the visual representations learned. It seems to me that the conclusions of the paper could be heavily biased because of this (when computing any measure based on inter and intraclass responses). In general, this is a relatively poorly written paper which would be hard to reproduce. For instance, the image generation for activating units (assuming it is novel) could be interesting but it is not even described with sufficient details so as to reproduce the results.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Selectivity metrics can overestimate the selectivity of units: a case study on AlexNet
Various methods of measuring unit selectivity have been developed in order to understand the representations learned by neural networks (NNs). Here we undertake a comparison of four such measures on AlexNet, namely, localist selectivity, precision (Zhou et al, ICLR 2015), class-conditional mean activity selectivity CCMAS; (Morcos et al, ICLR 2018), and a new measure called top-class selectivity. In contrast with previous work on recurrent neural networks (RNNs), we fail to find any 100\% selective `localist units* in AlexNet, and demonstrate that the precision and CCMAS measures provide a much higher level of selectivity than is warranted, with the most selective hidden units only responding strongly to a small minority of images from within a category. We also generated activation maximization (AM) images that maximally activated individual units and found that under (5\%) of units in fc6 and conv5 produced interpretable images of objects, whereas fc8 produced over 50\% interpretable images. Furthermore, the interpretable images in the hidden layers were not associated with highly selective units. These findings highlight the problem with current selectivity measures and show that new measures are required in order to provide a better assessment of learned representations in NNs. We also consider why localist representations are learned in RNNs and not AlexNet.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This is a paper with scattered potentially interesting ideas. But the execution is limited and the writing poor with critical details lacking. A major limitation of the paper is that it is not clear what contribution it makes. Some of the analyses are indeed interesting but 1) these analyses are mostly descriptive and 2) they are limited to one particular (outdated) architecture. How would batch norm or residual connections or any of the developments that have happened since AlexNet affect these results? As a side note, the references/comparisons between AlexNet and recurrent nets (see abstract, etc) are misleading. This is based on the claim that Bowers et al (2014) qualitatively different results but this is for entirely different domains (words). Indeed what could have made potentially the work more relevant would have been to show some kind of benchmarking between AlexNet and alternative architectures (possibly RNNs). As such the current study does not contribute much except for comparing different semi-arbitrary measures of selectivity for one specific (outdated) network architecture trained on a particular problem (ILSVRC). **** Minor points: The study is limited to correctly classified images as stated on page 3. This seems like a major confound in a study aimed at understanding the visual representations learned. It seems to me that the conclusions of the paper could be heavily biased because of this (when computing any measure based on inter and intraclass responses). In general, this is a relatively poorly written paper which would be hard to reproduce. For instance, the image generation for activating units (assuming it is novel) could be interesting but it is not even described with sufficient details so as to reproduce the results.
assistant
 This is a paper with scattered potentially interesting ideas. But the execution is limited and the writing poor with critical details lacking. A major limitation of the paper is that it is not clear what contribution it makes. Some of the analyses are indeed interesting but 1) these analyses are mostly descriptive and 2) they are limited to one particular (outdated) architecture. How would batch norm or residual connections or any of the developments that have happened since AlexNet affect these results? As a side note, the references/comparisons between AlexNet and recurrent nets (see abstract, etc) are misleading. This is based on the claim that Bowers et al (2014) qualitatively different results but this is for entirely different domains (words). Indeed what could have made potentially the work more relevant would have been to show some kind of benchmarking between AlexNet and alternative architectures (possibly RNNs). As such the current study does not contribute much except for comparing different semi-arbitrary measures of selectivity for one specific (outdated) network architecture trained on a particular problem (ILSVRC). **** Minor points: The study is limited to correctly classified images as stated on page 3. This seems like a major confound in a study aimed at understanding the visual representations learned. It seems to me that the conclusions of the paper could be heavily biased because of this (when computing any measure based on inter and intraclass responses). In general, this is a relatively poorly written paper which would be hard to reproduce. For instance, the image generation for activating units (assuming it is novel) could be interesting but it is not even described with sufficient details so as to reproduce the results.