What a difference a pixel makes: An empirical examination of features used by CNNs for categorisation
Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -- even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.
 This paper adds to a growing body of literature which suggests that modern CNNs use qualitatively different visual strategies for object recognition compared to human observers. More specifically, the authors create shapeless object features (by adding noise masks in various forms or single pixels that are predictive of categorization to object images) to study how much CNNs rely on shape information (as humans would) as opposed to shapeless arbitrary statistical dependencies between pixels. The hypotheses tested are straightforward and the experiments cleverly answer these questions. On the negative side, there is nothing groundbreaking in this study. As acknowledged by the authors, the results are not all that novel in light of recent work that has already shown that one could conduct adversarial attacks by corrupting a single pixel as well as work that has shown that CNNs do not generalize to noise degradations they have not seen. Still, there is value in the work presented as the empirical tests described address the role of shape in object recognition with CNNs. In a sense, the present study offers a null result and obviously, the work would have been much more significant had the authors offered a mechanism to get CNNs to learn to prioritize "shape" features (then verifying that such network would work on CIFAR, but performed poorly on the shapeless images). Additional analysis involving visualization methods to further explain why shape features were ignored would have been a plus– with bonus points for providing a heuristic to determine the "shapelessness" of a convolution kernel.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
What a difference a pixel makes: An empirical examination of features used by CNNs for categorisation
Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -- even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper adds to a growing body of literature which suggests that modern CNNs use qualitatively different visual strategies for object recognition compared to human observers. More specifically, the authors create shapeless object features (by adding noise masks in various forms or single pixels that are predictive of categorization to object images) to study how much CNNs rely on shape information (as humans would) as opposed to shapeless arbitrary statistical dependencies between pixels. The hypotheses tested are straightforward and the experiments cleverly answer these questions. On the negative side, there is nothing groundbreaking in this study. As acknowledged by the authors, the results are not all that novel in light of recent work that has already shown that one could conduct adversarial attacks by corrupting a single pixel as well as work that has shown that CNNs do not generalize to noise degradations they have not seen. Still, there is value in the work presented as the empirical tests described address the role of shape in object recognition with CNNs. In a sense, the present study offers a null result and obviously, the work would have been much more significant had the authors offered a mechanism to get CNNs to learn to prioritize "shape" features (then verifying that such network would work on CIFAR, but performed poorly on the shapeless images). Additional analysis involving visualization methods to further explain why shape features were ignored would have been a plus– with bonus points for providing a heuristic to determine the "shapelessness" of a convolution kernel.
assistant
 This paper adds to a growing body of literature which suggests that modern CNNs use qualitatively different visual strategies for object recognition compared to human observers. More specifically, the authors create shapeless object features (by adding noise masks in various forms or single pixels that are predictive of categorization to object images) to study how much CNNs rely on shape information (as humans would) as opposed to shapeless arbitrary statistical dependencies between pixels. The hypotheses tested are straightforward and the experiments cleverly answer these questions. On the negative side, there is nothing groundbreaking in this study. As acknowledged by the authors, the results are not all that novel in light of recent work that has already shown that one could conduct adversarial attacks by corrupting a single pixel as well as work that has shown that CNNs do not generalize to noise degradations they have not seen. Still, there is value in the work presented as the empirical tests described address the role of shape in object recognition with CNNs. In a sense, the present study offers a null result and obviously, the work would have been much more significant had the authors offered a mechanism to get CNNs to learn to prioritize "shape" features (then verifying that such network would work on CIFAR, but performed poorly on the shapeless images). Additional analysis involving visualization methods to further explain why shape features were ignored would have been a plus– with bonus points for providing a heuristic to determine the "shapelessness" of a convolution kernel.