Convolutional neural networks (CNNs) were inspired by human vision and, in some settings, achieve a performance comparable to human object recognition. This has lead to the speculation that both systems use similar mechanisms to perform recognition. In this study, we conducted a series of simulations that indicate that there is a fundamental difference between human vision and CNNs: while object recognition in humans relies on analysing shape, CNNs do not have such a shape-bias. We teased apart the type of features selected by the model by modifying the CIFAR-10 dataset so that, in addition to containing objects with shape, the images concurrently contained non-shape features, such as a noise-like mask. When trained on these modified set of images, the model did not show any bias towards selecting shapes as features. Instead it relied on whichever feature allowed it to perform the best prediction -even when this feature was a noise-like mask or a single predictive pixel amongst 50176 pixels. We also found that regularisation methods, such as batch normalisation or Dropout, did not change this behaviour and neither did past or concurrent experience with images from other datasets.
The paper seeks to establish via a series of well-designed experiments that CNNs trained for image classification differ in a fundamental way from human vision – they don’t encode shape-bias like human vision. Towards this goal, the authors modified the training data with ‘shortcut’ features to be functions of the category label using single diagnostic pixels and their placements, noise masks (salt and pepper, additive) and their parameters and demonstrate that image categorization CNNs learn whatever statistical features are there in the data most relevant to the learning task.
Investigation of the properties of neural architectures like CNNs and using the understanding thus developed to create better neural architectures, learning algorithms and training paradigms are good directions for the community and from that perspective, the direction explored in the paper is of great relevance and interest to the community. 
The paper presents careful experimentation to establish that image categorization CNNs learn the statistical features most relevant to the learning task. And, it seems to satisfactorily demonstrate this. It shows that such features could be single pixels, noise masks and even parameters of stochastic distributions which randomly produce these features, as long as the parameters are predictive of the image category. The experiments are well designed and they demonstrate this point quite well. They also demonstrate the well-known problem of catastrophic forgetting.
Nonetheless, there are significant drawbacks in the presented work:
1.	The experiments don't seem to effectively demonstrate the main claim of the paper that categorization CNNs do not have inductive shape bias (encode shape information). (Let’s make this claim more concrete: categorization CNNs when trained via supervised learning with paired training data of {(image, category_label)} do not have inductive shape bias.)
The best way to demonstrate this would have been to subject a trained image-categorization CNN to test data with object shapes in a way that the appearance information couldn’t be used to predict the object label. The paper doesn’t do this. None of the experiments logically imply that with an unaltered training regime, a trained network would not be predictive of the category label if shapes corresponding to that category are presented. 
2.	Due to the surprising results (especially the intensity of observed effects), we tried to reproduce some results from the paper in our lab and faced difficulties in doing so:
a.	We tried to replicate Figure 4(a) 'nopix' and 'same' cases on a standard setting (VGG-12-BN on CIFAR-10). The results deviated significantly (33%-72% margin) on ‘nopix’ case from the results reported in the paper on a much stronger setting (1/3072 pixels vs 1/50176 as in the paper). Please let me know any crucial settings (see below) that we might have missed.
Details: We used the vgg-cifar10 repository by chengyangfu. The only additions was fixing the pixel values while sending in the data. The code is anonymized and hosted here: https://file.io/qiziAK. The pixel values in CIFAR-10 using the pytorch dataloader are between [-0.45, 0.45] theoretically, typically much smaller. We set the (0,0) RGB pixels categorically spacing it uniformly from [-0.25, 0.25), [-0.025, 0.025), [-0.0025, 0.0025) as a simple experiment. The third case did not suffer any decrease in the nopix case or any increase in the pix at all. The first case showed significant deviations from the claimed results with the no-pix resulting in ~43% accuracy which is 33% off vis-à-vis the results in the paper. The ‘same’ setting didn’t achieve 100% either though it got close - achieving 98.4%. 
Summary: The paper presents an important line of investigation to understand the properties of CNNs. However, it fails to effectively demonstrate its main claim. Further, we had difficulties in reproducing the results. As it stands, the submission is not of publishable quality.
I encourage the authors to do more careful experimentation to demonstrate their main claim and perhaps work on strategies to encourage CNNs to learn more meaningful features, including ‘shape’-features and submit to a future conference.
Revision: Updated my rating to acknowledge that the reproducibility issue is addressed.