IMAGENET-TRAINED CNNS ARE BIASED TOWARDS TEXTURE; INCREASING SHAPE BIAS IMPROVES ACCURACY AND ROBUSTNESS
Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNettrained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'StylizedImageNet', a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.
Figure 1: Classification of a standard ResNet-50 of (a) a texture image (elephant skin: only texture cues); (b) a normal image of a cat (with both shape and texture cues), and (c) an image with a texture-shape cue conflict, generated by style transfer between the first two images.
Review of ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. 
In this submission, the authors provide evidence through clever image manipulations and psychophysical experiments that CNNs image recognition is strongly influenced by texture identification as opposed the global object shape (as opposed to humans). The authors attempt to address this problem by using image stylization to augment the training data. The resulting networks appear much more aligned with human judgements and less biased towards image textures.
If the authors address my major concerns, I would increasing my rating 1-2 points.
Major Comments:
The results of this paper are quite compelling and address some underlying challenges in the literature on how CNN's function. I particularly appreciated Figure 5 demonstrating how the resulting stylized-augmented networks more closely align with human judgements. Additionally, it is surprising to me how poor BagNet performs on Stylized-ImageNet (SIN) implying that ResNet-50 trained on Stylized ImageNet may be better perceptually aligned with global object structure. Very cool.
1. Please make sure to tone down the claims in your manuscript. Although I share enthusiasm for your results, please recognize that stating that your results are 'conclusive' is premature and not appropriate. (Conclusive requires more papers and much work by the larger scientific community for a hypothesis to become readily accepted). Some sentences of concern include:
  --> "These experiments provide conclusive behavioural evidence in favour of the texture hypothesis"
  --> "we conclude the following: Textures, not object shapes, are the most important cues for CNN object recognition."
I would prefer to see language such as "We provide evidence that textures provide a more powerful statistical signal then global object shape for CNNs." or "We provide evidence that CNNs are overly sensitive to textures in comparison to humans perceptual judgements". This would be more measured and better reflect what has been accomplished in this study. Please do a thorough read of the rest of your manuscript and identify other text accordingly.
2. Domain shifts and data augmentation. I agree with your comment that domain shifts present the largest confound to Figure 2. The results of Geirhos et al, 2018 (Figure 4) indicate that individual image augmentations/distortions do not generalize well. Given these results, I would like to understand what image distortions were used in training each and all of your networks. Did you try a baseline with no image distortions (and/or just Stylized-ImageNet)?
Although the robustness in Figure 6 are great, how much of this can be attributed solely to Stylized-ImageNet versus the other types of image distortions/augmentations in each network. For instance, would contrast-insensitivity in Stylized-ImageNet diminish substantially if no contrast image distortion were used during training?
3. Semantics of 'object shape'. I suspect that others in the field of computer vision may take issue with your definition of 'object shape'. Please provide a crisp definition of what you test for as 'object shape' in each of your experiments (i.e. "the convex outline of object segmentation", etc.).
Minor Comments:
- Writing style in introduction. Rather then quoting phrases from individual papers, I would rather see you summarize their ideas in your own language and cite accordingly. This would demonstrate how you regard for their ideas and how these ideas fit together.
- Figure 2. Are people forced to select a choice or could they select 'I don't know'? Did you monitor response times to see if the manipulated images required longer times for individuals to pass decisions? I would expect that for some of the image manipulations that humans would have less confidence about their choices and that to be reflected in this study above and beyond an accuracy score.
- In your human studies, please provide some discussion about how you monitored performance to guard against human fatigue or lack of interest.
- Why did you use AdaIN instead of the original Gatys et al optimization method for image stylization? Was there some requirement/need for fast image stylization?
- Do you have any comment on the large variations in the results across class labels in Figure 4? Are there any easy explanations for this variation across class labels?
- Please use names of Shape-ResNet, etc. in Table 2.
- Are Pascal-VOC mAP results with fixed image features or did you fine-tune (back-propagate the errors to update the image features) during training? The latter would be particularly interesting as this would indicate that the resulting network features are better generic features as opposed to having used better data augmentation techniques.
- A.2. "not not used in the experiment" --> "not used in the experiment"