We propose laconic classification as a novel way to understand and compare the performance of diverse image classifiers. The goal in this setting is to minimise the amount of information (aka. entropy) required in individual test images to maintain correct classification. Given a classifier and a test image, we compute an approximate minimal-entropy positive image for which the classifier provides a correct classification, becoming incorrect upon any further reduction. The notion of entropy offers a unifying metric that allows to combine and compare the effects of various types of reductions (e.g., crop, colour reduction, resolution reduction) on classification performance, in turn generalising similar methods explored in previous works. Proposing two complementary frameworks for computing the minimal-entropy positive images of both human and machine classifiers, in experiments over the ILSVRC test-set, we find that machine classifiers are more sensitive entropy-wise to reduced resolution (versus cropping or reduced colour for machines, as well as reduced resolution for humans), supporting recent results suggesting a texture bias in the ILSVRC-trained models used. We also find, in the evaluated setting, that humans classify the minimal-entropy positive images of machine models with higher precision than machines classify those of humans.
The paper proposes and studies a task where the goal is to classify an image that has been intentionally degraded to reduce information content - hence the name "laconic" image classification. The motivation for this task is to compare human and machine performance in a task that deviates from the standard ImageNet setup. To make different content reductions comparable, the authors measure the (approximate) entropy of an image via its PNG-compressed file size. As image transformations, the authors utilize quantization, downsampling, cropping, and a combination of the above. The authors find that convnets with higher accuracy are also more robust to these perturbations, and that humans perform well on the minimum-entropy examples of the networks (but not vice-versa).
Overall I find the comparison of human and machine performance interesting and hence recommend accepting the paper. However, there are multiple directions that could possibly strengthen the core experiment. Hence I only give a weak accept at this point. Concretely, these directions are:
- Do the results in the paper change if a different entropy measure is used (e.g., JPEG compression)?
- As suggested by the authors, training networks to be robust to the "laconic" image perturbations could be an interesting direction. For instance, standard data augmentation with the proposed perturbations would be a relevant baseline.
- To aid replicability and to compare the performance of different human test subjects, it would be interesting to conduct the experiment also on a crowdsourcing platform such as Mechanical Turk (as a complement to, not a replacement for, the university population in the paper).
- Also to add replicability and to make it easier to compare different human accuracy evaluations, it would be good to measure how well the annotators perform on the unperturbed images and on a simple noise transformation (e.g., Gaussian noise).
- It would be good to know how approximate the entropy measures in the paper are, e.g., to understand why humans perform worse in the "combined" perturbation setting.
- How did the results from the control group and the open / online evaluation differ?
In addition, I have the following suggestions for improving the paper:
- For the related work section, the authors may find the following papers on robustness of convnets to distortions interesting:
* Manitest: Are classifiers really invariant?
https://arxiv.org/abs/1507.06535
* Exploring the Landscape of Spatial Robustness
https://arxiv.org/abs/1712.02779
* Spatially Transformed Adversarial Examples
https://arxiv.org/abs/1801.02612
* Semantic Adversarial Examples
https://arxiv.org/abs/1804.00499
- It could be helpful for the reader to see some example images of the different transformations in the main text.
- Section 6: "[...] a bias in texture for images trained on the ILSVRC dataset [...]" - should this be "classifiers" instead of "images"?
- Section 6: "[...] would be interesting to explore in future" - insert "the" before "future"?