The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of largescale labeled data for training. On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images. It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations. In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models. This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition. Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy. Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them. We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition.
The paper proposed a novel image classifier comparison approach that went beyond one fixed testing set for all. Instead, for a pair of classifiers to be compared, it advocated to sample their "most disagreed" test set from a large corpus of unlabeled images. The level of disagreement was measured by a semantic-aware distance derived from WordNet ontology. Because of the efficacy of such "worst-case" comparison, the needed set size is very small and thus minimizes the human annotation workload. 
The proposed MAD competition distinguishes classifiers by finding their respective counterexamples. It is therefore an "error spotting" mechanism, rather than a drop-in replacement of standard test accuracy. I feel the approach to implicitly assume that the classifiers to be compared are already "reasonably accurate"; since if not, both classifiers might be easily falsified by certain trivial examples, making the "disagreed examples" not as meaningful. If that is true, I would suggest the authors to make this hidden assumption clearer in the paper
The idea shows clear liaison to the "differential testing" concept in software engineering besides the cited work of perceptual quality assessment. The idea has a cross-disciplinary nature and is fairly interesting to me. I can see the paper to be of interest to a quite broad audience and can motivate many subsequent works. 
One minor comment: for images in "Case III", the authors considered them "contribute little to performance comparison between the two classifiers" and therefore did not source labels for them. However, since the authors adopted an affinity-aware distance, two incorrect predictions can still be compared based on their semantic tree distances to the true class.