Neural networks have been shown vulnerable to adversarial samples. Slightly perturbed input images are able to change the classification of accurate models, showing that the representation learned is not as good as previously thought. To aid the development of better neural networks, it would be important to evaluate to what extent are current neural networks' representations capturing the existing features. Here we propose a way to evaluate the representation quality of neural networks using a novel type of zero-shot test, entitled Raw Zero-Shot. The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features. To evaluate the soft-labels of unknown classes, two metrics are proposed. One is based on clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of a given correct soft-label. Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks. Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.
=== A. Summary ===
This paper proposes to evaluate the representation (here, the output of 9 probabilities for 9 classes at the last softmax output layer) of an image taken from an unseen category by two metrics:
- The first metric (DBI) essentially measures how compact the learned manifold is (i.e. how nearby are the softmax probability vectors of images from an unseen category). 
- The second metric basically measures how close a 9-D probability output vector ("soft labels" in the paper) of a classifier trained on the 9 classes is to a "ground-truth" which is the same 9-D vector but obtained when the classifier is trained on the full 10 CIFAR-10 classes.  
The authors computed these "representation measures" for a set of 7 networks (6 convnets + CapsuleNet) trained on CIFAR-10.
Then, a set of white-box adversarial attacks were performed on these 7 networks and the results (both how accurate the classifier is, and the amount of L2 perturbations required to change a label) were used as a "robustness measures".
The authors then claim that the ranking of the 7 classifiers based on the representation measures match the rankings derived from the robustness measures.
The paper attempts to find the correlation between the representation quality of a classifier with its adversarial robustness. This general direction is important and worth pursuing!
However, the claimed correlation is very weakly supported by the evidence in the paper.
=== B. Decision ===
Reject.
The aim of connecting the representation quality with adversarial robustness is interesting!
However, this paper has several major issues:
0. The paper uses the wrong citation format of ICLR (changing from LastName et al. 2019 --> (12) ). The paper maybe 8.5 to 9 pages long if using the original format. If this format violation was accepted, it would be unfair to other submissions and be a bad example this citation violation would be OK.
1. The paper has many claims (e.g. "DBI metric matches extremely well") that were not supported by evidence or clarified.
2. The paper claims to study classifiers' "representation quality". However, all the experiments were conducted on only the 10-class CIFAR-10 dataset and the "representations" are taken at the softmax layer as opposed to some mid-CNN feature layer (e.g. the popular fc7 in AlexNet trained on the 1000-class ImageNet). 
3. The key Definition in Sec. 2 that the paper hinges on has 0 references and is a debatable definition.
=== C. Suggestions for Improvement ===
- I'd advise the authors to perform careful literature review to identify where the gap to fill is before conducting the research. There is a ton of work that has been done in the intersection of zero-shot, adversarial examples, deep features, and classifier caliberation. 
For example, [1] has looked at how out-of-distribution samples can be represented by the deep features of a classifier:
[1] Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563-1572).
- This work [2] has shown adding a "center loss" to force the deep features to be closer to the feature centroid (i.e. related to your DBI measure) helps improve the adversarial robustness. Indeed, the visualization in Fig. 3 is very interesting and is consistent with the result in [2].
[2] Agarwal, C., Nguyen, A., & Schonfeld, D. (2019, September). Improving Robustness to Adversarial Examples by Encouraging Discriminative Features. In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 3801-3505). IEEE.
- The word "representation quality" here is confusing when it refers to the soft labels, which is more relevant to *caliberation* than quality of deep features as the title implies. And the paper misses this body of work in the related work. See here: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html
- Try to avoid "extremely" or "clearly" in claims (and in general scientific writing) because they tend to be overclaims and it is hard for readers to interpret those intensifiers. 
- I don't see how the rankings by DBI or AM are similar to the original rankings by adversarial measures at all (except for CapsNet). At least, reporting Pearson correlation (or any statistical similarity measure) would be more convincing.
- Section 2 is highly debatable and has no references for the "Definition".
- I'd perform this work on ImageNet, at least to follow the intuition in Sec. 2. With only 10 CIFAR-10 classes, it is unrealistic how to semantically represent a "frog" using the other 9 classes (truck, airplane, etc).