Although convolutional neural networks (CNNs) are inspired by the mechanisms behind human visual systems, they diverge on many measures such as ambiguity or hardness. In this paper, we make a surprising discovery: there exists a (nearly) universal score function for CNNs whose correlation is statistically significant than the widely used model confidence with human visual hardness. We term this function as angular visual hardness (AVH) which is given by the normalized angular distance between a feature embedding and the classifier weights of the corresponding target category in a CNN. We conduct an in-depth scientific study. We observe that CNN models with the highest accuracy also have the best AVH scores. This agrees with an earlier finding that state-of-art models tend to improve on classification of harder training examples. We find that AVH displays interesting dynamics during training: it quickly reaches a plateau even though the training loss keeps improving. This suggests the need for designing better loss functions that can target harder examples more effectively. Finally, we empirically show significant improvement in performance by using AVH as a measure of hardness in self-training methods for domain adaptation.
The paper proposes, when given a CNN, an image and its label, a measure called angular visual hardness (AVH). The paper shows that AVH correlates with human selection frequency (HSF) [RRSS19].
Pros:
1. The authors experimented with a representative set of trained models in my opinion. (More on this in Con2)
2. In Section 6, the authors acknowledge a substantive counter-example/argument. (More on this in Con2) 
Cons (I put the ones that weighed the most on my decision first):
1. The presentation is confusing, and at times self-contradictory. For example, in Section 1, the paper asserts that “the two systems differ in what they view as hard examples that appear ambiguous or uncertain“ but then proceeds to claim that AVH being a CNN-derived quantity (more on this in Con2) correlates well with HSF. In fact, [RRSS19] (heavily cited here) seems to suggest exactly the opposite that harder examples for humans are also hard for CNNs. This is not very surprising as high accuracies of these CNNs imply their agreement with human judgment: we are learning a dataset labeled by humans. (More on this in Que2)
2. AVH is a function of a particular CNN (architecture and parameter values) and the target class label _in addition_ to the input image. These dependencies make AVH a measure of the ambiguity of an image very problematic. Granted that the paper presents evidence that AVH correlates with HSF for a number of _trained_ models but they will be of different values. 
3. The work is not well self-contained. HSF, the core quantity studied is not introduced with sufficient details. (See Que1 and Sug1)
Some possible mistakes/typos:
1. Feature vector x and class weight w in general do not lie on S^n. Indeed your definition of A(u, v) only relies on u, v being nonzero.
2. There is a missing { above Definition 1.
3. In Definition 1, “for any x” -> for any (x, y). 
4. In References, [10] is a duplicate of [11].
5. The captions in Figures 5, 6, and 7 in Appendix A might be wrong. They say ||x|| whereas the y-axis in the plots is labeled AVH.
Questions (I listed the important ones first):
1. What is human visual hardness (HVH)? How is HSF related to HVH? Why is being selected from a pool of images (in the procedure described in [RRSS19]) a good measure of HVH?
2. Since the class logit is exactly <x, w_c>, with arccos being a decreasing function, I expect AVH to behave very much like the opposite of model confidence (Definition 2). And this seems to be confirmed in Table 1 (performing a confidence calibration on validation set might increase this further). I wonder how AVH is different from model confidence _qualitatively_ and consequentially what insights do we gain (or should we expect to gain) by studying AVH instead of model confidence?
3. Degradation levels (DL) are mentioned early on but the experiments and figures were not shown in the main text (deferred till Appendix). What is the rationale? 
4. The middle row in Figure 3 has a small range of ~1e-4. Is that expected? Can you provide some simple arguments? The closeness of the initial and final values of AVH in the AlexNet plot also concerns me.
5. How is the visualization in Figure 1 generated? It is not immediately clear to me how the high dimensional space is projected into 2D. My concern is that though suggestive in the figure, the category weights w_c in general do not spread out evenly. Do they? I would suggest reporting the angular separation of the category weights (maybe by showing them in a CxC matrix).
6. In Figure 2, what happens to the dark stripes? Are there no data points with the specified range of HSF values?
Minor issues (factored little to none in my decision):
1. There are 60+ citations but their relevance to the current seems questionable in many cases. Many of them are accompanied by little or no technical comparison when they are mentioned. In particular, in Section 2 on the related work from psychology/neuroscience, little specifics are discussed to contextualize the current work. 
2. Many arguments come across as (highly) speculative and imprecise. As a result, I find the reasoning and logical story diluted and hard to follow.
3. The comparison with feature norm seems poorly motivated. The other quantities, namely AVH and model confidence, both depend on the class label. 
4. The term hardness has a rich history and connotation in the algorithmic analysis literature. I would suggest using a different term, as the hardness of a problem usually reflects some intrinsic aspects of its structure and not dependent on some algorithm.
Suggestions:
1. If DL is not important to the core results, it will help simplify and focus the presentation by leaving them out entirely.
2. Try to be more concise and more precise in the presentation. It might also benefit from more formalism wherever possible, and more procedural details, when human studies or notions is involved. The latter seems to be a lesson from [RRSS19] (in regard to reproducibility).
In summary, I do not recommend accepting the current article. 
(To authors and other reviewers) Please do not hesitate to directly point out my misunderstandings. I am open to acknowledging mistakes and revising my assessment accordingly.