Deep Neural Networks (DNNs) generalize well despite their massive size and capability of memorizing all examples. There is a hypothesis that DNNs start learning from simple patterns based on the observations that are consistently well-classified at early epochs (i.e., easy examples) and examples misclassified (i.e., hard examples). However, despite the importance of understanding the learning dynamics of DNNs, properties of easy and hard examples are not fully investigated. In this paper, we study the similarities of easy and hard examples respectively among different CNNs, assessing those examples contributions to generalization. Our results show that most easy examples are identical among different CNNs, as they share similar dataset-dependent patterns (e.g., colors, structures, and superficial cues in high-frequency). Moreover, while hard examples tend to contribute more to generalization than easy examples, removing a large number of easy examples leads to poor generalization, and we find that most misclassified examples in validation dataset are hard examples. By analyzing intriguing properties of easy and hard examples, we discover that the reason why easy and hard examples have such properties can be explained by biases in a dataset and Stochastic Gradient Descent (SGD).
The paper formulates a definition of easy and hard examples and studies the properties and the training implications of such examples. The paper does not attempt to present insights that change training for the better (although suggests this could be future work), so the primary value it claims to add is our understanding of neural networks. I think the paper presents findings that most deep learning practitioners already find intuitive, which is why I think the paper falls short in its primary mission. An exposé like this could be valuable for an introductory text in deep learning, but I do not think the analysis meets the bar for a cutting edge insight and do not think it should be accepted.
Strengths:
- Quantifying easy and hard and using that as a starting point for further analysis is not a bad starting point at all.
- The experiments form a good starting point for interesting analysis.
- The paper is easy to follow and understand.
Weaknesses:
- The biggest weakness is that I just didn't find any of conclusions from this study to be that surprising or interesting. I think it's pretty obvious that neural networks start by learning the most immediately discriminative features ("frequent patterns"). The visual examples of easy and hard examples are not surprising at all (I think you get similar clusters if you just show bottom and top of model confidences, which I think many of us have). In section 7.1, the result that most misclassified examples are hard examples is presented as a surprising result. This is confusing, because this exactly what I would have expected given how you define easy/hard. It would be far more surprising if misclassified examples were all considered easy under your definition.
- The paper only scratches the surface. In Figure 2, the results for the two different datasets are quite different. This means only two datasets is probably not enough for us to understand what is going on here in general. Just the conclusion that datasets may be different in terms of easy/hard samples does not take the analysis far enough. It's also unclear what the reader should make of these conclusions.
- In Table 1, let's say the bottom 30% of samples are actually equally easy. This would mean that the "easy" examples are just a random 1/3 of those samples. Basically, I'm worried about the implications of having a hard cut-off at 10% and if there are situations where the bottom 10% actually changed quite a bit, but the broader picture of easy really didn't change that much. I guess I'm saying that I didn't quite gain confidence that definitions of easy/hard and matching rate are the correct way to go here and there might be a better metric that can look at the continuum of easy/hard from e = 0 to 1. You could have some kind of distance function where if an example moved from 5th percentile to 12th percentile, it would constitute a distance of 7. This is perhaps not the right thing either, but presenting an alternative metric and showing that the numbers (up to scale) and conclusions are unchanged would be nice.
Other comments:
- The new terminology of "contradicted pattern" and "non-contradicted pattern" is a bit confusing. Why aren't you just calling these "non-discriminative" and "discriminative"? If a mantis and a ladybird are both typically on a leaf, the leaf is not discriminative for this task. However, if a mantis and a boat are typically on differently colored backgrounds, the background is discriminative.
Minor comments:
- page 1, "easy and hard examples differ on various CNNs architectures" -> "CNN"
- page 2, "as a criteria" -> "criterion"
- page 2, "We then redefine easy and hard" -> don't you mean just "define"? Or do you mean that the words already have casual meanings, so this is a redefinition? I still think "define" is less confusing here.
- page 2, I think it's confusing that both easy and hard use the threshold \tau, suggesting it is the same. Maybe put a subscript to make it clear that the two \taus are different.
- page 6, "accuracy does not drop" -> could use a "does not *even* drop" for clarity
- page 7, "7.1 Do misclassified examples in validation dataset are hard examples": "Do"->"Are", remove "are"