We study the robustness of image classifiers to temporal perturbations derived from videos. As part of this study, we construct ImageNet-Vid-Robust and YTBB-Robust, containing a total 57,897 images grouped into 3,139 sets of perceptually similar images. Our datasets were derived from ImageNet-Vid and Youtube-BB respectively and thoroughly re-annotated by human experts for image similarity. We evaluate a diverse array of classifiers pre-trained on ImageNet and show a median classification accuracy drop of 16 and 10 percent on our two datasets. Additionally, we evaluate three detection models and show that natural perturbations induce both classification as well as localization errors, leading to a median drop in detection mAP of 14 points. Our analysis demonstrates that perturbations occurring naturally in videos pose a substantial and realistic challenge to deploying convolutional neural networks in environments that require both reliable and low-latency predictions.
This paper presents new datasets based on ImageNet and Youtube-BB to assert networks performance consistency across time. Compared to previous work, it uses human labeler to further validate the dataset and discard frames that are deemed too different from the reference one. It provides results on image classification and detection using popular network architectures. Based on these results, the paper claims an accuracy drop of 10 to 16%.
The main contribution of this paper is to introduce a new, human annotated dataset for robustness assessment of image classifiers. In itself, it is valuable work, but it is not clear if the contribution is important enough for ICLR. However, I would still be ok with accepting the paper (better datasets are always useful) if it was not for the way the results are reported. I do not specifically have issues with "more stringent robustness metric" but it should not be used to claim incredible results (like an accuracy drop of 10 to 16% instead of 3% for previous work (Real et al. 2017)).
There is one thing for sure: using "accuracy drop" in this context is just misleading. The underlying concept to which "accuracy" refers is _not_ the "maximum error made by the network over the whole set of images". By this definition of accuracy, if the number of images around the reference frame were 100, missing a _single one_ each time (that is 99% of actual accuracy) would result, according to this peculiar redefinition, to a _0%_ accuracy. This is actually highlighted in Appendix G.1: the "accuracy" trend can only go down, since every supplemental frame brings one more chance to fail and obtain a 0% accuracy for this set of perturbed images.
Same thing goes for the detection, where the only frame that matters among all is the one _minimizing_ the AP. Same thing in Table 4, where the "accuracy" of the Original column means one thing (the amount of correctly identified images over the total number of images) while the "accuracy" of the Perturbed column right next to it means something completely different. Same thing in Table 2, which even provides a "delta" between two unrelated metrics.
I cannot see how this can be justified. Sure, there could be some usage for such strict metric, but again, this is _not_ accuracy and cannot be compared to any previous results. Having a more stringent metric is one thing, but in this case it just seems like a justification to get high drop numbers.
Keeping that in mind, these are the actual conclusions we can make from the paper:
1) Human reviewers removed or changed about 20% of the frames
2) This resulted in a relative accuracy improvement of about 4% for the reference frame (Table 4, column Original). The improvement for the perturbed frames are not actually provided.
3) The comparison (and improvements) to previous work due to the dataset cleaning remains unclear.
4) Comparison between different networks and training procedures
Overall, the paper presents impressive numbers but does not actually back them up. I am open to eventually consider acceptance given the value of the datasets, but the paper would then require a significant overhaul to remove all confusing aspects.