The ability to detect objects regardless of image corruptions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed PASCAL-C, COCO-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30-60% of the original performance). Furthermore, we provide evidence that increased performance on those benchmarks translates into increased robustness towards real-world "natural" distortions such as real-world fog, rain and snow. Using our benchmark we show that corruption robustness scales with performance on clean data and that a simple data augmentation trick-stylizing the training images-leads to a substantial increase in robustness for both synthetic and natural corruptions on all dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are available at https: //....
Summary:
- key problem or question: assessing / improving the robustness of object detectors to image corruptions (simulated fog, frost, snow, dragonfire...);
- contributions: 1) a benchmark (obtained by adding image-level corruptions to PASCAL, COCO, and Cityscapes) and an experimental protocol to measure detection robustness , 2) extensive experiments quantifying the severe lack of robustness of multiple state-of-the-art models, 3) experiments showing that data augmentation via style transfer (Geirhos et al, ICLR'19) improves robustness at little cost (at most -2% performance degradation on clean COCO images).
Recommendation: Weak Reject
Key reason: unclear novelty w.r.t. Geirhos et al ICLR 2019, especially due to the lack of specificity to object detection (which is the main goal of the paper).
- Although the paper from Geirhos et al. study a specific question (texture bias in CNNs), they propose a similar experimental protocol for assessing robustness, where the only differences are i) they use different corruptions (cf. Fig.6 in their paper, vs. Figs. 5, 7, 8, 9 here), ii) they do not have Cityscapes results (but they have PASCAL and COCO results).
- Furthermore, Geirhos et al 2019 also study the benefits of style-transfer-based data augmentation (this submission uses their technique), they report similar results and conclusions to this submission, including for object detection on PASCAL and COCO (cf. Table 2 in Geirhos et al 2019).
- What is, in the authors' opinion, the main differentiator of this submission? Does the difference in corruptions and evaluated models combined with the addition of Cityscapes yield new insights compared to Geirhos et al 2019?
- Beyond this similarity, what is specific to object detection vs. image classification in this submission? Besides the summary evaluation metrics, the corruptions and stylization are global (image-level) and not object-specific. Is the observed lack of robustness due to localization errors, mis-classifications, or other types of detection mistakes (cf. Hoiem et al ECCV'12)? What are the conclusions about detection robustness that differ from the image classification ones? What would be local corruptions that specifically degrade object detection performance (as studied for instance in the adversarial attack community, including physical attacks like Eykholt et al 2018)?
Additional feedback / questions:
- The results in section 3.4 / Fig.6 seem counterintuitive: more corruption should yield more degradation (as evidenced in Fig. 5 w.r.t. corruption severity). Is RMSE the right metric? Maybe SSIM would be better (more related to perceptual quality)?
- The aforementioned remark also raises the point that corruption difficulty night not be related to its intensity / noise level, but more with the (hard to quantify) domain gap w.r.t. clean data. For instance, if some images contain natural fog, fog corruption robustness should be naturally higher, whereas robust to never seen unrealistic dragonfire is expected to be naturally low. Can this relation between corruption and domain gap be somehow assessed? For instance using perceptual distance (using features from intermediate layers) to nearest clean neighbors? Or maybe by correlating with a subjective measure of realism of the corruption assessed globally per corruption type via a human study?
- Are corruption degradations dataset-specific? Fig. 7,8,9 seem to show different behaviors.
- How does the proposed benchmark compare to the Robust Vision Challenge (http://www.robustvision.net) proposed at CVPR 2018? How does robustness to corruptions and robustness across datasets correlate? Are they both similar "out of domain" robustness measures? How does this "out of domain" issue relate to adversarial examples (besides being "less extreme")?
- Could Fig. 5 include error bars / variance across corruption types?
- rPC is a good metric to compare models but I am not sure it is great to compare robustification methods, because when improving "clean" performance you mechanically decrease rPC, hence why combined is worse than stylized (e.g., in Table 2). What would be a better metric?
- Why does the stylized approach on COCO yields worse mPC (cf. Table 2 and 4), whereas it is expected to improve robustness under corruption (and does on other datasets), esp. since it sacrifices performance on the clean images by a lot?
- What is the impact of the choice of style images on robustness induced by stylization? Is diversity the most important factor (this can be tested by reducing the number of styles available)? If not, what is and how can it be measured?
- Do certain corruption (at certain severity levels) result in unrecoverable objects? For instance, dragonfire might completely occlude certain objects on the side, which might be a problem (e.g., there are frequent parked cars on the side in Cityscapes). What is the upper bound after corruption and how can it be measured?
- What is the human robustness for the new corruptions not present in Geirhos et al 2019? (Also relates to the aforementioned upper bound.)
- The paper is well written and I enjoyed the multiple pop culture references to Game of Thrones.