Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes. However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input. These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive. We propose to address this issue by discriminative detection of OOD pixels in input data. Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task. Instead, we train a dedicated OOD model which discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world. We perform our experiments on high resolution natural images in a dense prediction setup. We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset. We evaluate our approach on WildDash test, which is currently the only public test dataset that includes out-of-distribution images. The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.
Summary:
This paper addresses the problem of out-of-distribution detection for helping the segmentation process. Therefore, the detection is performed on a pixel basis. The application of the approach is to datasets used for autonomous driving, where semantic segmentation of the view of the road is a typical application. Since in a road view there will be pixels that are projections of objects that are likely not in the set of classes known by the semantic segmentation algorithm, it makes sense to flag them as being out of distribution (OOD), or not known, or to assign to them a low confidence level. The proposed approach is trivial: train a binary classifier that distinguishes image patches from a known set of classes from image patches coming from an unknown (background class). The classifier output applied at every pixel will give the confidence value. While there are different dataset options to represent the known classes, the background class is represented by images from ILSVRC. The results show that for the segmentation application the approach works better than using an adaptation of more elaborate out-of-distribution methods.
Quality and clarity:
The paper is well organized and is described very clearly and provides an ok set of results, despite the simplicity of the approach.
Originality and significance:
Unfortunately, I do not see any relevant technical novelty, and this is a major issue. Perhaps the only significant conclusion about this paper is that before designing a new OOD detector, if representing the set of “unknown” classes with ILVRC is reasonable, then it makes sense to simply train a binary classifier and see how it works.
Besides the novelty, I disagree with the way the paper has been positioned and motivated. It brings into play epistemic and aleatoric uncertainty concepts to justify (the simplicity of) the approach, and it overlooks a large body of machine learning (novelty detection, one-class classification, …). This is also a major issue.
Additional comments:
One of the biggest motivations for this work is that other approaches do not distinguish between epistemic and aleatoric uncertainty and this is why they do not work. This is regarded as a distinctive advantage of the proposed approach. It is claimed that the proposed formulation is insensitive to any aleatoric uncertainty. On the other hand, the paper is written in a way that ignores a large body of literature that goes under the name of “novelty detection”, “anomaly detection”, “one-class classification”, and related names. So, I am wondering how the approaches just mentioned compare with the proposed method, when epistemic and aleatoric uncertainty become part of the discussion. Isn’t every novelty detector insensitive to aleatoric uncertainty as well? Could the Authors clarify what they claim with that statement, while considering a broader view? 
The paper should relate to the literature mentioned above. In particular, I would point the Authors to a couple of recent works that seem to precisely contradict the premises of the proposed approach, which are given at the beginning of section 3:
- Adversarially Learned One-Class Classifier for Novelty Detection, CVPR 2018
- Generative Probabilistic Novelty Detection with Adversarial Autoencoders, arXiv, July 2018.
Again, related to novelty detection, it looks like the proposed approach still requires tuning one or more thresholds. Therefore, it would not be that different from tuning the threshold of a novelty detector, or a one-class classifier. It would have strengthened the paper if the approach was compared also to a novelty detector.
It is not clear if the fully convolutional OOD detector is working on a patch or on the entire image. If it is a patch, of what size?
Page 4, define the “ID” acronym.