Machine learning systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a different distribution from the one used for training. With their growing use in critical applications, it becomes important to develop systems that are able to accurately quantify its predictive uncertainty and screen out these anomalous inputs. However, unlike standard learning tasks, there is currently no well established guiding principle for designing architectures that can accurately quantify uncertainty. Moreover, commonly used OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. To address these problems, we first seek to identify guiding principles for designing uncertainty-aware architectures, by proposing Neural Architecture Distribution Search (NADS). Unlike standard neural architecture search methods which seek for a single best performing architecture, NADS searches for a distribution of architectures that perform well on a given task, allowing us to identify building blocks common among all uncertainty aware architectures. With this formulation, we are able to optimize a stochastic outlier detection objective and construct an ensemble of models to perform OoD detection. We perform multiple OoD detection experiments and observe that our NADS performs favorably compared to state-of-the-art OoD detection methods.
This paper needs crucial, quick fixes.
This paper throws neural architecture search at the problem of out-of-distribution detection. Rather than searching over multi-class classifier architectures, and using the result for OOD detection, they instead search for generative model architectures.
The approach appears to work on some of their cherry-picked OOD datasets.
I give this paper a 3 because they are quite possibly only showing their strongest results and not giving a complete picture, and there are numerous small errors throughout the paper.
After a more thorough evaluation, the technique will likely not look strong on datasets such as CIFAR-10 vs CIFAR-100.
This is acceptable since they are comparing density estimators vs multi-class OOD detectors, the latter of which has been vastly superior for many years.
Their technique brings density estimators within striking distance of multi-class classifier perofrmance, but the paper must give a complete picture. If these issues are fixed, the paper is easily a 6 or an 8, depending on the results of currently unshown OOD datasets. It is acceptable if their technique gets and 55% AUROC for CIFAR-10 vs CIFAR-100 while multi-class classifiers with Outlier Exposure get 96%. It is OK because this technique appears superior to other generative models, which have lagged far behind. Currently the paper leaves the impression that this is not only leapfrogging past previous density estimators, but that it also is beating multi-class classifiers. This likely isn't true. We need to see performance on other OOD datasets.
I am willing to increase my score from a 6 to an 8 even if the results are negative.
The CIFAR-10 model should have to detect OOD samples from CIFAR-100, Rademacher/Bernoulli noise, and Gaussian noise. In addition, they should train a model on CIFAR-100 since many OOD detection techniques exhibit much worse behavior on CIFAR-100, compared to CIFAR-10 or MNIST or SVHN.
In summary, I give this a 3 due to critical flaws, but if these are rectified, the paper will likely deserve a 6 or 8.
Miscellaneous Points:
Please include CIFAR-10/100 code. Currently the code is for MNIST.
There are several errors in discussing related work.
> Moreover, previous work on deep uncertainty quantification shows that a single model may not suffice to quantify uncertainty and detect OoD samples (Lakshminarayanan et al., 2017; Choi & Jang, 2018)
Ensembles do not perform appreciably better at OOD detection under systematic OOD benchmarking when using multi-class classifiers. For generative models, ensembles can help. Ensembles mainly help multi-class classifier _calibration_ on in-distribution in-class data, but they have miniscule to nonexistent utility in classifier OOD detection. Please add appropriate qualifiers.
> "With no access to OoD data, unsupervised/self-supervised generative models which maximize the likelihood of in-distribution data become the primary tools for uncertainty quantification."
I think they mean "with access to labels." Multi-class classifiers are the most performant tool for OOD detection, and unsupervised generative models are around chance-levels. _Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty_ (NeurIPS) uses self-supervised learning for OOD detection and achieves good performance, but this work is not cited.
> "However, these models counter-intuitively assign high likelihoods to OoD data (Nalisnick et al., 2019a; Choi & Jang, 2018)"
This was shown in previous work, such as Shafaei et al. 2019 and Hendrycks et al. 2019, and hence deserve mention.
> "Moreover, existing methods to calibrate model uncertainty estimates assume access to OoD data during training (Lee et al., 2018; Hendrycks et al., 2019). This is flawed when anomalous data is rare or not known ahead of time"
Lee et al. does not assume access to OOD data during training; they synthesize their own. Hendrycks et al. does assume access to OOD data, but not OOD data seen during evaluation; in this way, they do not assume data is "known ahead of time," which they reiterate throughout their paper. This sentence is painting with too broad a brush.
There is a smaller experimental problem. They compare to ODIN, but ODIN assumes access to OOD data from the test distribution. While this assumption is clearly questionable, their evaluation does not use the assumption they did, so their technique is not actually ODIN. I suggest just comparing against the Maximum Softmax Probability Baseline in Table 1 since the rest of this paper assumes OOD examples are not known ahead of time.
Update: I have changed my score to an 8.