A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
Detecting test samples drawn sufficiently far away from the training distribution statistically or adversarially is a fundamental requirement for deploying a good classifier in many real-world machine learning applications. However, deep neural networks with the softmax classifier are known to produce highly overconfident posterior distributions even for such abnormal samples. In this paper, we propose a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier. We obtain the class conditional Gaussian distributions with respect to (low-and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance. While most prior methods have been evaluated for detecting either out-of-distribution or adversarial samples, but not both, the proposed method achieves the state-of-the-art performances for both cases in our experiments. Moreover, we found that our proposed method is more robust in harsh cases, e.g., when the training dataset has noisy labels or small number of samples. Finally, we show that the proposed method enjoys broader usage by applying it to class-incremental learning: whenever out-of-distribution samples are detected, our classification rule can incorporate new classes well without further training deep models.
This paper proposes a new method for detecting out-of-distribution samples and adversarial examples in classification tasks when using deep neural networks. The core idea is simple. Replace the softmax classification layer by  Gaussian Discriminative Analysis. The assumption is that  given a layer of the network (e.g., the penultimate layer of the net) the conditional distribution of the features given a class, follow a Gaussian distribution. The method then consists in estimating the mean of each distribution and  the covariance assuming that the covariance matrix is the same for every class (similar to LDA as noted in Appendix).  Then, by using the fact that they have now a Generative model (p(x/y)), the can produce a "confidence score" for a new sample. This is score is simply the Mahalanobis distance to the closest distribution (the log of the probability for the closest class).  They show different applications of this framework to detect: out-of-distribution samples (in an experimental setup similar to previous work [22]), state-of-the-art adversarial attacks, and finally they introduce a way to deal with incremental learning.  The major strength of the work is its simplicity. The method is quite simple. The method is backed by empirical results. All experiments show superior performance in comparison to previous work.  The major issues are: -- No analysis. It is not clear whether the conditionally Gaussian distributed assumption is reasonable or not. Moreover, an augmented confidential score computed from the output at different levels of a Deepnet is proposed. But, this will be reasonable only if the output at each layer is Gaussian.  Intuition says that this is not the case unless we are very deep in the net. No analysis is given but only an empirical comparison of the results by taking features at different levels (Figure 2). More analysis is needed.  -- There is no justification why some choices are made. For example, why is the covariance matrix  the same for every class?   -- Novelty is limited. The paper builds a lot on top of [22]. There is no additional analysis. For instance, why does the input preprocessing improve the detection of out-of-distribution samples?  -- Several explanations are unclear. In particular Section 2.3  (incremental learning) is not clearly explained/analyzed. It seems that the idea is to use the features of a deep neural net computed at the last layer as a way to reduce the dimension, and then learn a Gaussian mixture where one can control the number of classes in an online fashion. Is this the case? Also, more analysis to support some conclusions (l173-l174 "if we have enough base classes and corresponding samples ...") is needed.   Is is possible to decide that we are in facing a new class? How could this be done? Algorithm 2, assumes that a set of samples from a new class (labeled) are given, so the method just need to updated the Gaussian mixture.  -- Details for reproducibility are lacking. For instance, l192- "The size of features maps (...) reduced by average pooling for computational efficiency" Which is the final dimension? Also, is this only a matter of computational efficiency? or it is also due the fact that one needs to learn a covariance matrix? More analysis is needed.  -- Experimental setup. The whole evaluation should be done assuming that you don't know how it looks the OOD set (or the adversarial attack). I mean, if you know how it looks, then it's just a typical toy problem of two class classification. I understand this is the way previous work address this problem, but It's time that this is changed. Regarding that point, the paper *does* this analysis ("comparison of robustness")  and the results are pretty interesting (performance goes a little down but results are still very good). But, the experimental section needs to better present and discuss these results. This has to be (better) highlighted in the experimental section).   -- Experimental setup regarding Incremental Learning (Section 3.3) is unclear (many unclear sentences). For instance, "we keep 5 exemplars for each class …. roughly matches with the size of the covariance matrix"). In fact,  the whole experiment is unclear to me. Figure 4 is not commented (there are some crossing in the base class accuracy experiment).   Minor comments  -- How are datasets handled. Are all images the same size (cropped/resized)? --l2 "the fundamental" → "a fundamental" -- l134-l142. Input preprocessing. "small controlled noise". What is added is not noise. The sample is perturbed in a way that the confidence score is increased. This is clearly not random, so please do not refer to this to noise. --"out proposed method" → our method or the proposed method -- metrics are not defined (TNR,TPR,AUPR,AUROC,detection accuracy) (in particular how is computed the detection accuracy) -- l213 - are further → is further -- l230-l234. This is unclear. -- l248 - regression detectors → regression detector -- l251 (and others) - "the LID" → LID or "the LID method" -- l252 -  the proposed method is tuned on → the proposed method tuned on  -- l254 - When we train → when training -- l260 - all tested case → all tested cases -- l261 - Algorithm 2 → Algorithm~2. -- l264 - and Euclidean classifier → and an Euclidean classifier -- l273 - ImageNet → imageNet classes -- l282-l285 - Conclusions should summarize the method and the findings.   **AFTER REBUTTAL** I thank the authors for their response. The authors did a good job clarifying many of the raised points in the reviews. The strength of the paper is that the proposed method is simple and does a good job in detecting out-of-distribution samples (SOTA). Nevertheless, I still think the paper lacks more analysis, *better and clearer justifications*, ablation study, better experimental setup (more controlled experiments).