The ability to detect when an input sample was not drawn from the training distribution is an important desirable property of deep neural networks. In this paper, we show that a simple ensembling of first and second order deep feature statistics can be exploited to effectively differentiate in-distribution and out-of-distribution samples. Specifically, we observe that the mean and standard deviation within feature maps differs greatly between in-distribution and out-of-distribution samples. Based on this observation, we propose a simple and efficient plug-and-play detection procedure that does not require re-training, pre-processing or changes to the model. The proposed method outperforms the state-of-the-art by a large margin in all standard benchmarking tasks, while being much simpler to implement and execute. Notably, our method improves the true negative rate from 39.6% to 95.3% when 95% of in-distribution (CIFAR-100) are correctly detected using a DenseNet and the out-of-distribution dataset is TinyImageNet resize. The source code of our method will be made publicly available.
Summary: A relatively simple approach for detecting out-of-distribution samples by having a parallel logistic regression model using simple statistics (mean and variance) over output of each batch normalisation layer, in order to discriminate between in-distribution and out-of-distribution samples. Results are appealing but presentation is lacking clarity at time and some doubts on the correctness of the experiments remain.
With the goal of detecting out-of-distribution sets, the authors propose to use logistic regression over simple statistics (mean and variance) of each batch normalization layer of CNN in order to discriminate between in-distribution (ID) and out-of-distribution (OOD) samples. They argue that ID and OOD samples can be discriminated with these statistics.
Quality: The motivations of the paper are clear, it aims at having better capacity to detect OOD samples with a method that involves less computations. However, the quality of the experiments is not good enough and I have doubts on their validity.
Originality: Ok. The proposal is relatively simple and is based on the intuition that statistics for the batch normalization is useful to detect OOD samples. The problem is not new, the approach is relatively ad hoc, but it works.
Significance of the work: The results reported are unreasonably good. Although the authors claim the improvement of detection of OOD is significant, the results achieved by detecting **all** the out-distribution samples sounds weird and irrational. How rejecting all Tiny-ImageNet is possible while there are several overlap between the classes presented in TinyImageNet vs. Cifar10/Cifar100 (cf. Table 8)? To me, it looks like the model either overfitted something else than the content of the images, maybe the background noise or similar regarding the nature of the data. More experiments with different in-distribution datasets should be made to be convincing. All experiments reported are using either Cifar10 or Cifar100 as in-distribution datasets.
The author also claim using few samples from a single OOD set is enough for training the regressor that provides OOD-ness score. Is it true for any OOD set or only a carefully chosen OOD set can demonstrate this behavior? What is the criteria for selecting a good OOD set for training the regressor?
Despite of the fact that the proposed method is heavily dependent on the threshold, the authors barely discuss of it. I am assuming that threshold is on OOD-ness score, is that correct? How does look like the OOD-ness score for an ID set over different OOD sets? Providing the OOD-ness score for ID and OOD could reflect how the proposed method is sensitive to a selected threshold. In other words, is selecting a fix threshold will to the TPR / FPR across different OODs.
The overall writing style is perfectible. I did not found the paper super clear in the presentation and it is difficult to really get all useful information for it. However, the authors appear knowledgeable of the literature and the overall structure is clear.
An example of lack of clarity in the explanations: in Table 7, I have difficulty to make sense of the 100% achieved for “Ours (pair)” vs “Ours”. Is the “Ours (pair)” the rate obtained with the exact pair used for adjusting the threshold, while “Ours” is on another dataset? If not, what this mean? Moreover, reporting columns all with 100% is not a good practice, it seems to be a stunt to impress the reader, while not carrying much in term of content and understanding.
In Table 8, I do not understand what the values in parenthesis means. 
Another element: why for training the regressor, the IN and OOD samples are not selected from their corresponding training sets instead of splitting their test sets to a validation and test sets?