This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs. Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs. More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples. This work introduces "Subset Scanning" methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks. Subset Scanning allows us to answer the question: "Which subset of inputs have largerthan-expected activations at which subset of nodes?" Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images. Such images are "weird together". Leveraging this common anomalous pattern we show increased detection power as the proportion of noised images increases in a test set. Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.
The paper is the first paper, in my knowledge, that introduces the problem of identifying anomalous (or corrupted) subset of data input to a neural network. The corrupted inputs are identified vis-a-vis a set of “clean” background set (e.g., the training/ validation data set). Experimental evaluation is performed on the problem of identifying the subset of noisy CIFAR-10 images created by adding targeted adversarial perturbations.
To achieve this, the problem is posed as that of subset detection and subset scanning approaches are adapted for the same. The activation of a node under consideration is converted to a p-value (how extreme the activation is, for that input data, w.r.t. the background set). 
A goodness of fit is then defined for any subset using the Berk-Jones test statistic which is used to create an NPSS scoring function to be maximized over the set of all subsets of the product set – (set of nodes x set of input images). The authors suggest that this combinatorial problem can be addressed efficiently as NPSS has been shown before to have the linear-time subset scanning (LTSS) property (Neill 2012). 
However, since the search is now over a product subspace and the LTSS property holds for individual subspaces (selection of activation nodes or input images while the other is held fixed), the LTSS property doesn’t trivially extend to the product space. The authors suggest an algorithm which alternatively iterates between the two LTSS steps. Since the above algorithm is not guaranteed to obtain the global optimum, multiple random restarts are suggested. This is the weakest part of the paper – neither is a formal proof of convergence provided, nor is the efficiency of the propped algorithm demonstrated empirically. Also, the performance of the algorithm (accuracy) can only be weakly used to judge the goodness of the local optimum obtained as it confounds the power of NPSS with the gap between the local and global optimum.
Contributions
In summary, the main contribution of the paper is the introduction of the problem of the identification of anomalous subsets of adversarially corrupted examples on deep neural networks and the first approach to address the problem via a NPSS scoring function that (partially) satisfies the LTSS property suggesting the existence of efficient (and exact?) algorithms for solving the problem. The work has incremental novelty as it seems to be largely based on prior art (not on DNNs). Initial results look promising.  
Negatives 
However, I have the following concerns:
- (Clarity) Mathematical formulation for the NPSS score function and the optimization problem to be solved should be clearly stated (and not inside paragraphs of text).
- The proposed algorithm is ad hoc - an obvious extension of the standard algorithm that exploits the LTSS property to the product of two set spaces. The algorithm may not find a good optimum. No proof, analysis or study of its properties – optimality gap, convergence, and, efficiency is presented. 
- At deployment, the attacks may be non-targeted or the target may be different for different corrupted inputs. This will dilute the signature as the set of nodes where the ‘extreme values’ appear may not be the same across the corrupted examples. It is not clear if it is the case if the target is the same. In this case, subset scanning may not work. This assumption should’ve been directly tested.
- ‘Berk-Jones can be interpreted as the log-likelihood ratio for testing whether p-values are uniformly distributed on [0,1] vs. … alternate distribution’. Why should this be the test? P-values on clean, background data can be directly obtained and whatever that distribution can be empirically tested or uniformity can be tested against after a whitening transformation?
- Comparison with reasonable ‘baselines’ – for example, with Feinman et al. 2017? Comparison with current state of the art defenses on CIFAR10?
- Why is the BIM attack (Kurakin et al. 2016b) used? There are more powerful attacks available now?
- The NPSS scoring function is supposed to maximize over significance values \alpha. What range was used and how was it decided? In the discussion on Detection Power reported in Table 1, it is suggested that different \alpha_max will trade off precision vs recall. This has not been experimented with.