Recent advances in adversarial Deep Learning (DL) have opened up a new and largely unexplored surface for malicious attacks jeopardizing the integrity of autonomous DL systems. We introduce a novel automated countermeasure called Parallel Checkpointing Learners (PCL) to thwart the potential adversarial attacks and significantly improve the reliability (safety) of a victim DL model. The proposed PCL methodology is unsupervised, meaning that no adversarial sample is leveraged to build/train parallel checkpointing learners. We formalize the goal of preventing adversarial attacks as an optimization problem to minimize the rarely observed regions in the latent feature space spanned by a DL network. To solve the aforementioned minimization problem, a set of complementary but disjoint checkpointing modules are trained and leveraged to validate the victim model execution in parallel. Each checkpointing learner explicitly characterizes the geometry of the input data and the corresponding high-level data abstractions within a particular DL layer. As such, the adversary is required to simultaneously deceive all the defender modules in order to succeed. We extensively evaluate the performance of the PCL methodology against the state-of-the-art attack scenarios, including Fast-Gradient-Sign (FGS), Jacobian Saliency Map Attack (JSMA), Deepfool, and Carlini&WagnerL2. Extensive proof-of-concept evaluations for analyzing various data collections including MNIST, CIFAR10, and ImageNet corroborate the effectiveness of our proposed defense mechanism against adversarial samples.
Summary:
 The paper presents an unsupervised method for detecting adversarial examples of neural networks. The method includes two independent components: an ‘input defender’ which tried to inspect the input, and a ‘latent defender’ trying to inspect a hidden representation. Both are based on the claim that adversarial examples lie outside a certain sub-space occupied by the natural image examples, and modeling this sub-space hence enables their detection. The input defender is based on sparse coding, and the latent defender on modeling the latent activity as a mixture of Gaussians. Experiments are presented on MInst, Cifar10, and ImageNet.
 
-	Introduction: The motivation for detecting adversarial examples is not stated clearly enough. How can such examples be used by a malicious agent to cause damage to a system? Sketching some such scenarios would help the reader understand why the issue is practically important. I was not convinced it is. 
Page 4: 
-	Step 3 of the algorithm is not clear:
o	How exactly does HDDA model the data (formally) and how does it estimate the parameters? In the current version, the paper does not explain the HDDA formalism and learning algorithm, which is a main building block in the proposed system (as it provides the density score used for adversarial examples detection). Hence the paper cannot be read as a standalone document. I went on to read the relevant HDDA paper, but it is also not clear which of the model variants presented there is used in this paper.
o	What is the relation between the model learned at stage 2 (the centers c^i) and the model learnt by HDDA? Are they completely different models? Or are the C^I used when learning the HDDA model (and how)? 
If these are separate models, how are they used in conjunction to give a final density score? If I understand correctly, only the HDDA model is used to get the final score, and the C^i are only used to make the \phy(x) representation more class-seperable. Is that right?
-	Figure 4, b and c: it is not clear what the (x,y,z) measurements plotted in these 3D drawings are (what are the axis).
Page 5:
-	Section 2: the risk analysis is done in a standard Bayesian way and leads to a ratio of PDFs in equation 5. However, this form is not appropriate for the case presented at this paper, since the method presented only models one of these PDFs (Specifically p(x | W1)  - there is not generative model of p(x|W2)).  
-	The authors claim in the last sentence of the section that p(x|W2) is equivalent to 1-p(x|W1), but this is not true: these are two continuous densities, they do not sum to 1, and a model of p(x|W2) is not available (as far as I understand the method)
Page 6:
-	How is equation 7) optimized?
-	Which patchs are extracted from images, for training and at inference time? Are these patchs a dense coverage of the image? Sparsely sampled? Densely sampled with overlaps?
-	Its not clear enough what exactly is the ‘PSNR’ value which is used for the adversarial example detection, and what exactly is ‘profile the PSNR of legitimate samples within each class’. A formal definition of PSNR and’profiling’ is missing (does profiling simply mean finding a threshold for filtering?)
Page 7:
-	Figure 7 is not very informative. Given the ROC curves in figure 8  and table 1 it is redundant. 
Page 8:
-	The results in general indicate that the method is much better than chance, but it is not clear if it is practical, because the false alarm rates for high detection are quite high. For example on ImageNet, 14.2% of the innocent images are mistakenly rejected as malicious to get 90% detection rate. I do not think this working point is useful for a real application
-	Given the high flares alarm rate, it is surprising that experiments with multiple checkpoints are not presented (specifically as this case of multiple checkpoints is discussed explicitly in previous sections of the paper).  Experiments with multiple checkpoints are clear required to complete the picture regarding the empirical performance of this method
-	The experiments show that essentially, the latent defenders are stronger than the input defender in most cases. However, an ablation study of the latent defender is missing: Specifcially, it is not clear a) how much does stage 2 (model refinement with clusters)  contribute to the accuracy (how does the model do without it? And 3) how important is the HDDA and the specific variant used (which is not clear) important: is it important to model the Gaussians using a sub-space? Of which dimension?
Overall:
Pros:
-	 A nice idea with some novelty,  based on a non-trivial observation
-	The experimental results how the idea holds some promise
Cons
-	The method is not presented clearly enough: the main component modeling the network activity is not explained (the HDDA module used)
-	The results presented show that the method is probably not suitable for a practical application yet (high false alarm rate for good detection rate)
-	Experimental results are partial: results are not presented for multiple defenders, no ablation experiments
After revision:
Some of my comments were addressed, and some were not.
Specifically, results were presented for multiple defenders and some ablation experiments were highlihgted
Things not addressed:
 - The risk analysis is still not relevant. The authors removed a clearly flawed sentence, but the analysis still assumes that two densities (of 'good' and 'bad' examples) are modeled, while in the work presented only one of them is. Hence this analysis does not add anything to the paper-  it states a general case which does not fit the current scenario and its relation to the work is not clear. It would have been better to omit it and use the space to describe HDDA and the specific variant used in this work, as this is the main tool doing the distinction.
I believe the paper should be accepted.