Deep learning models are often sensitive to adversarial attacks, where carefully-designed input samples can cause the system to produce incorrect decisions. Here we focus on the problem of detecting attacks, rather than robust classification, since detecting that an attack occurs may be even more important than avoiding misclassification. We build on advances in explainability, where activity-map-like explanations are used to justify and validate decisions, by highlighting features that are involved with a classification decision. The key observation is that it is hard to create explanations for incorrect decisions. We propose EXAID, a novel attack-detection approach, which uses model explainability to identify images whose explanations are inconsistent with the predicted class. Specifically, we use SHAP, which uses Shapley values in the space of the input image, to identify which input features contribute to a class decision. Interestingly, this approach does not require to modify the attacked model, and it can be applied without modelling a specific attack. It can therefore be applied successfully to detect unfamiliar attacks, that were unknown at the time the detection model was designed. We evaluate EXAID on two benchmark datasets CIFAR-10 and SVHN, and against three leading attack techniques, FGSM, PGD and C&W. We find that EXAID improves over the SoTA detection methods by a large margin across a wide range of noise levels, improving detection from ∼ 70% to over 90% for small perturbations.
A Simple method to detect adversarial examples, but needs more work.
#Summary:
The paper proposed a method that utilizes the model’s explainability to detect adversarial images whose explanations that are not consistent with the predicted class.  The explainability is generated by SHAP, which uses Shapley values to identify relative contributions of each input to a class decision. It designs two detection methods: EXAID familiar, which is aimed to detect the known attacks and EXAID unknown, which is against unknown attacks. Both of the two methods are evaluated on perturbed test data which are generated by FGSM, PGD and CW attack with perturbations of different magnitudes. Qualitative results also show that the proposed method can effectively detect adversaries, especially when the perturbation is relatively small.
#Strength
The method is easy to implement and using the idea of interpretation for detecting adversarial examples seems interesting.
Good results are demonstrated compared with other comparators.
#Weakness
The idea of this paper is based on the interpretation method of DNN. However, it has been shown that these interpretation methods are not reliable and easy to be manipulated [1][2]. Therefore, although the method is simple to design, it also brings other security concerns.
Unfortunately, the paper does not address these issues. In addition, the comparators listed in the experiments are not state-of-art or common baselines. It is either not clear why authors modified the existing method and develop their own “unsupervised” version. 
In the experiments, many details are omitted. For example, how is the “noise level” defined? Are they based on L1, L2 or L-inf perturbation? For PGD attack, how many iterations does the generation run and what is the step size? How many effective adversarial examples are generated for training and testing? And all the experiments are conducted in a relatively small dataset, it is also suggested to do experiments on large datasets, e.g. Imagenet.
In the evaluation part, it looks strange to me why the EXAID familiar performs worse than EXAID unknown in evaluating FGSM attack on SVHN since the EXAID familiar is trained using FGSM attack.
#Presentation
I think the authors used a wrong template to generate the article. The font looks strange and the headnote indicates it is prepared for ICLR2020. The paper contains many typos and even the title contains a misspelling. Poor coverage of citations. There are more works for detecting adversarial examples that are published, e.g. [3][4][5]. On the other hand, the paper does not have the literature review for work related to the model interpretation.
Overall, I think the paper is not good enough for publication at ICLR.
[1] Dombrowski, Ann-Kathrin, et al. "Explanations can be manipulated and geometry is to blame." arXiv preprint arXiv:1906.07983 (2019).
[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. "Interpretation of neural networks is fragile." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
[3] Meng, Dongyu, and Hao Chen. "Magnet: a two-pronged defense against adversarial examples." In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135-147. ACM, 2017.
[4] Liao, Fangzhou, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. "Defense against adversarial attacks using high-level representation guided denoiser." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787. 2018.
[5] Ma, Shiqing, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. "NIC: Detecting Adversarial Samples with Neural Network Invariant Checking." In NDSS. 2019.