Robustness of neural networks has recently been highlighted by the adversarial examples, i.e., inputs added with well-designed perturbations which are imperceptible to humans but can cause the network to give incorrect outputs. In this paper, we design a new CNN architecture that by itself has good robustness. We introduce a simple but powerful technique, Random Mask, to modify existing CNN structures. We show that CNNs with Random Mask achieve state-of-the-art performance against black-box adversarial attacks without applying any adversarial training. We next investigate the adversarial examples which "fool" a CNN with Random Mask. Surprisingly, we find that these adversarial examples often "fool" humans as well. This raises fundamental questions on how to define adversarial examples and robustness properly.
The authors propose a simple method for increasing the robustness of convolutional neural networks against adversarial examples. This method is simple but seems to achieve surprisingly good results. It consist in randomly remove neurons from the network architecture. The deleted neurons are selected before training and remain deleted during the training and test phase.  The authors also study the adversarial examples that still fool the network after applying their method and find than those examples also fool human. This finding raises the question of what is an adversarial example if both humans and networks are fooled by the same example. 
Using Random Masks in neural network is not a new idea since it was already proposed for DropOut or DropConnect (Regularization of Neural Networks using DropConnect, ICML2013) and in the context of adversarial attacks (Dhillon et al. 2018)  as reported by the authors. The discussion (Section 2) about the impact of random masks on what convolution layers capture in the spatial organisation of the input is interesting: whereas standard CNNs focus on detecting the presence of a feature in the output, random mask could force the CNN layers to learn how a specific feature distributes on the whole input maps. This limitation of the CNN has already been pointed up and solutions have been proposed for example Capsule Networks (Dynamic Routing Between Capsules, NIPS 2017). This intuition is experimentally supported by a simple random shuffle by block of the input image  (Appendix A).
In Section 3, the authors present a large number of experiments to demonstrate the robustness of their method. Most of the details are given in the 13 (!) pages of appendix. Experiments against black-box attack, random noise, white-box attack, grey-box are presented. Most of the experiments are on CIFAR10 but one experiment is also presented on MNIST. One could regret that only one architecture of CNN is tested (ResNet18) except for gray-box attack, for which DenseNet121 and VG19 are tested. One could ask why the type of models tested is not consistent across the different experiments.  For black-box attack, random masks compare favourably to Madry’s defence. For white box defence, Random Mask is not compared to another defence method, which seems a weakness to me but I am not familiar enough with papers in this area to estimate if this is a common practice. In most of the experiments, the drop ratio is between 0.5 and 0.9, which seems to indicate that the size the initial network could be reduced by more than 50% to increase the robustness to attack. This ratio is larger than what is usually used for dropout (0.5 at most).  
In section 3.3, different strategies for random masks are explored : where to apply random masks, random mask versus random channels, random masks versus same masks. Results are given in table 2. The caption of Table 2 could be more explicit : what are the presented percent ?
Experiments on masking shallow versus deep layers are interesting. Best results for robustness are obtained with masking shallow layers at quite a high ratio (0.9). One could ask if this result could be due to the type or the parameters of adversarial attacks which are not adapted to such a high sparseness on shallow layers or to the specific kind of sparseness induced by the masks. A comparison to a regular network with the same number of free parameters as the masked network could give insight on this aspect. 
pros : simple to implement, good robustness shown agains a variety of attack types
cons : mainly tested on a single architecture (ResNet) and on a single datatbase CIFAR. Maybe not robust against the latest techniques of adversarial attack.