Recent evidence shows that convolutional neural networks (CNNs) are biased towards textures so that CNNs are non-robust to adversarial perturbations over textures, while traditional robust visual features like SIFT (scale-invariant feature transforms) are designed to be robust across a substantial range of affine distortion, addition of noise, etc with the mimic of human perception nature. This paper aims to leverage good properties of SIFT to renovate CNN architectures towards better accuracy and robustness. We borrow the scale-space extreme value idea from SIFT, and propose EVPNet (extreme value preserving network) which contains three novel components to model the extreme values: (1) parametric differences of Gaussian (DoG) to extract extrema, (2) truncated ReLU to suppress non-stable extrema and (3) projected normalization layer (PNL) to mimic PCA-SIFT like feature normalization. Experiments demonstrate that EVPNets can achieve similar or better accuracy than conventional CNNs, while achieving much better robustness on a set of adversarial attacks (FGSM,PGD,etc) even without adversarial training.
This paper presents a SIFT-feature inspired modification to the standard convolutional neural network (CNN). Specifically the authors propose three innovations: (1) a differences of Gaussians (DoG) convolutional filter; (2) a symmetric ReLU activation function (referred to as a truncated ReLU; and (3) a projected normalization layer. The paper makes the claim that the proposed CNN variant (referred to as the EVPNet) demonstrates superior performance as well as improved robustness to adversarial attacks.
Clarity: Overall, the paper is not particularly well written. There are multiple missing articles and other grammatical errors that make it a bit arduous to read, though I do not believe they have obstructed my ability to understand the contributions. The section describing the projected normalization layer (second half of page 5) is a bit confusing. Figure 2(c) is not helpful in shedding light on the details, though I think a more detailed figure could be quite helpful. Beyond these issues, the paper is relatively clear in the presentation of the material. 
Novelty: Over the last few years there have been many, many proposals for how to vary the basic CNN architecture to improve performance. Some of these have lead to genuine performance gains and have become part of the standard CNN specification. ReLUs, ResNets and Batch Normalization are particularly prominent examples of contributions that have been shown to lead to improvements in performance. Yet the vast majority of these sorts of proposals ultimately make little or no impact on the field. In light of this, I would rate the novelty of the basic goal of this paper as relatively low, though the specific proposal is novel to me and seems reasonable. 
Impact: The impact potential for this paper lies with the performance offered by the proposed innovations. With respect to overall performance improvement the proposed method has not been shown to perform quite at a state-of-the-art level, as given by these resource:
SVHN: https://paperswithcode.com/sota/image-classification-on-svhn
CIFAR-10: https://paperswithcode.com/sota/image-classification-on-cifar-10
The authors compare the performance of their proposed EVPNet against a fair baseline - a squeeze-and-excite ResNet model. These sorts of controlled experiments are useful, but the actual reported performance for both models are somewhat off of the state-of-the-art and it's not clear that the relatively small benefit the authors show over their baselines are maintained for higher performing architectural configurations. Can this architecture be competitive with the state-of-the-art? The current paper in it's current 
Most of the results relate to the claim that the proposed model is robust to adversarial examples. Unfortunately, this is not a particular area of expertise for me, so it's difficult for me to provide a confident assessment of the contribution here, though I will say two things: (i) the method seems to provide a significant increase in adversarial robustness across the
baseline architectures investigated. (ii) the authors demonstrate that the benefit provided by the proposed architecture seems to persist even when training for adversarial defence is introduced. 
I would have liked to see more datasets explored in Experiments section. I  especially would have liked to see results on ImageNet. 
My current rating is weak reject based on the weakness of the writing and the lack of strong empirical evidence in support of the effectiveness of the proposed contributions.