Despite the impressive performance of deep neural networks (DNNs) on numerous learning tasks, they still exhibit uncouth behaviours. One puzzling behaviour is the subtle sensitive reaction of DNNs to various noise attacks. Such a nuisance has strengthened the line of research around developing and training noise-robust networks. In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input. We provide an efficient and simple approach to approximate such a regularizer for arbitrarily deep networks. This is done by leveraging the analytic expression of the output mean of a shallow neural network, avoiding the need for memory and computation expensive data augmentation. We conduct extensive experiments on LeNet and AlexNet on various datasets including MNIST, CIFAR10, and CIFAR100 to demonstrate the effectiveness of our proposed regularizer. In particular, we show that networks that are trained with the proposed regularizer benefit from a boost in robustness against Gaussian noise to an equivalent amount of performing 3-21 folds of noisy data augmentation. Moreover, we empirically show on several architectures and datasets that improving robustness against Gaussian noise, by using the new regularizer, can improve the overall robustness against 6 other types of attacks by two orders of magnitude.
This paper proposes a regularization method for achieving robustness to noisy inputs, with relatively less computation compared to standard data augmentation approaches. Specifically, the authors analyze the analytic expression of the loss on the noisy inputs, and using Jensen’s inequality, propose to minimize a surrogate loss over the expectation of noisy inputs. To minimize the loss over the expectation, the authors impose a regularization over the first moment of the network weights. The authors validate the model with the proposed regularization technique for its robustness against Gaussian attack and other types of attacks, whose results show that the model is robust. 
Pros
- The general idea of the regularization that replaces the generation of noisy samples and optimization over it is conceptually appealing and seems practically useful.
- The derivation of the moment-based regularization makes sense. 
- The proposed regularizer seems to be effective to a certain degree, on the sets of experiments done by the authors.
Cons
- Experimental validation seems highly inadequate due to lack of baselines. Thus it is difficult to assess the degree of robustness the proposed model achieves. The authors should perform extensive evaluation against state-of-the-art techniques against multiple types of attacks, in order to demonstrate the effectiveness of the proposed method.
- While the authors emphasize the computational efficiency of the method, the authors do not report computational cost or actual runtime.  
- The types of non-Gaussian attacks should be better described. Which ones use L-infinity attacks and which use L2 attacks?
- Figure 3 doesn’t seem like a very favorable result to the proposed model, since we are generally more concerned with adversarial examples generated with small perturbations, as large perturbations may change the input semantics.
In sum, while I like the overall idea and find the work novel and potentially practical, it is difficult to properly evaluate the work due to lack of comparison against state-of-the-art data augmentation methods for achieving robustness. Therefore I temporarily give this paper a weak reject, but may change the rating with more experimental results provided in the rebuttal.