In this work we revisit gradient regularization for adversarial robustness with some new ingredients. First, we derive new per-image theoretical robustness bounds based on local gradient information. These bounds strongly motivate input gradient regularization. Second, we implement a scaleable version of input gradient regularization which avoids double backpropagation: adversarially robust ImageNet models are trained in 33 hours on four consumer grade GPUs. Finally, we show experimentally and through theoretical certification that input gradient regularization is competitive with adversarial training. Moreover we demonstrate that gradient regularization does not lead to gradient obfuscation or gradient masking.
This paper revisits the method of gradient regularization, which regularizes the loss function by adding the norm of the input gradient, aiming to improve adversarial robustness. The standard gradient regularization is implemented with "double backpropagation", which could be time consuming for large networks. This paper proposes to replace the exact gradient by its discretization via finite difference, which is computationally more efficient comparing to "double backpropagation".  Experiments are conducted to show the effectiveness of the method in reducing training time. 
Overall, the paper is well written but the contribution is very limited and I find some of the experimental comparisons unfair.  Thus I do not support publication of the paper. Here are my detailed arguments.
1) The gradient regularization is not novel
The main contribution of the paper is to use the finite difference in the gradient regularization in order to improve the training time.  The method of gradient regularization is not new, hence the contribution is only the computational effectiveness. This would be fine if one could improve the state-of-the-art method's training time by a lot, but according to the experiments, the performance of adversarial robustness is far from the adversarial training, for example in CIFAR10 epsilon=8/255, there is a 12% drop, which is a huge gap. 
2) Unfair comparison in the experiments
In table 2, the performance of robustness with respect to the l2 norm is presented. However, in the baseline method of adversarial training, the used attack is in l_{infty} norm. This is unfair since l2 norm and l_{infty} norm has very different characteristics. Moreover, the gradient regularization uses l2 norm instead of l_{1} as in table 1, which makes the comparison unfair. 
Furthermore, the finite difference improves the training time versus the standard gradient regularization, but it does not imply that the performance will be the same. In particular, it would be better to include the performance of standard gradient regularization in table 1 and 2 as well. 
3) Comment on the motivation/theory
The theory part is fairly straightforward and it clearly shows that the norm of gradient (w.r.t input) itself is not sufficient to guarantee robustness. As a evidence, even under standard training (for example on MNIST), the gradient norm could be very small, in order of 10^{-4} but still have adversarial example with very small perturbation.  Thus, what we also need is to control how fast this gradient changes (Lipschitz constant or w-bound). However, the gradient norm regularization does not take into account how gradient changes. It is claimed that the finite discretization implicitly reduces how the gradient changes, however, I am not convinced by the argument since as long as we take h to be small, it is still a very local measure. An interesting question would be how h affect the performance and it is not discussed in the paper.