DEFENSIVE QUANTIZATION: WHEN EFFICIENCY MEETS ROBUSTNESS
Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their fullprecision counterparts, while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.
Summary of paper
This paper presents an approach  for quantising neural networks such that the resulting quantised model is robust to adversarial and random perturbations.
The core idea of the paper is to enforce the Lipschitz constant of each linear layer of the network approximately close to 1. Since the Lipschitz constant of the neural network is bounded by the product of the
Lipschitz constant of its linear layer (assuming Lipschitz 1 activation functions) the Lipschitz constant of the trained neural network is bounded by 1. This results in a model which is robust to adversarial and random noise ad all directions in the model space are non-expansive. Algorithmically, controlling the Lipschitz constant is achieved by using the orthogonal regulariser presented in the paper Cisse et.al which has the same motivation for this work but for standard neural network training but not quantising. The authors presents thorough experimental study showing why standard quantisation schemes are prone to adversarial noise and demonstrate clearly how this approach improves robustness of quantised network and sometimes even improve over the accuracy of original model. 
Review:
The paper is well written with clear motivation and very easy to follow. 
The core idea of using orthogonal regulariser for improving the robustness of neural network models have been presented in Cisse et.al and the authors re-use it for improving the robustness of quantised models. The main contribution of this work is in identifying that the standard quantised models are very vulnerable to adversarial noise which is illustrated through experiments and then empirically showing that the regulariser presented in Cisse et. al improves the robustness of quantised models with rigorous experiments. The paper add value to the research community through thorough experimental study as well as in industry since quantised models are widely used and the presented model is simple and easy to use. 
Some suggestions and ideas:
1. It will be great if the authors could add a simple analytical explanation why the quantised networks are not robust.                      
2. The manifold of Orthogonal matrices does not include all 1 - Lipschitz matrices and also the Orthogonal set is not convex. I think a better strategy for this problem is to regularise the spectral norm to be 1.  Regularising the spectral norm is computationally cheaper than Orthogonal regulariser when combined with SGD using power iterations.  Moreover the regulariser part of the model becomes nice and convex.
3. Another strategy to control the Lipschitz constant of the network is to directly penalise the norm of the Jacobian as explained in Improved Training of Wasserstein GANs (Gulrajani et. al).