We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized ondemand to different bit-widths as energy and memory requirements of the application change. Unlike quantization-aware training using the straight-through estimator that only targets a specific bit-width and requires access to training data and pipeline, our regularization-based method paves the way for "on the fly" posttraining quantization to various bit-widths. We show that by modeling quantization as a ∞ -bounded perturbation, the first-order term in the loss expansion can be regularized using the 1 -norm of gradients. We experimentally validate our method on different architectures on CIFAR-10 and ImageNet datasets and show that the regularization of a neural network using our method improves robustness against quantization noise.
This paper shows that if we add L1 regularization on gradient in the training phase, the obtained model can achieve better post-training quantization performance since it is more robust to Linf perturbation on the weights. I like the intuition of the paper but there are several weaknesses: 
1. The main concern is that the proposed method cannot outperform quantization-aware fine-tuning. This probably limits the application of the method --- it will only be used when there's not enough time budget for quantization-aware fine tuning for each specific choice of #bits. It will be good if the authors can discuss in what practical scenario their algorithm can be applied. 
2. The method is only tested under uniform symmetric quantization. I believe to demonstrate that the L1 regularized models are indeed easier to be quantized, we need to test it on several different kinds of quantizations. 
3. I have concerns about the hyper-parameter selection for lambda. The authors mentioned that lambda is chosen by grid-search, but what's the grid search criteria? In other words, are the hyper-parameters trying to minimize the validation error of the "unquantized model", or they are minimizing the validation error of the "post-quantized model"? 
4. Some minor suggestions: 
- The current paper uses boldfaced n as perturbation which is quite confusing (since small n is the dimension). I would suggest to replace it by something else, e.g, \Delta. 
- Section 2.3 seems redundant. It's clearly that L1 regularization is better given it's the dual norm of Linf, so clearly it's better than L2 norm. You have proved L2 is not good anyway in experiments. 
===========
After seeing the rebuttal, my concerns about the parameters have been well addressed. Also, I agree with the authors that there are use cases for post quantization, and personally I think post quantization is much easier to do in practice than quantization-aware training. However, this is quite subjective so the fact that the proposed method doesn't outperform quantization-aware training is still a weakness of the paper. 
I would like to slightly raise the score to borderline/weak-accept. I hope the authors can have some experiments on non-uniform quantization if the paper is being accepted; I really think that will demonstrate the strength of the method. People will likely to use this method if it can consistently improve many different kinds of post quantization.