High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including 1) channel-wise scale factors of weights, especially for depthwise convolution, 2) Winograd convolution, and 3) topology-wise 8-bit support. We experiment the techniques on top of a widely-used deep learning framework. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution. We show that the inference throughput and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6% 1 to no loss in accuracy from FP32 baseline. We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks. All the code and models will be publicly available soon.
This paper designs a system to automatically quantize the CNN pretrained models. This system contains three main components: 1) different scale factors for channel-wise network; 2) Winograd 8bit quantization; 3) topology wise 8bit operation support. All these three techniques are standard ways to perform model quantization. The work is solid in the sense that 1) as far as I know there is no work actually using all of these quantization schemes, and designs a system to automatically do quantization with additional algorithm support(retrain strategy). 2)  Significantly amount of experiment on quantizing 18 existing widely used CNN models for different applications, e.g., image classification, image segmentation, etc. 3) it reports the actually inference speed up comparing INT 32 to INT 8, although most of the speed up is less than 2.0.
Several questions:
1) What is the difference between retrain and calibration? As mentioned in the paper, the system does not require retrain, but it seems to me that calibration step, e.g., sampling to find the maximum values, etc, is a form of retrain. I think if that is the case, maybe some quantized models with retrain is worthy comparing with.
2) Many quantization techniques are used in the system, are there any conclusions for what techniques are most important for a particular CNN network/application?
3) Are there any new/novel quantization algorithms in the system? 
4) The inference speed up is mostly less than 2 times, with some are achieving 2.1 speed up while some are without any speedup. Any reason for that? Also what is the overhead of the inference time?