summaries: This paper presents an interesting quantization technique that is, unusually, end-to-end trainable and not just an inference technique. According to the experiments, the method achieves better performance and computational savings as compared to other quantization method baselines. The results are admirably demonstrated on a variety of models, including CNN and RNN-based neural nets, as well as on several datasets in different domains, including ImageNet, CIFAR10, and PTB. We see the method seems to generalize across all of these. This paper outlines a new method that allows using a variety of precision in the numerical representation of the network to increase performance (both in terms of accuracy and speed). They learn a threshold value for which all activation values above the threshold are learned at full precision, while all below are learned at reduced precision. This enables substantial performance gains.  This paper introduces Precision Gating, a novel mechanism to quantize neural network activations to reduce the average bitwidth, resulting in networks with fewer bitwise operations. The idea is to have a learnable threshold Delta that determines if an output activation should be computed in high or low precision, determined by the most significant bits of the value. Assuming that high activations are more important, these are computed at higher precision.
abstractText: We propose precision gating (PG), an end-to-end trainable dynamic dual-precision quantization technique for deep neural networks. PG computes most features in a low precision and only a small proportion of important features in a higher precision to preserve accuracy. The proposed approach is applicable to a variety of DNN architectures and significantly reduces the computational cost of DNN execution with almost no accuracy loss. Our experiments indicate that PG achieves excellent results on CNNs, including statically compressed mobile-friendly networks such as ShuffleNet. Compared to the state-of-the-art prediction-based quantization schemes, PG achieves the same or higher accuracy with 2.4× less compute on ImageNet. PG furthermore applies to RNNs. Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.7× computational cost reduction on LSTM on the Penn Tree Bank dataset.
authors.name: Yichi Zhang
references.authors: Martı́n Abadi Paul Barham Jianmin Chen Zhifeng Chen Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Geoffrey Irving Michael Isard Manjunath Kudlur Josh Levenberg Rajat Monga Sherry Moore Derek G. Murray Benoit Steiner Paul Tucker Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zheng
references.title: TensorFlow: A System for Largescale Machine Learning
references.venue: USENIX Conf. on Operating Systems Design and Implementation,
references.authors: Shijie Cao Lingxiao Ma Wencong Xiao Chen Zhang Yunxin Liu Lintao Zhang Lanshu Nie Zhi Yang
references.title: SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity through Low-Bit Quantization
references.venue: Conf. on Computer Vision and Pattern Recognition (CVPR),
references.authors: Liang-Chieh Chen George Papandreou Iasonas Kokkinos Kevin Murphy Alan L. Yuille
references.title: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
references.venue: IEEE Trans. on Pattern Analysis and Machine Intelligence,
references.authors: Jungwook Choi Zhuo Wang Swagath Venkataramani Pierce I-Jen Chuang Vijayalakshmi Srinivasan Kailash Gopalakrishnan
references.title: PACT: Parameterized Clipping Activation for Quantized Neural Networks
references.venue: arXiv e-print,
references.authors: Matthieu Courbariaux Itay Hubara Daniel Soudry Ran El-Yaniv Yoshua Bengio
references.title: Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
references.venue: arXiv e-print,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: ImageNet: A Large-Scale Hierarchical Image Database
references.venue: Conf. on Computer Vision and Pattern Recognition (CVPR),
references.authors: Zhen Dong Zhewei Yao Amir Gholami Michael W. Mahoney Kurt Keutzer
references.title: HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision
references.venue: Int’l Conf. on Computer Vision (ICCV),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep Residual Learning for Image Recognition
references.venue: Conf. on Computer Vision and Pattern Recognition (CVPR),
references.authors: Qinyao He He Wen Shuchang Zhou Yuxin Wu Cong Yao Xinyu Zhou Yuheng Zou
references.title: Effective Quantization Methods for Recurrent Neural Networks
references.venue: arXiv e-print,
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long Short-Term Memory
references.venue: Neural Computation,
references.authors: Weizhe Hua Yuan Zhou Christopher De Sa Zhiru Zhang G. Edward Suh
references.title: Boosting the Performance of CNN Accelerators with Dynamic Fine-Grained Channel Gating
references.venue: Int’l Symp. on Microarchitecture (MICRO),
references.authors: Weizhe Hua Yuan Zhou Christopher M De Sa Zhiru Zhang G. Edward Suh
references.title: Channel Gating Neural Networks
references.venue: Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv Yoshua Bengio
references.title: Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
references.venue: Journal of Machine Learning Research (JMLR),
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
references.venue: arXiv e-print,
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning Multiple Layers of Features from Tiny Images
references.venue: Tech report,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet Classification with Deep Convolutional Neural Networks
references.venue: Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Siu Kwan Lam Antoine Pitrou Stanley Seibert
references.title: Numba: A LLVM-based Python JIT Compiler
references.venue: Workshop on the LLVM Compiler Infrastructure in HPC,
references.authors: Darryl D. Lin Sachin S. Talathi V. Sreekanth Annapureddy
references.title: Fixed Point Quantization of Deep Convolutional Networks
references.venue: Int’l Conf. on Machine Learning (ICML),
references.authors: Yingyan Lin Charbel Sakr Yongjune Kim Naresh Shanbhag
references.title: PredictiveNet: An energyefficient convolutional neural network via zero prediction
references.venue: Int’l Symp. on Circuits and Systems (ISCAS),
references.authors: Thang Luong Hieu Pham Christopher D. Manning
references.title: Effective Approaches to Attention-based Neural Machine Translation
references.venue: Conf. on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Ningning Ma Xiangyu Zhang Hai-Tao Zheng Jian Sun
references.title: ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
references.venue: European Conf. on Computer Vision (ECCV),
references.authors: Mitchell P. Marcus Mary Ann Marcinkiewicz Beatrice Santorini
references.title: Building a Large Annotated Corpus of English: The Penn Treebank
references.venue: Comput. Linguist.,
references.authors: Asit Mishra Eriko Nurvitadhi Jeffrey J Cook Debbie Marr
references.title: WRPN: Wide Reduced-Precision Networks
references.venue: Int’l Conf. on Learning Representations (ICLR),
references.authors: Israt Nisa Aravind Sukumaran-Rajam Sureyya Emre Kurt Changwan Hong P. Sadayappan
references.title: Sampled Dense Matrix Multiplication for High-Performance Machine Learning
references.venue: Int’l Conf. on High-Performance Computing (HIPC),
references.authors: Eunhyeok Park Sungjoo Yoo Peter Vajda
references.title: Value-aware Quantization for Training and Inference of Neural Networks
references.venue: European Conf. on Computer Vision (ECCV),
references.authors: Olaf Ronneberger Philipp Fischer Thomas Brox
references.title: U-Net: Convolutional Networks for Biomedical Image Segmentation
references.venue: Medical Image Computing and Computer-Assisted Intervention (MICCAI),
references.authors: Florian Schroff Dmitry Kalenichenko James Philbin
references.title: FaceNet: A Unified Embedding for Face Recognition and Clustering
references.venue: Conf. on Computer Vision and Pattern Recognition (CVPR),
references.authors: Mingcong Song Jiechen Zhao Yang Hu Jiaqi Zhang Tao Li
references.title: Prediction Based Execution on
references.venue: Deep Neural Networks. Int’l Symp. on Computer Architecture
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin
references.title: Attention Is All You Need
references.venue: Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Kuan Wang Zhijian Liu Yujun Lin Ji Lin Song Han
references.title: HAQ: Hardware-Aware Automated Quantization With Mixed Precision
references.venue: Conf. on Computer Vision and Pattern Recognition
references.authors: Peiqi Wang Xinfeng Xie Lei Deng Guoqi Li Dongsheng Wang Yuan Xie
references.title: HitNet: Hybrid Ternary Recurrent Neural Network
references.venue: Advances in Neural Information Processing Systems (NeurIPS),
references.authors: Bichen Wu Alvin Wan Xiangyu Yue Peter Jin Sicheng Zhao Noah Golmant Amir Gholaminejad Joseph Gonzalez Kurt Keutzer
references.title: Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
references.venue: Conf. on Computer Vision and Pattern Recognition (CVPR),
references.authors: Bichen Wu Yanghan Wang Peizhao Zhang Yuandong Tian Peter Vajda Kurt Keutzer
references.title: Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
references.venue: arXiv e-print,
references.authors: Xiaowei Xu Yukun Ding Sharon Xiaobo Hu Michael Niemier Jason Cong Yu Hu Yiyu Shi
references.title: Scaling for Edge Inference of Deep Neural Networks
references.venue: Nature Electronics,
references.authors: Hengshuang Zhao Xiaojuan Qi Xiaoyong Shen Jianping Shi Jiaya Jia
references.title: ICNet for RealTime Semantic Segmentation on High-Resolution Images
references.venue: European Conf. on Computer Vision (ECCV),
references.authors: Ritchie Zhao Weinan Song Wentao Zhang Tianwei Xing Jeng-Hau Lin Mani Srivastava Rajesh Gupta Zhiru Zhang
references.title: Accelerating Binarized Convolutional Neural Networks with SoftwareProgrammable FPGAs
references.venue: Int’l Symp. on Field-Programmable Gate Arrays (FPGA),
references.authors: Ritchie Zhao Yuwei Hu Jordan Dotzel Christopher De Sa Zhiru Zhang
references.title: Improving Neural Network Quantization using Outlier Channel Splitting
references.venue: Int’l Conf. on Machine Learning (ICML),
references.authors: Shu-Chang Zhou Yu-Zhi Wang He Wen Qin-Yao He Yu-Heng Zou
references.title: Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
references.venue: Journal of Computer Science and Technology,
references.authors: Shuchang Zhou Yuxin Wu Zekun Ni Xinyu Zhou He Wen Yuheng Zou
references.title: DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
references.venue: arXiv preprint,
references.authors: Chenzhuo Zhu Song Han Huizi Mao William J. Dally
references.title: Trained Ternary Quantization
references.venue: Int’l Conf. on Learning Representations (ICLR),
sections.text: We propose precision gating (PG), an end-to-end trainable dynamic dual-precision quantization technique for deep neural networks. PG computes most features in a low precision and only a small proportion of important features in a higher precision to preserve accuracy. The proposed approach is applicable to a variety of DNN architectures and significantly reduces the computational cost of DNN execution with almost no accuracy loss. Our experiments indicate that PG achieves excellent results on CNNs, including statically compressed mobile-friendly networks such as ShuffleNet. Compared to the state-of-the-art prediction-based quantization schemes, PG achieves the same or higher accuracy with 2.4× less compute on ImageNet. PG furthermore applies to RNNs. Compared to 8-bit uniform quantization, PG obtains a 1.2% improvement in perplexity per word with 2.7× computational cost reduction on LSTM on the Penn Tree Bank dataset.
sections.heading: 1 INTRODUCTION
sections.text: In recent years, deep neural networks (DNNs) have demonstrated excellent performance on many computer vision and language modeling tasks such as image classification, semantic segmentation, face recognition, machine translation, and image captioning (Krizhevsky et al., 2012; He et al., 2016a; Ronneberger et al., 2015; Chen et al., 2016; Zhao et al., 2018; Schroff et al., 2015; Luong et al., 2015; Vaswani et al., 2017). One evident trend in DNN design is that as researchers strive for better accuracy, both the model size and the number of DNN layers have drastically increased over time (Xu et al., 2018). At the same time, there is a growing demand to deploy deep learning technology in edge devices such as mobile phones, VR/AR glasses, and drones (Wu et al., 2019). The limited computational, memory, and energy budgets on these devices impose major challenges for the deployment of large DNN models at the edge.
  DNN quantization is an important technique for improving the hardware efficiency of DNN execution (Zhao et al., 2017). Numerous studies have shown that full-precision floating-point computation is not necessary for DNN inference — quantized fixed-point models produce competitive results with a small or zero loss in prediction accuracy (Lin et al., 2016; He et al., 2016b; Zhou et al., 2016; 2017). In some cases, quantization may even improve model generalization by acting as a form of regularization. Existing studies mainly focus on static quantization, in which the precision of each weight and activation is fixed prior to inference (Hubara et al., 2017; He et al., 2016b). Along this line of work, researchers have explored tuning the bitwidth per layer (Wu et al., 2018b; Wang et al., 2019; Dong et al., 2019) as well as various types of quantization functions (Wang et al., 2018; Courbariaux et al., 2016; Li et al., 2016; Zhou et al., 2016). However, static DNN quantization methods cannot exploit input-dependent characteristics, where certain features can be computed in a lower precision during inference as they contribute less to the classification result for the given input. For
  example, in computer vision tasks, the pixels representing the object of interest are typically more important than the background pixels.
  In this paper, we reduce the inefficiency of a statically quantized DNN via precision gating (PG), which computes most features with low-precision arithmetic operations and only updates few important features to a high precision. More concretely, PG first executes a DNN layer in a low precision and identifies the output features with large magnitude as important features. It then computes a sparse update to increase the precision of those important output features. Intuitively, small values make a small contribution to the DNN’s output; thus approximating them in a low precision is reasonable. Precision gating enables dual-precision DNN execution at the granularity of each individual output feature, and therefore greatly reducing the average bitwidth and computational cost of the DNN. We further introduce a differentiable gating function which makes PG applicable to a rich variety of network models.
  Experimental results show that PG achieves significant compute reduction and accuracy improvement on both CNNs and LSTMs. Compared to the baseline CNN counterparts, PG obtains 3.5% and 0.6% higher classification accuracy with up to 4.5× and 2.4× less computational cost for CIFAR-10 and ImageNet, respectively. On LSTM, compared to 8-bit uniform quantization PG boosts perplexity per word (PPW) by 1.2% with 2.8× less compute on the Penn Tree Bank (PTB) dataset. Our contributions are as follows:
  1. We propose precision gating (PG), which to the best of our knowledge is the first end-toend trainable method that enables dual-precision execution of DNNs. PG is applicable to a wide variety of CNN and LSTM models. 2. PG enables DNN computation with lower average bitwidth than other state-of-the-art quantization methods. By employing a low-cost gating scheme, PG has the potential to reduce DNN execution costs in both commodity and dedicated hardware. 3. PG achieves the same sparsity during back-propagation as forward propagation, which dramatically reduces the computational cost for both passes. This is in stark contrast to prior dynamic DNN optimization methods that focus only on reducing the inference cost.
sections.heading: 2 RELATED WORK
sections.text: Quantizing activations. Prior studies show that weights can be quantized to low bitwidth without compromising much accuracy (Zhu et al., 2017); however, quantizing activations with a low bitwidth (e.g., 4 bits) typically incurs a nontrivial accuracy degradation (Mishra et al., 2018; Zhou et al., 2016). This is partially caused by large activation and weight outliers, which stretch the quantization grid too wide and too sparse under a low precision, thus increasing the error (Park et al., 2018; Zhao et al., 2019). To address this problem, Choi et al. (2018) propose PACT to reduce the dynamic range of the activations through clipping the outliers using a learnable threshold. PACT provides a more effective quantization scheme under a very low bitwidth (e.g., 4 bits). In this work we incorporate the PACT method in the training flow of PG to handle the large outliers.
  Prediction-based execution. Prior works have explored predicting ReLU-induced zeros and maxpooling compute redundancy to lower the computational cost of CNNs. For example, Lin et al. (2017); Song et al. (2018) propose zero-prediction to utilize a few of the most-significant bits in the input activations to predict the sign of the output activation. Zero-prediction removes redundancy by exploiting the fact that negative outputs will be suppressed by the ReLU anyway. Yet, this method only applies to ReLU activations and only when a linear layer is directly followed by ReLU. Hence such method does not apply to RNNs that use sigmoid or tanh as the activation function and many modern CNN networks that use batch normalization (Ioffe & Szegedy, 2015) before ReLU. Hua et al. (2019a;b) propose channel gating to dynamically turn off a subset of channels that contribute little to the model prediction result. Precision gating is orthogonal to this pruning technique as channel gating executes the whole network at the same precision.
  More recently, Cao et al. (2019) propose SeerNet, which also executes a CNN model in dual precision. For each convolutional layer, SeerNet first executes a quantized version of the layer and uses the results to predict the output sparsity induced by the ReLU or the computational sparsity induced by the max-pooling layer. For those activations that are not suppressed according to the prediction, SeerNet computes the original convolution in full-precision (32-bit float). One key dif-
  Input Activations = << + ℎ
  ference between PG and SeerNet is that PG reuses the result of the low-precision compute as a partial product when it performs the high-precision multiplication in the update phase. In contrast, the full-precision compute in SeerNet does not reuse the output from the quantized layer, which incurs a higher execution cost.
  Feature-level precision tuning. There is also prior work that uses a different precision to handle the outlier quantization. Park et al. (2018) propose value-aware quantization where the majority of data are computed at reduced precision while a small number of outliers are handled at high precision. Our approach is significantly different because we allow dual precision for every feature, not only for the outliers.
sections.heading: 3 PRECISION GATING (PG)
sections.text: In this section we first describe the basic mechanism of PG. We then discuss how to design the gating scheme to accelerate both forward and backward passes. Finally, we consider incorporating outlier clipping to reduce the quantization error.
sections.heading: 3.1 BASIC FORMULATION
sections.text: We first define a linear layer in a neural network (either convolutional or fully-connected) as O = I∗W, where O, I, and W are the output, input, and weights, respectively. Suppose I is represented in a B-bit fixed-point format, which is shown in Figure 1. PG partitions I into (1) Ihb, the Bhb most-significant bits (MSBs), and (2) Ilb, the remaining Blb least-significant bits (LSBs). Here B = Bhb +Blb. More formally, we can write:
  I = Ihb << Blb + Ilb (1)
  Here<< denotes the left shift operator. We can then reformulate a singleB-bit linear layer into two lower-precision computations as:
  O = Ohb + Olb = [W ∗ (Ihb << Blb)] + [W ∗ Ilb] (2)
  Ohb is the partial product obtained by using the MSBs of input feature (Ihb) whereas Olb represents the remaining partial product computed with Ilb.
  Precision gating works in two phases. In the prediction phase, PG performs the computation Ohb = W ∗ (Ihb << Blb). Output features of Ohb greater than a learnable gating threshold ∆ are considered important. In the update phase, PG computes Olb = W ∗ Ilb only for the important features and adds it to Ohb. The overall execution flow of PG is illustrated in Figure 2, where unimportant output features only take the upper path while important ones are computed as the sum of both paths. More precisely, this can be summarized as follows:
  O =
  { Ohb Ohb ≤ ∆
  Ohb + Olb Ohb > ∆ (3)
  In essence, PG intends to compute the majority of (unimportant) features with Bhb bits and only a small set of important features with B bits. The importance of each element in the output O is calculated by comparing the magnitude of its partial sum Ohb to ∆. Let Sp be the percentage of
  unimportant activations over all features. PG saves Sp·BlbB fraction of the compute in the original DNN model. PG thus achieves dual-precision execution using a lightweight gating mechanism which only adds a comparison operation. The goal is to minimize the average bitwidth of the multiply-add operations in the DNN execution.
sections.heading: 3.2 EFFICIENT LEARNABLE GATING SCHEME
sections.text: PG automatically learns a gating threshold ∆c,l during training for each output channel in each DNN layer. A larger ∆c,l indicates that more output features are computed in low-precision, resulting in greater computational savings but possibly at the expense of reduced model accuracy. Define ∆ as the vector containing each gating threshold ∆c,l. We formulate the problem of optimizing ∆ as minimizing the original model loss L along with an L2 penalty term:
  min W,∆
  L(I, y;W,∆) + σ ‖∆− δ‖2 (4)
  Here y is the ground truth label, σ is the penalty factor, and δ is the gating target, a target value for the learnable threshold. The penalty factor and gating target are hyperparameters which allow a user to emphasize high computation savings (large σ or δ) or accuracy preservation (small σ or δ).
  Training a model with precision gating can be performed on commodity GPUs using existing deep learning frameworks. We implement PG on GPU as the equation O = Ohb + mask Olb, where mask = 1Ohb>∆ is a binary decision mask and represents element-wise multiplication. During forward propagation, most elements in mask are 0. PG therefore saves hardware execution cost by only computing a sparse Olb in the update phase. If PG is implemented in a dedicated hardware accelerator, MSBs and LSBs will be wired separately; The prediction phase can be controlled by a multiplexer, and only computes LSB convolutions while Ohb exceeds the threshold, thus achieving savings in both compute cycles and energy.
  The mask is computed using a binary decision function (i.e., a step function), which has a gradient of zero almost everywhere. To let gradients flow through mask to ∆, we use a sigmoid on the backward pass to approximate the step function. Specifically, we define mask = sigmoid(α(Ohb− ∆)) only on the backward pass as in the previous work Hua et al. (2019b). Here α changes the slope of the sigmoid, thus controlling the magnitude of the gradients.
sections.heading: 3.3 SPARSE BACK-PROPAGATION
sections.text: A sparse update phase only reduces the computational cost during the inference (forward propagation). We further propose to save the compute during the back-propagation by modifying the forward function of the PG block. Specifically, we square the mask element-wise.
  O = Ohb + mask 2 Olb (5)
  Given that mask is a binary tensor, mask2 in Eq. (5) preserves the same value as mask. Thus the forward pass remains unchanged. During the back-propagation, an additional mask term is introduced in computing the gradient of O with respect to the gating threshold ∆ in Eq. (6) because of the element-wise square. Consequently, the update of ∆ only requires the result of mask Olb which has already been computed during the forward pass.
  ∂O ∂∆ ≈ ∂O ∂mask ∂sigmoid(α(Ohb −∆)) ∂∆ = 2 ·mask Olb ∂sigmoid(α(Ohb −∆)) ∂∆ (6)
  The gradient of O with respect to the weights W in Eq. (7) employs the same sparse mask Olb as the update of ∆. Therefore, precision gating can reduce the computational cost of both forwardprogagtion (inference) and back-propagation by the same factor.
  ∂O ∂W = ∂Ohb ∂W + mask2 ∂Olb ∂W = ∂Ohb ∂W + ∂mask Olb ∂W (7)
sections.heading: 3.4 OUTLIER CLIPPING
sections.text: PG predicts important features using low-precision computation. One difficulty with this is that DNN activations are distributed in a bell curve, with most values close to zero and a few large outliers. The top row of Figure 3(a) shows some activations as blue dots, including a single outlier. If we quantize each value to 4 bits (second column) and use 2 most-significant bits in the prediction phase (third column), we see that almost all values are rounded to zero. In this case, PG can only distinguish the importance between the single outlier and the rest of the values no matter what ∆ is. Thus, the presence of large outliers greatly reduces the effectiveness of PG.
  To address this, we combine PG with PACT (Choi et al., 2018), which clips each layer’s outputs using a learnable clip threshold. The bottom row of Figure 3 shows how clipping limits the dynamic range of activations, making values more uniformly distributed along the quantization grid. Now
  the 2 most-significant bits can effectively separate out different groups of values based on magnitude. We apply PACT to PG in CNNs, which commonly use an unbounded activation function such as ReLU. RNNs, on the other hand, typically employ a bounded activation function (e.g., tanh, sigmoid) that often makes PACT unnecessary.
sections.heading: 4 EXPERIMENTS
sections.text: We evaluate PG using ResNet-20 (He et al., 2016a) and ShiftNet-20 (Wu et al., 2018a) on CIFAR10 (Krizhevsky & Hinton, 2009), and ShuffleNet V2 (Ma et al., 2018) on ImageNet (Deng et al., 2009). ResNet is a very popular CNN architecture for image classification. ShiftNet and ShuffleNet are more compact architectures designed specifically for mobile and edge devices. We set the expansion rate of ShiftNet to be 6 and choose the 0.5× variant of ShffleNet V2 for all experiments. On CIFAR-10, the batch size is 128, and the models are trained for 200 epochs. The initial learning rate is 0.1 and decays at epoch 100, 150, 200 by a factor of 0.1 (i.e., multiply learning rate by 0.1). On ImageNet, the batch size is 512 and the models are trained for 120 epochs. The learning rate decays linearly from an initial value of 0.5 to 0.
  We also test an LSTM model (Hochreiter & Schmidhuber, 1997) on the Penn Tree Bank (PTB) (Marcus et al., 1993) corpus. The model accuracy is measured by perplexity per word (PPW), where a lower PPW is better. Following the configuration used by He et al. (2016b) and Hubara et al. (2017), the number of hidden units in the LSTM cell is set to 300, and the number of layers is set to 1. We follow the same training setting as described in He et al. (2016b), except that the learning rate decays by a factor of 0.1 at epoch 50 and 90. All experiments are conducted on Tensorflow (Abadi et al., 2016) with NVIDIA GeForce 1080Ti GPUs. We report the top-1 accuracy for all experiments.
  We replace the convolutional layers in CNNs and the dense layers in LSTM with the proposed PG block. Moreover, the following hyperparameters in PG need to be tuned appropriately to achieve a low average bitwidth with a high accuracy.
  • The full bitwidth B – this represents the bitwidth for high-precision computation in PG. B is set to 5 or less on CIFAR-10, 5 or 6 on ImageNet, and 3 or 4 on PTB. • The prediction bitwidthBhb – this represents the bitwidth for low-precision computation. • The penalty factor σ – this is the scaling factor of the L2 loss for gating thresholds ∆. • The gating target δ – the target gating threshold. We use a variety of values δ ∈ [−1.0, 5.0]
  in our experiments. • The coefficient α in the backward pass – α controls the magnitude of gradients flowing
  to ∆. We set α to be 5 across all experiments.
sections.heading: 4.1 CNN RESULTS
sections.text: To compare the hardware execution efficiency across different techniques, we compute the average bitwidth (Bavg) of all features in a DNN model:
  Bavg = Bhb + (1− Sp)× (B −Bhb) (8)
  Here Sp denotes sparsity in terms of the percentage of low-precision activations (i.e., number of unimportant features divided by total features). The computational cost of DNNs is proportional to the average bitwidth. Our results on CNNs are presented in Tables 1 and 2.
  We first compare PG against two widely adopted quantization schemes — uniform quantization (UQ) and PACT (Choi et al., 2018). In Table 1, columns 3 and 4 show the average bitwidth and model accuracy of PG, respectively; columns 5-7 list the bitwidth and corresponding model accuracy of UQ and PACT, respectively. At each row, PG achieves a better accuracy with a lower average bitwidth (Bavg vs. Bits). Specifically, PG achieves 6.7% and 1.6% higher accuracy with 1.25× computational cost reduction, and 6.6% and 0.5% higher accuracy with 1.54× computational cost reduction than 2-bit UQ and PACT on ShiftNet-20 and ResNet-20 for CIFAR-10 (rows 3 and 6), respectively. We observe the same trend on ShuffleNet for ImageNet, where PG improves the accuracy of 5-bit UQ and PACT by 1.0% and 1.4% with 1.22× computational cost reduction (row 9). It is also worth noting that PG can recover the accuracy of the floating-point ShiftNet-20, ResNet-20, and ShuffleNet V2 0.5×with 3.9, 3.2, and 4.7 average bitwidth, respectively. This demonstrates that PG, using a learnable threshold, can predict unimportant features and reduce their bitwidth without compromising accuracy. The results compared to quantization baselines are visualized in Figure 4, which plots accuracy vs. average bitwidth — uniform quantization (squares), PACT (triangles), and PG (circles) are shown on separate curves. Results closer to the upper-left corner are better.
  We then compare PG with Fix-Threshold, which is an extension of the zero-prediction (Lin et al., 2017; Song et al., 2018). Lin et al. (2017); Song et al. (2018) explicitly predict ReLU-induced zeros during inference. To have a fair comparison, we extend their technique to predict an arbitrary fixed threshold to achieve the same or lower Bavg as PG, reported as Fix-Threshold in Table 1. For CIFAR-10, we observe that PG achieves 20.2% and 18.7% higher accuracy with 1.75× and 1.38× computational cost reduction than Fix-Threshold on ShiftNet-20 and ResNet-20 (rows 3 and 6), respectively. The gap in accuracy becomes even larger on ShuffleNet V2 for ImageNet dataset. With the same or a lower average bitwidth, the accuracy of PG is at least 26% higher than FixThreshold (rows 7-9). In conclusion, PG consistently outperforms Fix-Threshold because PG uses a learnable gate function which can adjust its threshold to the clipped activation distribution.
  We further compare PG with SeerNet (Cao et al., 2019) in Table 2. For each convolutional layer, SeerNet executes a quantized version of the same layer to predict the output sparsity. It then computes in full precision the output activations that are not suppressed according to the prediction. Since the code of SeerNet is currently unavailable, we implement the network in Tensorflow and boost its accuracy by retraining the network. We reduce the average bitwidth of SeerNet while keeping a comparable accuracy as PG. For CIFAR-10, PG reduces the computational cost of ResNet-20 3.3× more than SeerNet at the same prediction accuracy. Meanwhile for the hardware-friendly ShiftNet-20, PG achieves 3.2% higher accuracy with 4.5× less compute than SeerNet. For ImageNet, PG on ShuffleNet also achieves 0.4% higher accuracy with 2.4× computational cost reduction than SeerNet. It is worth noting that SeerNet does not reuse the outputs from the quantized layer, which may incur a nontrivial overhead in execution time. In contrast, when PG invokes the update phase, it reuses the outputs of the low-precision computation from the prediction phase.
  In order to evaluate the efficacy of the learnable thresholds, we further sweep a series of manually set thresholds on the ResNet-20 for CIFAR-10. Table 3 shows the results where we use a dual-precision mode of B/Bhb = 3/2. As the threshold decreases from 3 to -4, the average bitwidth in the update phase consistently increases. This is expected because we compute more output features in a high precision. The model prediction accuracy therefore increases. Compared to these manually set thresholds, PG achieves a much improved model accuracy (91.2%) with a higher sparsity (90.1%).
  To quantify the impact of sparse back-propagation described in Section 3.3, we run PG with and without sparse back-propagation on CNNs. Table 4 compares the sparsity in the update phase of PG with and without back-propagation under the same model accuracy and average bitwidth. Interestingly, we find that the sparsity in the update phase of PG with sparse back-propagation is consistently higher than that of PG without sparse back-propagation across the tested models and datasets. For both CIFAR-10 and ImageNet, the sparsity increases by between 6% and 21%. We hypothesize that sparse back-propagation zeros out the gradients flowing to non-activated LSB convolutions in the update phase, which leads to a higher sparsity.
sections.heading: 4.2 LSTM RESULTS
sections.text: PG also works well on RNNs. Table 5 reports our results applying PG on an LSTM model for the PTB corpus. Here, we only compare with the uniform quantization since PACT, Fix-Threshold, and SeerNet do not work for sigmoid or tanh activation functions. Although Hubara et al. (2017) claim that quantizing both weights and activations to 4 bits does not lower PPW, we observe a PPW degradation when B decreases from 8 to 4 bits in our implementation. The LSTM with 8-bit activations is therefore considered as the full-accuracy model. We observe the same trend as in the CNN evaluation where PG enables the 3-bit LSTM cell to improve the PPW by 1.2% and reduce the computational cost by 2.7× compared to the 8-bit uniform quantization.
sections.heading: 4.3 SPARSE KERNEL SPEEDUP
sections.text: The sparse update phase of PG can be implemented efficiently with a new kernel called sampled dense-dense matrix multiplication (SDDMM). A convolutional layer with PG is then factorized into a regular low-precision convolution bounded with a low-precision SDDMM. To evaluate the potential speedup of PG, we implement the SDDMM kernel in Python leveraging a high performance JIT compiler Numba (Lam et al., 2015) and test it on the ResNet-20 model for CIFAR-10. Table 6 shows the layer-wise sparsity and the kernel speedup compared to the dense matrix multiplication baseline on Intel Xeon Silver 4114 CPU (2.20GHz). With the high sparsity (from 76% to 99%) in each layer induced by PG, the SDDMM kernel achieves up to 8.3× wall clock speedup over the general dense matrix-matrix multiplication (GEMM) kernel. The significant wall clock speedup shows a good potential of deploying PG on commodity hardware.
  For a GPU implementation, we need to replace the GEMM kernel with the SDDMM kernel to accelerate the update phase. Mainstream deep learning frameworks such as Tensorflow currently do not provide a built-in operator for SDDMM, which is essential for achieving high performance on GPUs. Nevertheless, Nisa et al. (2018) have recently demonstrated that a highly optimized SDDMM kernel with a similar sparsity shown in Table 6 can achieve about 4× speedup over a GEMM ker-
  nel. This shows strong evidence that PG has a potential to obtain high speedup on GPUs as well. Additionally, our approach is a good fit for the specialized accelerator architectures proposed by Lin et al. (2017) and Song et al. (2018). Due to the high sparsity, DNNs with PG are estimated to get at least 3× speedup and 5× energy efficiency on these dedicated hardware accelerators. We leave the deployment of the SDDMM kernel on GPU and dedicated hardware to future work.
sections.heading: 5 CONCLUSIONS
sections.text: We propose precision gating, a dynamic dual-precision quantization method that can effectively reduce the computational cost of DNNs. PG assigns higher precision to important features and lower precision to the remaining features at run-time. The proposed technique is end-to-end trainable, allowing individual models to learn to distinguish important and unimportant features. Experimental results show that PG outperforms state-of-the-art quantization and prediction approaches by a large margin on both CNN and RNN benchmarks on datasets such as Imagenet and Penn Tree Bank. We will release the source code on the author’s website.
sections.heading: ACKNOWLEDGMENTS
sections.text: This work was supported in part by the Semiconductor Research Corporation (SRC) and DARPA. One of the Titan Xp GPUs used for this research was donated by the NVIDIA Corporation. We thank Jordan Dotzel (Cornell) for his helpful discussions during the camera-ready revision.
sections.heading: A APPENDIX
sections.text: A.1 FEATURE VISUALIZATION
  In precision gating, we expect that the model will learn to compute features whose prediction values exceed the threshold using a high-precision while keeping others computed using a low precision. In the image recognition task, we expect that high-precision features are mostly in the region where an object lies. To provide evidence that PG can effectively learn to identity those regions, in Figure 5, we visualize the decision maps extracted from the final convolutional layer that is modified to support PG in the ResNet-20 model. A decision map is a gray scale 2D image that has the same spatial size as the output feature map in the same convolutional layer. The brighter a pixel is in the decision map, the more probably the same spatial location in the output feature map will be computed using a high precision. The first row contains the original input images in CIFAR-10, and the second row shows their corresponding decision maps. We can see clearly that in each decision map, the locations of bright pixels roughly align with the object in the original image.
  A.2 ADDITIONAL RESULTS
  We provide more supplementary results in this section as shown in Table 7.
title: PRECISION GATING: IMPROVING NEURAL NETWORK EFFICIENCY WITH DYNAMIC DUAL-PRECISION ACTI- VATIONS