summaries: The paper studies the sensitivity of a neural network with respect to quantizing its weights and activations. The idea is to use Monte Carlo Arithmetic (MCA) in order to calculate the number of significant bits in the training loss (e.g. cross entropy) that are lost due to floating-point arithmetic. The results show that the number of significant bits lost correlates with the reduction in classification accuracy when quantizing the weights and activations of the neural network. The authors propose a scalable method based on Monte Carlo arithmetic for quantifying the sensitivity of trained neural networks to floating point rounding errors. They demonstrate that the loss of significance metric K estimated from the process can be used for selecting networks that are more robust to quantization, and compare popular architectures (AlexNet, ResNet etc.) for their varying sensitivities.
abstractText: Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks. Quantized floating point representations have received recent interest due to their hardware efficiency benefits and ability to represent a higher dynamic range than fixed point representations, leading to improvements in accuracy. We present a novel technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), for determining the sensitivity of Deep Neural Networks to quantization in floating point arithmetic. We do this by applying Monte Carlo Arithmetic to the inference computation and analyzing the relative standard deviation of the neural network loss. The method makes no assumptions regarding the underlying parameter distributions. We evaluate our method on pre-trained image classification models on the CIFAR-10 and ImageNet datasets. For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate a lower loss of significance from the Monte Carlo trials. Additionally, we can apply MCDA to compare the sensitivity of different network topologies to quantization effects.1
references.authors: Jan Achterhold Jan M. Köhler Anke Schmeink Tim Genewein
references.title: Variational network quantization
references.venue: In ICLR,
references.authors: Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio
references.title: Neural machine translation by jointly learning to align and translate
references.venue: URL http://arxiv.org/abs/1409.0473. cite arxiv:1409.0473Comment: Accepted at ICLR
references.authors: Léonard Blier Yann Ollivier
references.title: The description length of deep learning models
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Charles Blundell Julien Cornebise Koray Kavukcuoglu Daan Wierstra
references.title: Weight uncertainty in neural networks
references.venue: In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37,
references.authors: R.P. Brent
references.title: Algorithms for minimization without derivatives
references.authors: Alexander Buchholz Florian Wenzel Stephan Mandt
references.title: Quasi-Monte Carlo variational inference
references.venue: Proceedings of the 35th International Conference on Machine Learning,
references.authors: Yoojin Choi Mostafa El-Khamy Jungwon Lee
references.title: Towards the limit of network
references.venue: quantization. ArXiv,
references.authors: Ronan Collobert Jason Weston Léon Bottou Michael Karlen Koray Kavukcuoglu Pavel Kuksa
references.title: Natural language processing (almost) from scratch
references.venue: J. Mach. Learn. Res.,
references.authors: Julian Faraone Nicholas Fraser Michaela Blott Philip HW Leong
references.title: Syq: Learning symmetric quantization for efficient deep neural networks
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Michael Frechtling Philip H.W. Leong
references.title: Mcalib: Measuring sensitivity to rounding error with monte carlo programming
references.venue: ACM Trans. Program. Lang. Syst.,
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks
references.venue: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS10). Society for Artificial Intelligence and Statistics,
references.authors: David Goldberg
references.title: What every computer scientist should know about floating-point arithmetic
references.venue: ACM Comput. Surv.,
references.authors: Suyog Gupta Ankur Agrawal Kailash Gopalakrishnan Pritish Narayanan
references.title: Deep learning with limited numerical precision
references.venue: In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Nicholas J. Higham
references.title: Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics, Philadelphia, PA
references.venue: USA, 2nd edition,
references.authors: Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: Mobilenets: Efficient convolutional neural networks for mobile vision
references.venue: applications. ArXiv,
references.authors: Peter J. Huber
references.title: Robust estimation of a location parameter
references.venue: Annals of Mathematical Statistics,
references.authors: Forrest N. Iandola Matthew W. Moskewicz Khalid Ashraf Song Han William J. Dally Kurt Keutzer
references.title: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size
references.venue: CoRR, abs/1602.07360,
references.authors: Benoit Jacob Skirmantas Kligys Bo Chen Menglong Zhu Matthew Tang Andrew Howard Hartwig Adam Dmitry
references.title: Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference
references.venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks
references.venue: Advances in Neural Information Processing Systems
references.authors: Christos Louizos Karen Ullrich Max Welling
references.title: Bayesian compression for deep learning
references.venue: Advances in Neural Information Processing Systems
references.authors: Naveen Mellempudi Sudarshan Srinivasan Dipankar Das Bharat Kaul
references.title: Mixed precision training with 8-bit floating point. CoRR, abs/1905.12334, 2019
references.authors: Paulius Micikevicius Sharan Narang Jonah Alben Gregory Diamos Erich Elsen David Garcia Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh Hao Wu
references.title: Mixed precision training
references.venue: In International Conference on Learning Representations,
references.authors: Douglas Stott Parker David Langley
references.title: Monte carlo arithmetic: exploiting randomness in floatingpoint arithmetic. In Monte Carlo Arithmetic: exploiting randomness in floating-point arithmetic
references.venue: Computer Science Department,
references.authors: Douglas Stott Parker Brad Pierce Paul R. Eggert
references.title: Monte carlo arithmetic: how to gamble with floating point and win
references.venue: Computing in Science and Engineering,
references.authors: Shaoqing Ren Kaiming He Ross Girshick Jian Sun
references.title: Faster r-cnn: Towards real-time object detection with region proposal networks
references.venue: Advances in Neural Information Processing Systems
references.authors: Charbel Sakr Naresh Shanbhag
references.title: An analytical method to determine minimum per-layer precision of deep neural networks
references.venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Charbel Sakr Naresh Shanbhag
references.title: Per-tensor fixed-point quantization of the back-propagation algorithm
references.venue: In International Conference on Learning Representations,
references.authors: Charbel Sakr Yongjune Kim Naresh Shanbhag
references.title: Analytical guarantees on numerical precision of deep neural networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Mark Sandler Andrew G. Howard Menglong Zhu Andrey Zhmoginov Liang-Chieh Chen
references.title: Mobilenetv2: Inverted residuals and linear bottlenecks
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Mingxing Tan Quoc Le
references.title: EfficientNet: Rethinking model scaling for convolutional neural networks
references.venue: In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning,
references.authors: Mingxing Tan Bo Chen Ruoming Pang Vijay Vasudevan Quoc V. Le
references.title: Mnasnet: Platformaware neural architecture search for
references.venue: mobile. ArXiv,
references.authors: Naigang Wang Jungwook Choi Daniel Brand Chia-Yu Chen Kailash Gopalakrishnan
references.title: Training deep neural networks with 8-bit floating point numbers
references.venue: In Proceedings of the 32Nd International Conference on Neural Information Processing Systems,
references.authors: James H. Wilkinson
references.title: Rounding Errors in Algebraic Processes
references.authors: Bichen Wu Alvin Wan Xiangyu Yue Peter Jin Sicheng Zhao Noah Golmant Amir Gholaminejad Joseph Gonzalez Kurt Keutzer
references.title: Shift: A zero flop, zero parameter alternative to spatial convolutions
references.venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018a
references.authors: Shuang Wu Guoqi Li Feng Chen Luping Shi
references.title: Training and inference with integers in deep neural networks
references.venue: In International Conference on Learning Representations,
references.authors: Guandao Yang Tianyi Zhang Polina Kirichenko Junwen Bai Andrew Gordon Wilson Christopher De Sa
references.title: Swalp : Stochastic weight averaging in low precision training
references.authors: Jackson H.C. Yeung Evangeline F.Y. Young Philip H.W. Leong
references.title: A monte-carlo floatingpoint unit for self-validating arithmetic
references.venue: In Proceedings of the 19th ACM/SIGDA International Symposium on Field Programmable Gate Arrays,
references.authors: Shuchang Zhou Zekun Ni Xinyu Zhou He Wen Yuxin Wu Yuheng Zou
references.title: Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth
references.venue: gradients. ArXiv,
sections.heading: 1 INTRODUCTION
sections.text: Deep Neural Networks have achieved state-of-the-art performances in many machine learning tasks such as such as speech recognition (Collobert et al., 2011), machine translation (Bahdanau et al., 2014), object detection (Ren et al., 2015) and image classification (Krizhevsky et al., 2012). However, excellent performance comes at the cost of significantly high computational and memory complexity, typically requiring teraops of computation during inference and Gigabytes of storage. To overcome these complexities, compression methods have been utilized, aiming to exploit the inherent resilience of DNNs to noise. These engender representations which maintain algorithm performance but significantly improve latency, throughput and power consumption of hardware implementations. In particular, exploiting reduced numerical precision for data representations through quantization has been emphatically promising, whereby on customizable hardware, efficiency scales quadratically with each bit of precision.
  Quantization of fixed-point arithmetic (Q-FX) for DNN inference has been extensively studied, and more recently there has been increasing interest in quantized floating point (Q-FP) arithmetic for both DNN inference and training (Wang et al., 2018). Q-FP has the advantage of higher dynamic range compared to equivalent Q-FX representations and reduced hardware cost over single-precision floating point (FP). This has influenced application specific integrated circuits (ASICs) such as Google’s tensor processing unit (TPU), which supports 16-bit floating point (16-FP) and soft processors such as Microsoft’s Project Brainwave which utilizes 8-FP.
  To illustrate these hardware benefits, we synthesized arithmetic logic units (ALUs) in different formats and different precision on an FPGA and present performance estimates in operations per second (OPs) and area estimates in Look-up Tables (LUTs) per operation (LUTs/Op) in Figure 1. As shown, 8-bit fixed point (8-FX) achieves improved performance and area over 8-FP. However, 7-FP is a significant improvement over 8 or 12-FX and 8 or 9-FP. These examples demonstrate substantial performance and area benefits from reducing FP precision by only 1 to 2-bits. Thus, if we
  1Source code will be available if the paper is accepted
  can design networks which not only achieve high accuracy but are robust to quantization, higher performing hardware solutions are possible.
  Since in Q-FP, we are trying to represent the infinite set of real numbers using a finite number of bits, quantization and rounding artefacts will be introduced, with inaccuracies being cascaded along the computation graph (IEEE, 1985; Goldberg, 1991; Higham, 2002). This paper proposes Monte Carlo Deep Neural Network Arithmetic (MCDA), a novel way to apply Monte Carlo Arithmetic (MCA) (Parker et al., 2000) for determining the sensitivity of Deep Neural Networks to Q-FP representations. It allows hardware-software designers to quantify the impact of quantization, enabling more efficient systems to be discovered. We do this by exploiting Monte Carlo simulations under which rounding effects are randomized. This, in turn, allows one to infer the sensitivity of executing a computation graph to quantization effects.
  Our MCDA technique is highly sensitive, allowing very small differences in quantization behaviour to be detected. The technique makes no assumptions regarding data distributions and directly measures the effects of quantization on the problem under study. This allows us to provide insights into the precision requirements of any inference network for any given dataset. Additionally we can use the technique to select weight parameters which are more robust to floating point rounding. The theoretical and practical contributions of this work can be summarized as follows:
  • We introduce a novel and rigorous analysis technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), which measures the sensitivity of Deep Neural Network inference computation to floating point rounding error.
  • When applied to neural network inference, we show MCDA can determine the precision requirements of different networks, rank them, and detect small differences between different neural network topologies and weight sets.
  • We demonstrate that while a network with the same topology but different weights may have the same loss and validation accuracy, their sensitivity to quantization can be vastly different. Using the CIFAR-10 and ImageNet datasets, we introduce a method to choose weights which are more robust to rounding error, resulting in a greatly improved accuracyarea tradeoff over state-of-the-art methods.
  It is worth noting that although we consider convolutional neural networks for image classification, this method could be applied for any neural network model architectures and applications. Moreover, while the experiments in this paper are limited to inference, it may be possible to apply the same idea to analyze training algorithms.
sections.heading: 2 RELATED WORK
sections.text: Low-precision representations of deep learning have been extensively studied. Many training methods have been developed to design representations for fixed point inference (Jacob et al., 2018; Faraone et al., 2018; Zhou et al., 2016) and training (Wu et al., 2018b; Yang et al., 2019; Gupta et al., 2015; Sakr & Shanbhag, 2019). Other methods have also utilized Q-FP arithmetic for inference and training whilst maintaining single-precision accuracy. (Micikevicius et al., 2018) implemented 16- FP arithmetic training whilst storing a 32-FP master copy for the weight updates. Additionally,
  (Wang et al., 2018; Mellempudi et al., 2019) with 8-FP arithmetic whilst using a 16-FP copy of the weights and 16/32-bits for the accumulator. Techniques for determining per-layer sensitivity to quantization have also been studied (Choi et al., 2016; Sakr & Shanbhag, 2018). Further, other studies have successfully determined the minimum fixed point precision requirements for a given DNN accuracy threshold (Sakr et al., 2017). The accuracy and stability of various numerical algorithms in finite precision arithmetic has been studied in (Higham, 2002; Wilkinson, 1994). This has led to techniques for tracking information lost from finite precision arithmetic using random perturbation such as Monte Carlo Arithmetic (Parker et al., 2000; Frechtling & Leong, 2015). Monte Carlo methods have also been used in Bayesian Neural Networks (Buchholz et al., 2018; Blier & Ollivier, 2018). In particular, (Achterhold et al., 2018) introduced a quantizing prior to learn weights which are either close to a quantized representation or have high variance. (Louizos et al., 2017) used hierarchical priors to prune nodes and posterior uncertainties to determine the optimal fixed point precision. (Blundell et al., 2015) use a Monte Carlo approach to learn a probability distribution on the weights of a neural network. To the best of our knowledge, our work is the first to present a technique for directly determining the sensitivity of DNNs to floating point rounding and to explicitly compute precision bounds of a trained network. These ideas can be very usefully applied to extending the limits of low-precision representations in deep learning applications.
sections.heading: 3 BACKGROUND
sections.text: In this section, we describe background theory upon which our technique for determining the sensitivity of DNNs to floating point rounding is based.
sections.heading: 3.1 FLOATING POINT ARITHMETIC
sections.text: The IEEE-754 binary floating point format (IEEE, 1985) represents most real numbers x by a subset in normal form as: x̂ = (−1)sx(1 +mx)2ex (1) where sx ∈ {0, 1} is the sign bit, ex is an integer representing the exponent of x̂ and mx is the mantissa of x̂. Such number formats can be described as a (sx, ex,mx) tuple. In binary form the representation is (bs, be1, b e 2, ..., b e Bex , bm1 , b m 2 ..., b m Bmx
  ) ∈ {0, 1}B , with Bex and Bmx being the number of exponent and mantissa bits, respectively. The infinite set of real numbers R is represented in a computer withB = 1+Bex+Bmx bits, and we define the finite set of real numbers representable in floating point format as exact values, F ⊂ R. Real numbers which aren’t representable are rounded to their nearest exact value. We call this set of numbers inexact values, I, where I∪F = R. The approximation x̂ = F(x) = x(1 + δ), given x ∈ I, introduces rounding error into the computation. The value of δ = ∥∥x−x̂ x
  ∥∥, represents the relative error which is a function of the machine hardware precision, p, as δ ≤ , where = 2−p (IEEE, 1985; Goldberg, 1991; Higham, 2002). In general, inexactness can be caused by finite representations or errors propagating from earlier parts of the computation. Often the primary cause of error in floating point arithmetic is catastrophic cancellation which causes numerical inaccuracy. Catastrophic cancellation occurs when for example, two near equal FP numbers, sharing k significant digits, are subtracted from one another as shown in (2) (Higham, 2002).
  0. f1 f2 ... fk f(k+1) ... ft
  − 0. f1 f2 ... fk g(k+1) ... gt = 0. 0 0 .... 0 h(k+1) ... ht
  (2) 0. f1 f2 ... fk f(k+1) ... ft r(t+1) ... rp
  − 0. f1 f2 ... fk g(k+1) ... gt r̂(t+1) ... r̂p = 0. 0 0 .... 0 h(k+1) ... ht i(t+1) ... ip (3)
  In normalized form, the leading zeros are removed by shifting the result to the left and adjusting the exponent accordingly. The result is 0.hk+1...hti1...ik which has only (t − k) accurate digits and digits i which are unknown. Additionally, the remaining accurate digits h are most likely affected by rounding error in previous computations. This can significantly magnify errors, especially in computing large computational graphs such as that of state-of-the-art DNNs.
  If either operand in (2) is inexact, then the digits h are no more significant than any other sequence of digits. Yet, FP arithmetic has no mechanism of recording this loss of significance. By padding both our operands with random digits r and r̂ in (3), the resulting digits i are randomized. If k
  digits are lost in the result, then k random digits will be in the normalized result and when computed over many random trials, the results will disagree on the trailing k digits. In this case, we are able to detect catastrophic cancellation because the randomization over many trials provides a statistical simulation of round-off errors. We can use techniques from numerical analysis such as Monte Carlo methods to appropriately insert precision-dependant randomization in this way.
sections.heading: 3.2 MONTE CARLO ARITHMETIC
sections.text: Monte Carlo methods can be used to analyze rounding by representing inexact values as random variables (Parker et al., 2000; Frechtling & Leong, 2015). The real value x, as represented in (1), can be modelled to t digits, using:
  inexact(x̂, t, δ) = x̂+ 2ex−tδ = (−1)sx(1 +mx + 2−tδ)2ex (4)
  where δ ∈ U(− 12 , 1 2 ) is a uniformly distributed random variable and t is a positive integer representing the virtual precision of concern. For the same input x̂ in (4), we can run many Monte Carlo trials which will yield different values on each trial, where 0 < t < p so that the MCA can be run accurately on a computer with machine precision p. The ability to vary t is useful because it allows us to then evaluate the hardware precision requirements of a given system or computational graph for a given DNN.
  MCA is a method to model the effect of rounding on a computational graph by randomizing all arithmetic operations. The randomization is applied for both generating inexact operands and also in rounding. In each operation using MCA, ideally both catastrophic cancellation and rounding error can be detected. An operation using MCA is defined as:
  x ◦ y = round(inexact(inexact(x) ◦ inexact(y))) (5)
  where ◦ ∈ ( +,−,×,÷). By applying the inexact function to both operators we make it possible to detect catastrophic cancellation. Furthermore, applying the inexact function to the operation output and then rounding this value implements random rounding and hence is used to detect rounding error (Parker & Langley, 1997). Hence, for the same input into the system, each trial will yield different operands and output.
  After modifying the inexact and rounding operations as described, we use random sampling to simulate Monte Carlo trials. For each trial, we collect data on the resulting output of the system and compute summary statistics to quantify its behaviour (Parker et al., 2000). With sufficiently large number of Monte Carlo trials and virtual precision t, the expected value of the output from these trials will equal the value from using real arithmetic. As explained in the next section, we can determine the total number of digits lost to rounding error and the minimum precision required to avoid a total loss of significance.
sections.heading: 3.3 ANALYSIS
sections.text: The relative error is bounded by δ ≤ 2−p from the design of IEEE FP arithmetic (Wilkinson, 1994; Goldberg, 1991). With this inequality, we can determine the expected number of significant binary digits available from a p-digit FP system as p ≤ −log2(δ). These definitions can be adapted for MCA by replacing the precision of the FP system, p, by the virtual precision, t, of an MCA operation. Thus, the relative error of an MCA operation, for virtual precision t, is δ ≤ 2−t and the expected number of significant binary digits in a t-digit MCA operation is at most t. Using this definition and the proof provided in (Parker & Langley, 1997), the total significant binary digits in a set of M trials is s′ = log2 µσ where µ is the mean and σ the standard deviation. The output of the system should be some scalar value so that we can perform such analysis. For experimentation, M trials are run for all of t ∈ {1, 2, 3, ..., tmax}. The total number of base-2 significant digits lost in a set of M trials is Kt, in (6):
  Kt = t− s′ = t− log2( µ
  σ ) = log2 Θ + t (6)
  where Θ = σµ , for (µ 6= 0), is the relative standard deviation (RSD) of the MCA results. The virtual precision t controls the perturbation strength applied by the inexact function. For a given Kt, as we reduce t, the RSD should increase according to equation 6. At some point, an unexpected loss of
  significance (Frechtling & Leong, 2015) is encountered due to the nonlinear effects of quantization. The value at which this occurs is defined as tmin. The number of significant digits lost for the system being analyzed is then computed by averaging all Kt whereby t > tmin as shown in (7):
  K =
  { 1 tmax−tmin ∑tmax t=tmin Kt where tmax > 1
  Kt where tmax = 1 (7)
  For DNN inference, we propose to use K as a sensitivity measure for the network to FP rounding. The method for implementing this is discussed in the next section.
sections.heading: 4 MONTE CARLO DEEP NEURAL NETWORK ARITHMETIC
sections.text: We now describe MCDA, a methodology for applying MCA techniques to DNN computation, allowing us to understand the sensitivity of a given network and its weight representation to FP rounding.
sections.heading: 4.1 NETWORK MODEL
sections.text: We consider a generalized non-linear L-layer neural network with an output vector yL, input data vector x and learnable weight parameter tensor w = ⋃ wl (l = 1, . . . , L), whereby yL = f(x;w). To compute y, several layers consisting of general matrix multiplication (GEMM) operations (such as convolutional and fully-connected layers) between the layer input xl and weight parameters wl, followed by a non-linear activation function, h, producing intermediate layer outputs xl, i.e. yl = h(xl ⊗wl). The output of a given layer becomes the input to the subsequent layer, i.e. xl+1 = yl, with x1 = x. A loss function is the objective function to minimize updating w via an optimizer such as stochastic gradient descent. For a given set of input data X , the total network loss during inference is calculated by applying a loss function loss(f(x;w), ŷ(x)) where ŷ(x) is the target ground truth output for x. The total loss for X is then a scalar output, such that:
  L(X;w) = 1 |X| ∑ x∈X loss(f(x;w), ŷ(x)) (8)
  Updates of w are usually done in small batches over subsets of X .
  Naively applying MCA to each operation (fine-grained MCA) as described in equation (5) poses significant computational difficulties for DNN models. We observe two primary issues with employing fine-grained MCA to a DNN computational graph:
  • Firstly, the number of required trials for Monte Carlo experiments to generate robust results can typically be in the hundreds or thousands. As DNN inference of state-of-the-art networks typically consist of billions of operations, the computational requirements of applying MCA after each operation will be very large, making the technique impractical. • Using the accuracy as the system output for MCA experiments is problematic because it is
  a discrete value. For high values of t, Monte Carlo results across different t then become indistinguishable and the standard deviation for a given t is potentially zero.
sections.heading: 4.2 MONTE CARLO NETWORK INFERENCE
sections.text: To reduce the computational cost of Monte Carlo experiments, we employ MCDA, which is a coarsegrained approach to MCA for GEMM operations. Conveniently, these can be naturally implemented in modern machine learning frameworks such as PyTorch. Furthermore, to ensure the system output is a continuous value, the loss function output is used, rather than the accuracy. In this case, small perturbations in layer operands are more likely to produce observable changes in the output.
  MCDA applies a vector version of (5) to the DNN inference computational problem in (8). Our operands in this case are vectors and ◦ represents a neural network layer operation. For example, the output from performing a GEMM operation can thus be represented by:
  yl = round(inexact(inexact(xl)⊗ inexact(wl))) (9) Since the inexact function is applied to the inputs and outputs of a GEMM operation, an optimized implementation can be used. This is in contrast to full MCA which requires the application of (5)
  to every individual scalar operation. The vector form in (9) is applied to each edge of neural network computational graph where multiply (division) and/or add (subtract) operations are performed. Hence, it is not applied to operations such as MaxPool and ReLu. As an example, in Figure 2 we show where the inexact function is applied for a residual block with folded batch normalization, which is a repeating sequence of layers found in ResNet models (He et al., 2015). At the final output of the network, the loss is computed with (8) and the inexact function is applied to the outputs y and also the loss output scalar value L. From analyzing the behavior of the loss, we infer the sensitivity of the accuracy of the system to FP rounding. By using MCDA for the GEMM operations, we will not be able to detect all instances of catastrophic cancellation. However, we significantly reduce execution time over fine-grained MCA and show in the next section that we can still retrieve valuable information about our system. In fact, for one trial with one batch of 32 images on ImageNet running on a Nvidia Titan Xp GPU, the speed up of regular inference without MCDA is only 1.05× (for a single Monte Carlo trial). We also note that fine-grained MCA could be applied with a simple modification and would be possible given appropriate customized hardware support for parallel Monte Carlo computations (Yeung et al., 2011).
sections.heading: 5 EXPERIMENTAL RESULTS
sections.text: In this section, we present experimental results for applying MCDA to exemplary convolutional neural networks. We use the CIFAR-10 and ImageNet image classification datasets to compare MobileNet-v2 (Sandler et al., 2018), EfficientNet (Tan & Le, 2019), AlexNet, ResNet (He et al., 2015), SqueezeNet (Iandola et al., 2016) and MnasNet (Tan et al., 2018). For CIFAR-10, we use a batch size of 128, whilst we use a batch size of 32 for ImageNet experiments. Cross-entropy is used as the loss function for both datasets. For a given network, dataset, weight representation and t, we apply MCDA with M = 1000 trials. The resulting loss from each trial is computed with the same single batch of images and hence additional data is not required. We compute Θt from our results for all t ∈ {1, 2, 3, ..., 16}. Following this, we run linear regression analysis using the MCALIB2 (Frechtling & Leong, 2015) R library, on our Θt values, to determine tmin and K. tmin is defined as the point of lowest t where the difference between the regression line and the equivalent Θt is less than half a binary digit, i.e. log10(2
  0.5). Further detail on the calculation of K and tmin from MCALIB can be found in Appendix A.1. We report Q-FP validation accuracy using the quantization function from (Wang et al., 2018) with stochastic rounding3 (See Appendix A.2).
sections.heading: 5.1 DISTINGUISHING WEIGHT PARAMETER REPRESENTATIONS
sections.text: As discussed in Section 3.1, the inexactness in FP arithmetic largely depends on the numerical value of operands. Two instances of the same network and dataset, with the same validation accuracy, but vastly different weight representations, will likely produce differing sensitivities to FP rounding. We first train 8 instances of EfficientNet-b0 and MobileNet-V2 on CIFAR-10 from scratch with random initialization from (Glorot & Bengio, 2010), all achieving within 1% validation accuracy of one another. Using MCDA, we calculate the K values for each model (See Appendix A.3).
  2https://github.com/mfrechtling/mcalib 3https://github.com/Tiiiger/QPyTorch
  We then test their percentage validation accuracy decrease from using post-training quantization (i.e. no finetuning) with varying Q-FP precisions. In Figure 3, we see that the models with higher K values typically experience a larger drop in Q-FP accuracy, indicating they are more sensitive to floating point rounding error. Notably, the model with lowest K for 7-bit MobileNet-v2 experiences a lower percentage validation accuracy drop than three of the 8-bit models. In this case, MCDA model selection enables the saving of a bit of precision while achieving smaller accuracy decrease than some of the trained 8-FP models.
sections.heading: 5.2 COMPARISON TO PREVIOUS WORK
sections.text: One practical use case from the insights gained by MCDA is model selection for quantization. Typically when quantizing a given model trained on a given dataset, the model with highest validation accuracy is chosen and the sensitivity to quantization is assumed to be the same across models. As discussed, for Q-FP representations this is not necessarily the case. We can use K from MCDA to predict which models will be more robust to quantization. To demonstrate this, in Table 1 we compare post-training quantization results for model selection based on K from MCDA, against a baseline model chosen based on the highest single-precision validation accuracy. Evidently, even though the single-precision accuracy is initially as much as 0.9% higher, after quantizing the network to 8-5 bits, the accuracy of the network chosen by smallest K is always significantly higher.
sections.heading: 5.3 NETWORK COMPARISON
sections.text: Modern DNNs consist of convolutional blocks with highly varying computational graphs (Wu et al., 2018a; Howard et al., 2017). Using MCDA we can also compute and compare their sensitivites to floating point rounding error to determine which networks will be robust to Q-FP representations.
  In Figure 4 we show the Θt of pre-trained models, trained on the ImageNet dataset from PyTorch4 5, for differing values of t and run linear regression analysis over our data points. From here we can then assign aK value to each network and compare their loss of significance. At each t, the distance from the regression lines to the ideal line represents the values of Kt, as described in (6). From the MCDA results, AlexNet is the least sensitive to rounding and ResNet-50 is the most, with various models in between these two. Additionally we compare EfficientNet at two different model scales and evidently the larger model has much larger sensitivity. We then also compare the validation accuracy percentage decrease of all models at 10, 9 and 8-FP post-training in Figure 5. At 8-FP, besides MnasNet which experiences a large accuracy drop, K is able to predict validation accuracy degradation. Thus, MCDA provides very valuable information about Q-FP model design.
sections.heading: 6 CONCLUSION
sections.text: We present a novel, highly sensitive, technique to quantify rounding error in DNNs. This is the first method to successfully compare the sensitivity of networks to floating point rounding error. Ultimately, this technique provides a tool for enabling the design of networks which perform higher when quantized. We do this by applying concepts from Monte Carlo Arithmetic theory to DNN computation. Furthermore, we show that by calculating the loss of significance metric K from MCDA, on the CIFAR-10 and ImageNet datasets, we can compare network sensitivities to floating point rounding error and gain valuable insights to potentially design better neural networks. This is an important contribution due to the increasing interest in low-precision floating point arithmetic for efficient DNN hardware systems. The theoretical and practical contributions of this paper will likely translate well to analyzing floating point rounding in backpropagation in future work.
  4https://github.com/pytorch/vision/tree/master/torchvision 5https://github.com/rwightman/pytorch-image-models
sections.heading: A APPENDIX
sections.text: A.1 EXPERIMENTAL SETUP
  In all our experiments, for a given network, dataset, weight representation and t, we run 1000 Monte Carlo trials. The network loss output from each trial is computed with the same single batch of images from the training dataset. We then independently compute the Θt of the network loss for t ∈ {1, 2, 3, ..., tmax} where tmax = 16. Following this, we then run linear regression and calculate K and tmin using MCALIB6. To compute the linear regression, MCALIB uses a log transformed variable, with log(Θ) as the dependant variable and t as the exploratory variable (Frechtling & Leong, 2015).
  log10(Θ) = log10(2 K−t) (10)
  = − log10(2)t+ log10(2)K (11) = mt+ c (12)
  6https://github.com/mfrechtling/mcalib
  Algorithm 1 Summary of Linear Regression Analysis for MCDA Initialize: Pre-train a single-precision DNN model. Set tmax. Set number of trials M . Inputs: Batch of inputs & targets (X, Ŷ ), loss function loss(f(x;w), ˆy(x)), current weights w Outputs: tmin and K
  Monte Carlo Trials: for t = 1 to tmax do
  for trials=1 to M do L(X;w) = ForwardPath (X, Ŷ ,w, t) using (9) end for Compute µ and σ of all trials Compute Θt using (6)
  end for
  Calculate K and tmin: for t = 1 to tmax do
  Compute Pt using (10) - (14) Compute Pt −Θt if Pt −Θt < log10(20.5) then tmin = t break else continue
  end if end for Compute K using (6) and (7)
  where m = − log10(2) = −0.30103 is the slope and c is the intercept such that K = log2(10c). Given these inputs, the intercept c is calculated by minimizing the following objective function using Brents method (Brent, 1973) for single variable optimization:
  f(x) = tmax∑ t=1 γtmax−iρH(ei) (13)
  where ei = Θi − (mti + c) is the residual error, c ∈ [(Θtmax −mtmax)± 2m] is the initial search space for the intercept, γ = 0.75 and ρH(e) is the Huber loss function (Huber, 1964):
  ρH(e) =
  { 1 2e
  2 for |e| ≤ k k |e| − 12k 2 for |e| > k (14)
  where k = 1.345σe and σe is the standard deviation of the residual error set, e. After determining the linear regression model, Pt = mt+ c, we determine whether each Θt is an outlier. If a value for Θt differs to the equivalent Pt by more than half a binary digit, then it is classed an outlier. tmin is then defined as the lowest t where Pt − Θt < log10(20.5). To compute K for a given network, we then average the values for Kt whereby t > tmin. This removes the outliers from the computation of K. We have summarized how the experiments were simulated in Algorithm 1.
  A.2 QUANTIZED FLOATING POINT WITH STOCHASTIC ROUNDING
  To quantize our pre-trained networks to Q-FP representations, we used stochastic rounding as described in (Wang et al., 2018) and implemented in QPytorch7. Two common forms of rounding for FP arithmetic are round-to-nearest and stochastic rounding. The former discards information in the least significant bit (LSB) which is rounded off. This information loss can be significant, especially when quantizing to a small number of bits. Stochastic rounding provides a method to capture this information loss from rounding off the LSB. Given x = (−1)sx(1 + mx)2ex as described in (1).
  7https://github.com/Tiiiger/QPyTorch
  Assume thatmx is in fixed precision with z′ bits which needs to be rounded to z bits, then stochastic rounding is as follows:
  Round(x) = { (−1)sx(1 + bmxc+ )2ex with probability mx−bmxc (−1)sx(1 + bmxc)2ex with probability 1− mx−bmxc
  (15)
  where bmxc is the truncation of z′ − z LSBs of m, = 2−z . For each Q-FP accuracy reported in our experiments, we tested all possible combinations of the number of bits allowed for the exponent and mantissa which satisfied the desired precision. We then chose the combination which produced the highest accuracy.
  A.3 CIFAR-10 REGRESSION ANALYSIS
  In Figure 6, we display the linear regression analysis using MCALIB for the 8 MobileNet-v2 and EfficientNet-b0 models trained on CIFAR-10. As shown, Monte Carlo trials were run for each of these models for t ∈ {1, 2, 3, ..., 16}. The corresponding tmin and K values were then calculated using methods discussed in Appendix A.1. Thus, in Figure 3, it is these K values which are plotted against 8 and 7-bit Q-FP validation accuracy. Also, the models with lowest K values are chosen for MCDA model selection in Table 1.
title: MONTE CARLO DEEP NEURAL NETWORK ARITH-