Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks. Quantized floating point representations have received recent interest due to their hardware efficiency benefits and ability to represent a higher dynamic range than fixed point representations, leading to improvements in accuracy. We present a novel technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), for determining the sensitivity of Deep Neural Networks to quantization in floating point arithmetic. We do this by applying Monte Carlo Arithmetic to the inference computation and analyzing the relative standard deviation of the neural network loss. The method makes no assumptions regarding the underlying parameter distributions. We evaluate our method on pre-trained image classification models on the CIFAR-10 and ImageNet datasets. For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate a lower loss of significance from the Monte Carlo trials. Additionally, we can apply MCDA to compare the sensitivity of different network topologies to quantization effects.
Summary:
The paper studies the sensitivity of a neural network with respect to quantizing its weights and activations. The idea is to use Monte Carlo Arithmetic (MCA) in order to calculate the number of significant bits in the training loss (e.g. cross entropy) that are lost due to floating-point arithmetic. The results show that the number of significant bits lost correlates with the reduction in classification accuracy when quantizing the weights and activations of the neural network.
Decision:
Overall, this is an interesting paper with interesting results. However, I think there is considerable room for improvement, and that more details are needed in order to assess the significance of the results, as I detail in the rest of my review. For these reasons, I recommend weak reject for now, but I encourage the authors to continue working on improving the paper and to provide more details in the updated version.
Contribution:
The paper considers an important problem, that of quantizing the weights and activations of a neural network in order to reduce computational and memory cost, while maintaining the machine-learning performance as high as possible. 
In my opinion, the main contribution of the paper is the experimental findings, and in particular that the sensitivity of the training loss with respect to the precision of the weights and activations correlates with the accuracy of the network. It seems to me that these results may relate to work on Bayesian neural networks, sharp vs flat minima, and minimum-description length approaches to variational inference. Work on these areas has also shown that sensitivity of the training loss with respect to the precision of the weights (which intuitively happens when the network is at a "sharp" local minimum vs a "flat" one) is related to poor generalization performance, and vice versa. I would encourage the authors to explore the potential relationship of their work with these areas, and possibly discuss them in an updated version of the paper.
Originality:
The paper describes a method for assessing the sensitivity of a neural network with respect to the precision of the weights and activations. The method is a straightforward application of Monte Carlo Arithmetic (MCA) to neural networks. I believe that the application of MCA to neural networks for this particular purpose is novel, and that the results are original. However, the introduction of the paper gives the impression that the proposed method is brand new, and even uses the acronym MCA to refer to the proposed method, which can be confusing to readers. I would suggest to the authors to rewrite the introduction so as to reflect more accurately that the contribution is not a brand-new method, but rather the application of an existing method (MCA) in a novel way.
Writing quality:
The paper is generally easy to read, but there is considerable room for improvement. There are mistakes, and often the writing is sloppy and imprecise. I give some more specific suggestions on what to improve later on.
Technical quality:
The method is well motivated and the experiments seem reasonable. However, there is very little detail on the experiments, which makes it hard to assess their correctness/significance. I would suggest to the authors to rewrite the experimental section with full detail, or put more details in an appendix. In particular:
- Is each Monte Carlo trial done on the same batch of training images or a different one? If different, how are the trials averaged, and does that mean that the standard deviation over trials also includes a contribution due to different batches?
- In sections 5.1 and 5.2, how were the results for different t combined/aggregated? Did you use linear-regression analysis as in section 5.3?
- When you say "accuracy", do you mean accuracy on the training set, validation set, or test set? This is particularly important for assessing the significance of the results, and is something that is currently missing from the description of the experiments.
- How was the quantization of the neural networks performed? It would be good to explain this at least on a high level, in addition to citing Wang et al., (2018).
- In section 5.2, how was the model selection for each method performed exactly? In the baseline method, was the model to be quantized selected based on validation performance before quantization or after quantization?
Specific suggestions for improvement:
The citation format, i.e. (Smith et al., (2019)), is unusual and uses unnecessarily many parentheses. Use \citep for (Smith et al., 2019), and \citet for Smith et al. (2019).
The illustration of fig. 1 is not fully convincing as a motivation for floating-point arithmetic. Even though it makes the case that Float(7, 7) is more efficient than Float(8, 8) and Float (9, 9), the comparison between Float(7, 7) and Fixed(12, 12) is hard to interpret, as we can't conclude whether the efficiency gain is due to reducing the number of bits or to switching from fixed-point to floating-point arithmetic. A more convincing illustration would compare fixed-point with floating-point arithmetic using the same number of bits.
It would be better if fig. 1 were 2D, as 3D doesn't add anything but makes it harder to compare sizes visually.
Please avoid exaggerations, such as "exquisitely sensitive" or "extremely sensitive", when "sensitive" would suffice.
Section 2 is grammatically sloppy:
- arihtmetic --> arithmetic
- Last line of page 2 seems to be missing a verb.
- this has lead --> this has led
The related-work section is too short and in many cases it doesn't explain what previous work has actually done. For example, "rounding of inexact values to their nearest FP approximation has been studied in several publications" is vague: what exactly these publication have done? This lack of detail makes it hard to assess the originality of the current paper, and how it differs from existing work.
Section 3 is often unclear with imprecise mathematical notation:
- "e is the base-2 exponent of x in binary floating point arithmetic": surely, the exponent is represented as an integer?
- (bs, be1, be2, ..., bex, bm1, bm2, ..., bmx) is sloppy, as it indicates that the indices run from 1 to x.
- Bx = sx + ex + mx  is also sloppy; what is meant here is the number of bits to represent sx, ex, mx and not the values themselves.
- F(x) = x(1 + θ), shouldn't θ be δ?
- "which is typically the cause of horrific numerical inaccuracy from numerical analysis literature", the phrase "from numerical analysis literature" doesn't make much sense here.
- In eq. (4), substituting the expression for x from eq. (1) doesn't yield the same result.
- "The number of trials is an important consideration because [...] it can produce adverse effects on results". What is meant by "adverse effects"? Do you mean that with few trials Monte Carlo doesn't give accurate results? Please be more specific.
- "we can determine the expected number of significant binary digits available from a p-digit FP system as p ≥ −log2(δ)". I'm unable to follow this statement, please explain further. Also, from applying logs to δ ≤ 2^{−p} one gets an inequality that doesn't match the one in p ≥ −log2(δ).
- "The relative error of an MCA operation is, for virtual precision t, is δ ≤ 2^-t", "is" is used twice here.
- "the expected number of significant binary digits in a t-digit MCA operations is at least t", operations --> operation. Also, shouldn't it be at most t, otherwise K becomes negative?
-  is discussed in the section --> is discussed in the next section.
Some mistakes in section 4.1:
- y = (x; w) --> y = f(x; w)
- Eq. (8) is sloppy, it uses X for both the set and its size. Use |X| or something similar for the size.
- In the caption of fig. 2, baes --> base
I'm not convinced by the second bullet point in section 4.1, that the averaging over many images used to obtain the accuracy is the reason why MCA doesn't work well. Surely, the training loss (cross entropy) is also an average over many images? To me it would seem more plausible that the main reason MCA works with training loss but not accuracy is because accuracy is discrete, whereas training loss is continuous.
Fig. 3 would be much easier to read if the axes were labelled, and if the axes had the same range (so that different plots can be compared visually).
Fig. 5 would be easier to read if the networks were sorted with respect to K.
In section 5, CIFAR-10 is sometimes written as CIFAR10.
Appendix A is empty, so it should be removed.