Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in QNNs, but also reduces the variance in gradients. We evaluate BQNs on MNIST, Fashion-MNIST, KMNIST and CIFAR10 image classification datasets. compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood).
This paper proposed Bayesian Quantized Neural Networks (BQN), which is a rough Bayesian treatment for quantized neural networks (QNN). The advantages the BQN over QNN is that it provides better uncertainty estimates as well as accuracy. The disadvantage is that BQN loses the memory efficient property of QNN, that is, BQN in general still needs to store real-value parameters, which parameterize the binary weights. The authors compare BQN to EQNN and show that BQN outperforms the latter.
In general, the paper is well organized and fairly readable, though the first 2 sections could have been more concise. I appreciate the careful description of notations. 
The topic of Bayesian formulation and inference in the context of QNN is interesting. I like the analysis provided for handling small/medium/large fan-in for Bayesian QNN.
One of my concerns is that some description in the provided theorems (as well as the proof) is not very rigorous and sometimes misleading. For example, Theorem 3.2 claims that there is an analytic form for Bayesian inference in softmax layers. One would expect that it is either an exact integration over the distribution of h, or a valid bound. Unfortunately, it is neither. Looking at the proof, it seems it is a rather coarse approximation. Note that in the Appendix, inequality in Equation (42) and Equation (47) takes different direction, meaning that this is not a valid bound for the error term L. The authors may want to adjust the description to make it more rigorous and avoid confusion.
The paper describes BQN in very general terms. It is unclear what distributions are used to parameterize the weights in the experiments. Do you use Bernoulli distributions to parameterize the binary weights {-1, 1}, and learn the real-valued parameters for the distributions during training?
Theorem 3.1 is straightforward. The rough approximation provided in Theorem 3.2 is interesting. It would be interesting to compare this with Gaussian natural-parameter networks, which bypass the softmax problem using multi-logistic regression for classification, where the convolution of Gaussian and sigmoid functions has a closed form.
In terms of related work, references such as natural-parameter networks (NPN) and its variants (one of which specifically handles the softmax case) are missing [1, 2]. Note that, similarly to BQN, NPN also provides a general probabilistic (Bayesian) framework for sampling-free learning/prediction. Theorem 3.3 is also covered in Section 3.2 of [1].
This brings me to another of my concerns. The only baseline provided is E-QNN from three years ago. There are various state-of-the-art baselines, either on QNN or BNN. It would certainly make the paper much stronger if such baselines are included, especially when the authors decide to take up the full ten-page space in this submission.
Minor:
P2: bf Bayesian -> Bayesian
P2: theta that predicted -> theta that predict
P6: Lemma 4.2 -> Theorem 4.2
P6: by with -> with
Assuming most of my concerns can be addressed during the feedback period, I tend to give a ‘weak accept’ to this submission.
Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016
Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018