Deep neural networks have led to a series of breakthroughs, dramatically improving the state-of-the-art in many domains. The techniques driving these advances, however, lack a formal method to account for model uncertainty. While the Bayesian approach to learning provides a solid theoretical framework to handle uncertainty, inference in Bayesian-inspired deep neural networks is difficult. In this paper, we provide a practical approach to Bayesian learning that relies on a regularization technique found in nearly every modern network, batch normalization. We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models, and we demonstrate how this finding allows us to make useful estimates of the model uncertainty. Using our approach, it is possible to make meaningful uncertainty estimates using conventional architectures without modifying the network or the training procedure. Our approach is thoroughly validated in a series of empirical experiments on different tasks and using various measures, showing it to outperform baselines on a majority of datasets with strong statistical significance.
The authors show how the regularization procedure called batch normalization,
currently being used by most deep learning systems, can be understood as
performing approximate Bayesian inference. The authors compare this approach to
Monte Carlo dropout (another regularization technique which can also be
considered to perform approximate Bayesian inference). The experiments
performed show that the Bayesian view of batch normalization performs similarly
as MC dropout in terms of the estimates of uncertainty that it produces.
Quality:
I found the quality to be low in some aspects. First, the description of what
is the prior used by batch normalization in section 3.3 is unsatisfactory. The
authors basically refer to Appendix 6.4 for the case in which the weight decay
penalty is not zero. The details in that Appendix are almost none, they just
say "it is thus possible to derive the prior...".
The results in Table 2 are a bit confusing. The authors should highlight in
bold face the results of the best performing method.
The authors indicate that they do not need to compare to variational methods
because Gal and Ghahramani 2015 compare already to those methods. However, Gal
and Ghahramani's code used Bayesian optimization methods to tune
hyper-parameters and this code contains a bug that optimizes hyper-parameters
by maximizing performance on the test data. In particular for hyperparameter
selection, they average performance across (subsets of) 5 of the training sets
from the 20x train/test split, and then using the tau which got the best
average performance for all of 20x train/test splits to evaluate performance:
https://github.com/yaringal/DropoutUncertaintyExps/blob/master/bostonHousing/net/experiment_BO.py#L54
Therefore, the claim that 
"Since we have established that MCBN performs on par with MCDO, by proxy we
might conclude that MCBN outperforms those VI methods as well."
is not valid.
At the beginning of section 4.3 the authors indicate that they follow in their
experiments the setup of Gal and Ghahramani (2015). However, Gal and Ghahramani
(2015) actually follow Hernández-Lobato and Adams, 2015 so the correct
reference should be the latter one.
Clarity:
The paper is clearly written and easy to follow and understand.
I found confusing how to use the proposed method to obtain estimates of
uncertainty for a particular test data point x_star. The paragraph just above
section 4 says that the authors sample a batch of training data for this, but
assume that the test point x_star has to be included in this batch.
How is this actually done in practice?
Originality:
The proposed contribution is original. This is the first time that a Bayesian
interpretation has been given to the batch normalization regularization
proposal.
Significance:
The paper's contributions are significant. Batch normalization is a very
popular regularization technique and showing that it can be used to obtain
estimates of uncertainty is relevant and significant. Many existing deep
learning systems can use this to produce estimates of uncertainty in their
predictions.