In this work the authors show that Bayesian Neural Networks (BNNs) can be efficiently applied to calibrate state-of-the-art Deep Neural Networks (DNN). Our approach acts offline, i.e., it is decoupled from the training of the DNN to be calibrated. This offline approach allow us to apply our BNN calibration to any model regardless of the limitations that the model may present during training. Note that this offline setting is also appropriate in order to deal with privacy concerns of DNN training data or implementation, among others. We show that our approach clearly outperforms other simple maximum likelihood based solutions that have recently shown very good performance, as temperature scaling (Guo et al., 2017). As an example, we reduce the Expected Calibration Error (ECE%) from 0.52 to 0.24 on CIFAR-10 and from 4.28 to 2.46 on CIFAR-100 on two Wide ResNet with 96.13% and 80.39% accuracy respectively, which are among the best results published for these tasks. Moreover, we show that our approach improves the performance of online methods directly applied to the DNN, e.g. Gaussian processes or Bayesian Convolutional Neural Networks. Finally, this decoupled approach allows us to apply any further improvement to the BNN without considering the computational restrictions imposed by the deep model. In this sense, this offline setting is a practical application where BNNs can be considered, which is one of the main criticisms to these techniques. In terms of reproducibility, we provide all the implementation details in https://github.com/2019submission/bnn.2019.
-- Paper summary --
The primary goal of this paper is to investigate the suitability of BNNs for carrying out post-calibration on trained deep learning models. The results are compared to equivalent models calibrated using temperature scaling, and the proposed technique is shown to yield superior uncertainty calibration.
-- General Commentary --
The overall goal of this work is rather modest and the scope of the evaluation is limited. While not without challenges, carrying out offline calibration as a corrective measure is a simpler problem to tackle than developing well-calibrated models upfront, and limiting the comparison to just one other post-calibration method greatly narrows the overall vision such a paper should have. For instance, isn’t post-calibration more likely to result in overfitting than a model that is implicitly calibrated at training time?
I have plenty of concerns with the submission itself, listed below:
- First and foremost, the paper is full of typos and grammatical errors. I genuinely struggled to read the paper end-to-end without being continually distracted by these issues. While some mistakes may indeed be genuine, others are only there due to sheer negligence and because the authors didn’t properly check the paper before submission.
- While the overall objective of this work (i.e. improving calibration of deep models) is clearly established, the overall presentation of ideas is very muddled and I initially struggled to properly understand what’s being proposed.  A simple diagram or illustration would have clarified some of the notation at the very least.
- The sloppiness in the presentation is also manifested in other ways. For example, in Figure 1, the plots should be individually titled (‘uncalibrated’, ‘temp-scal' and ‘BNN') in order to immediately distinguish between them; instead, all this information is contained in the caption whereas it could just as easily have been added to the plot.
- As alluded to earlier, I am disappointed by the lack of scope in the paper. The experimental evaluation should have been widened to include direct comparisons against BNN models which one might expect to be slightly better-calibrated upfront. There has also been significant interest in improving the calibration of deep models by stacking different architectures in such a way that the model is implicitly calibrated at training time. Examples of such papers include ‘Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks’ (Bradshaw et al, 2017), and ‘Calibrating Deep Convolutional Gaussian Processes’ (Tran et al, 2018). The deep kernel learning schemes developed by Wilson et al. also discuss similar hybrid models. 
- With reference to the papers cited above, one possible extension the authors could consider is to use a Gaussian process for post-calibration instead of a BNN, although I suspect this may have already been investigated in the past. In any case, this warrants further discussion. 
- I can’t disentangle the two contributions listed at the bottom of Pg 2 and the top of Pg 3. There is no theoretical evaluation of the ‘alternative hypothesis’ being mentioned, and the investigation is entirely limited to the offline setting, so I’m not entirely sure what distinction the authors are trying to make here. 
- In the same section, the authors then remark that ‘Our results open new perspectives to improve the variational approximation…’ and ‘we believe our results might foster further research in…’, before proceeding to list a dozen or so papers which might be inspired by this work. However, I can’t really see how the single contribution being presented in this paper can have significant impact on the related work. I encourage the authors to substantiate their claims with more concrete examples rather than simply include vague mentions of other papers. 
- The structure and content of Section 3 is quite perplexing. Effectively, up until Equation 3, the authors are simply restating how to use VI for BNNs, with no mention whatsoever of how this fits in the storyline of model calibration. Whereas such a section should have contained novel methodology and/or intuition, the only reference to using BNNs for post-calibration is found in a small paragraph at the end of Pg 4, before immediately proceeding to the Experiments section. Once again, this makes any contributions of the paper unclear and inconclusive. Spurious comments such as the inconsequential connection to MDL further accentuate the paper’s lack of identity and focus.
- There are also some problematic technical details in this section, such as the definitive choice of using a two-layered BNN with no justification whatsoever. It is well known than plain BNNs also struggle to deliver well-calibrated outputs, and yet the authors immediately settle on a two-layered fully-connected network without stopping to consider whether some other network configuration or initialisation scheme might be more appropriate. Some introspection is later given in the experiment accompanied by Figure 2, but the analysis carried out there is just not sufficient. 
- There are some instances where the authors use text while in math mode, which gives poor formatting as exemplified by ‘conf’ in Equation 4. 
- Referring to ‘datasets’ as ‘databases’ in Section 4.1 is unusual. Some of the commentary in this subsection is also very difficult to interpret. For example, what is meant by ‘uses BNNs’? Does this mean that a BNN appears in the model being calibrated or is this referring to the BNN used to carry out calibration? The majority of these ambiguous statements could have been avoided had more care been given to checking the paper properly before submission.
- In their discussion of the results, the authors state that ‘We cannot conclude that BNNs are calibrating at the cost of losing accuracy’, which I consider to be an overly sunny view of the results. Even if minor, a dip in accuracy is observed in almost every example provided in the Experiments section, dropping as much as 3% for CIFAR-100. Given that calibration is the primary focus of this paper, it might also be worth including another metic for validating this criteria, such as the Brier score.
-- Recommendation --
Unfortunately, the material presented here is neither significant enough nor sufficiently explored to spark much interest. The overall scope of the paper is disappointingly limited, while novel ideas and design choices are poorly motivated and communicated throughout. This submission feels rushed and incomplete, and consequently well below the conference’s standards.
Pros/Cons summary:
+  The proposal yields good results in the provided experiments
-   Minor contributions that are not convincing enough
-   Muddled presentation of ideas
-   Dubious or weakly motivated design choices
-   Poorly written with plenty of typos
-   Difficult to follow