Bayesian Neural Networks (BNNs) provides a mathematically grounded framework to quantify uncertainty. However BNNs are computationally inefficient, thus are generally not employed on complicated machine learning tasks. Deep Ensembles were introduced as a Bootstrap inspired frequentist approach to the community, as an alternative to BNN's. Ensembles of deterministic and stochastic networks are a good uncertainty estimator in various applications (Although, they are criticized for not being Bayesian). We show Ensembles of deterministic and stochastic Neural Networks can indeed be cast as an approximate Bayesian inference. Deep Ensembles have another weakness of having high space complexity, we provide an alternative to it by modifying the original Bayes by Backprop (BBB) algorithm to learn more general concrete mixture distributions over weights. We show our methods and its variants can give better uncertainty estimates at a significantly lower parametric overhead than Deep Ensembles. We validate our hypothesis through experiments like non-linear regression, predictive uncertainty estimation, detecting adversarial images and exploration-exploitation trade-off in reinforcement learning.
Summary: This paper proposes to use either relaxed mixture distributions or relaxed mixtures of matrix Gaussians as the approximate posterior for Bayesian neural networks. Naturally, taking the mixture variance to zero allows a stretched interpretation of ensembling to become Bayesian as well. Experiments are performed on small neural networks on regression tasks, as well as uncertainty estimation experiments on deep networks. Finally, a downstream task of bandit optimization (the mushroom experiment) is performed. 
post-rebuttal: I'd like to sincerely thank the authors of the paper for their enlightening discussions about their work. I'm inclined to think that this paper has a future, and definitely encourage the authors to resubmit - taking into account the reviews here. In particular, it would be greatly helpful if they cleaned up some of the writing and explanation, as our discussion points to.
However, my stance on the codebase remains mostly unchanged - while a notebook ended up being released with their method, it was too close to the deadline and in a bit too rough of shape to really be able to poke through their implementation. 
From a quick pass, their implementation seems to be correct - although rather than attempting to base their code on the old torch/lua implemenations, I'd still suggest using a pre-written version. However, without the completely trained numbers, I cannot in good conscience vote to accept the paper. 
Tldr: I vote to reject this paper for several reasons. The interpretation of ensembling as variational inference is both flawed and well known. Mixture distributions have been previously proposed for variational inference. There are significant enough weaknesses in the empirical results that make me question the authors’ own implementations. I will increase my score if these issues are resolved.
Originality:
1)	It is well known (and immediately obvious) that one can interpret ensembles as a Bayesian method – with a mixture of delta functions around the independently trained models. Furthermore, the Bayesian bootstrap (Rubin, 1983) is a Bayesian interpretation of the bootstrap (identical under many conditions), while re-weighted ensembles are Bayesian models (Newton, 1995; Newton, 2018) that converge to the true posterior under many conditions – intriguingly, this version of re-weighting could potentially have good uncertainty quantification properties. For the specific case of stacking (which is closely related to ensembling), there are interpretations of stacking as Bayesian model combination (Yao et al, 2018, Clyde & Iversen, 2013). As a result, this view of ensembling as a Bayesian method is not novel.
2)	Additionally, the derivation of dropout as Bayesian inference (which relies on a similar interpretation on taking the variance parameter in the Gaussian to zero) is known to be flawed (Hron et al, 2018); this flaw is directly applicable to the interpretation of ensembling as variational inference as described in Section 3 of this paper. The issue is fundamental: when taking the variance parameter to zero, the supports of the variational distribution and the true distributions have a mismatch, causing the KL divergence to become infinite. As a result, one cannot minimize the objective (as it is infinite by design).
Clarity:
1)	The proposed methods are clearly explained in the setting under which one might be able to re-implement them, particularly in the description of the approximate posteriors.
post rebuttal: to my knowledge, this was never really addressed. In particular, "why do we want to interpet ensembling in a Bayesian manner?" is still an open question to me.
2)	Why is that we should expect (approximate) Bayesian procedures to be better calibrated than frequentist ones? It may be the case that if the integrals are accurately estimated (e.g. we have reached the true posterior and the model used for the data is in fact accurate), then we should expect Bayesian procedures to be well calibrated. However, this does not seem to be the case. A reference or argument for why (even in the model incorrect & approximate inference cases) in the introduction would greatly enhance the strength of the paper.
post rebuttal: I think that this issue was addressed in a reasonable manner through the rebuttal, but will require a bit of rewriting.
3)	From a quick pass through Appendix A.2, the derivation is quite unstructured and difficult to follow. It is tough to see exactly what is being meant by the probabilities in the first line. 
Additionally, the phrasing of “We further lower bound…” seems to suggest that the bound being used is not the well known bound of Durrieu et al, 2012, but a looser bound. If so, what is the approximation accuracy of this bound in relation to the other bounds compared against in this setting?
4)	In sum, three different methods are proposed: concrete mixtures of Gaussians, concrete ensembles, and concrete mixtures of matrix variate Gaussians. Throughout the experiments, it seems like __one__ of these methods always wins on the task at hand. However, it does not always seem to be the same method. Is there a recommendation for practitioners as to which method to use – or is this simply task dependent? Specifically, is there an understanding as to why each method performs vastly differently on each task?
A priori, I would expect the concrete mixture of matrix variate Gaussians to always perform best because it does seem to be able to model multivariate posteriors the best. However, it seems to perform worse on several of the predictive uncertainty tasks.
5)	For the uncertainty experiments, it would be nice to have more qualitative numbers – for example the expected calibration error – rather than just the plots on in and out of distribution example. After all, not only do we want to be able to recognize out of distribution examples, but we want to be able to trust the probabilities that are output. 
6)	In Figure 4, thank you for providing the adversarial attacks used to fool the networks – the step size parameter doesn’t give a great intuition, and the images are welcome and useful.
Significance:
post rebuttal: I think that this issue was satisfactorily addressed.
1)	Mixture distributions as the variational family have already been proposed – at least once – see Miller et al, 2017 for an example that additionally uses the reparameterization trick. There, updates to the posterior are performed via selectively adding mixture components when necessary. Given that they also run experiments on neural networks, it would be nice to see a comparison to that method. 
Quality:
1)	I am very concerned by the performance of the ResNet20 models trained on CIFAR10 in the paper. The reported results for deep ensembles (an ensemble of _three_ independently trained models) in Appendix A are 85.23% accuracy. By comparison, the number in the original paper (He et al, 2015) is 91.25% for a _single_ model. Furthermore, the first link to a PyTorch repo from a Google search (https://github.com/akamaster/pytorch_resnet_cifar10) comes up with a re-implementation that gets 91.63% (again for a single model).
__This issue alone is enough for me to vote to reject the paper until the authors’ implementations are fixed and run with deep ensembled ResNets that get similar accuracy. I hope to see the implementation issues fixed in the rebuttal period. __
2)	Furthermore, in Appendix 2, I see a potential issue in using the KL divergence and would like the authors to clarify.
If the secondary lower bound is being used to approximate the KL divergence between the mixture distributions, it becomes very unclear if what is being optimized is in fact a __lower bound__ on the log probability. After all, the standard ELBO is written as:
	Log p(y) \geq ELBO := E_q(\log p(y | \theta)) – KL(q(\theta) || p(\theta)) 
Implying that another lower bound on the KL would be greater than the ELBO and thus not necessarily a lower bound on the log marginal likelihood.
3)	Bottom of page 7: “The posterior is used only for fully connected layers…” Why is this the case? Alternative Bayesian methods can be used on the full network.
Minor Comments:
-	Please attempt to use the math_commands.tex in the ICLR style file to write math. This makes the math standardized throughout.
-	Figure 1 is especially unclear and the legends and captions should be made considerably larger. Zooming in at 800% seems to be necessary to even be able to make out which method is which. Consider, placing all methods on a single plot and then coloring/dashing them separately. Additionally, please compare to HMC on this task (and an equivalent RBF GP) to show the uncertainties from the “gold standard” methods.
-	Please additionally make the figure legends larger for the rest of the figures. 
-	What is the meaning of time in Figure 2? I assume that it means training time, but it is not especially clear from the caption. With that being said, I think that it is quite interesting that the marginals do seem to look multi-modal, even at the end of training.
-	Line before Eq 7: please use \citet if possible to cite Gupta & Nagar, rather than stating the clunky Gupta et al. \citep{Gupta & Nagar}.
-	Eq 2: please do not use \hspace{-…} to save space in equations, so that the equation does not overwrite lines before.
-	Page 2: “Deep Ensembles have lacked theoretical support …” In order to make this claim, you must first explain why one ought to be Bayesian in the first place. See above for the history of interpreting ensembling from a Bayesian perspective. There is no a priori reason why one would not just wish to have frequentist justifications of ensembling (and potentially even calibration).
-	Please do not capitalize Ensembling or Variational Inference throughout.
References:
Clyde & Iversen, Bayesian model averaging in the M-open framework. In Bayesian Theory and Applications, 2013. DOI:10.1093/acprof:oso/9780199695607.003.0024
He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition, CVPR, 2016. https://arxiv.org/abs/1512.03385
Hron, Matthews & Ghahramani, Variational Bayesian dropout: pitfalls and fixes, ICML, 2018. https://arxiv.org/abs/1807.01969
Miller, Foti & Adams, Variational Boosting: Iteratively Refining Posterior Approximations, ICML, 2017. https://arxiv.org/abs/1611.06585
Newton & Raftery, Approximate Bayesian Inference with the Weighted Likelihood Bootstrap. JRSS:B, 1994. https://www.jstor.org/stable/2346025?seq=1#metadata_info_tab_contents
Newton, Polson & Xu, Weighted Bayesian Bootstrap for Scalable Bayes, 2018; https://arxiv.org/abs/1803.04559
Rubin, The Bayesian Bootstrap, 1981. Annals of Statistics. https://projecteuclid.org/euclid.aos/1176345338
Yao, Vehtari, Simpson, & Gelman, Using Stacking to Average Bayesian Predictive Distributions, Bayesian Analysis, 2018; https://projecteuclid.org/euclid.ba/1516093227