Bayesian neural networks (BNNs) introduce uncertainty estimation to deep networks by performing Bayesian inference on network weights. However, such models bring the challenges of inference, and further BNNs with weight uncertainty rarely achieve superior performance to standard models. In this paper, we investigate a new line of Bayesian deep learning by performing Bayesian reasoning on the structure of deep neural networks. Drawing inspiration from the neural architecture search, we define the network structure as gating weights on the redundant operations between computational nodes, and apply stochastic variational inference techniques to learn the structure distributions of networks. Empirically, the proposed method substantially surpasses the advanced deep neural networks across a range of classification and segmentation tasks. More importantly, our approach also preserves benefits of Bayesian principles, producing improved uncertainty estimation than the strong baselines including MC dropout and variational BNNs algorithms (e.g. noisy EK-FAC).
This paper proposes to do approximate Bayesian inference in neural networks by treating the neural network structure as a random variable (RV), while inferring the parameters with point estimates. 
While performing Bayesian inference for the neural network structure is sensible, I am not convinced by the approach taken in this work. 
The biggest problem is that the model uses a point estimate of the same weights for different, random network structures. 
Major problems:
- In the motivation the authors write “DBSN places distributions on the network structure, introducing more global randomness, thus is probable to yield more diverse predictions, and ensembling them brings more calibrated uncertainty”. What is “more global randomness”? This is used multiple times. Does it refer to the hierarchy in the graphical model? Please be precise here and point that out in your model by using an equation or graphical model. Or is it just an intuition? 
- Generally, I would agree that integrating out multiple network structures provides better calibrated uncertainty. However, given that the authors use point estimates for the weights, it is not clear if that is still true, especially since the number of different architectures used in practice is small. 
- What’s more, the approach uses *the same* point estimates for different structures. This leads to a graphical model, where the weights are not conditioned on the architecture/structure. This modeling choice could be a big limitation, because the weights now have to fit multiple different architectures; it may thus defeat the calibration completely. One can easily imagine that only a single random architecture works well with the learned point estimates, thus resulting in an (almost) deterministic model. I assume that this modeling choice was made for practical reasons, but could you expand on its implications / interpretation / limitations? Does the posterior of such a constrained model not quickly converge to an “almost” dirac, effectively just one network structure? 
- Sec. 3.1. presents the above problem resulting from a modeling decision as a “training challenge”. To counter this problem, the authors propose to reduce the variance of the structure distribution. By doing so, the approach becomes even less Bayesian and the predictive uncertainty becomes even less reliable. 
- “We only learn the connections between the B internal nodes, as shown in Appendix D”. All deterministic weights are learned, but the structure only for some parts of the model? If this is the case, then approach becomes again less probabilistic.
Regarding the experiments, the stddevs are calculated from 3(!) independent runs and thus completely misleading (imagine the stddev of the stddev estimate). 
In summary, the model choice of point estimates for the weights, which are not conditioned on the architecture, leads to various problems. The authors have to introduce tricks such as reducing the variance of the random network structures or learning only a part of the whole structure to make the approach converge. The resulting probabilistic model and its predictive uncertainty is questionable. For this reason, this paper should be rejected. 
Minor problems
- Sec. 3.2. “Improvements of the structure learning space”. What is a “structure learning space”? 
- Section 3 introduces the ELBO in Eq. (4) before the complete model is specified. Please specify the whole model first. How do w and alpha depend on each other in your model? 
- The prior for the weights is omitted; at the same time it is mentioned in the experiments (Sec.5.1.) that weight decay is applied. Why not just be explicit about it and say that a Gaussian prior is used?
- Background Sec. 2.2. is not clear.  what is a cell? some deterministic transformation in general? bunch of neural network layers? What are the operation (last term in Eq. (2)) doing? This is not detailed and abstract to me. Are the alphas probabilities? Is Eq. (2) consequently a mixture model of different architectures? Or is this here just a weighted sum, where the weights take arbitrary values? A small visualization (additionally) might help here, but can probably be rectified by better explanation.
- Bayesian reasoning on the structure. Inference?
- Writing that you propose a new “framework” is a bit grandiose for what is actually proposed. There has been previous work in which the architecture is inferred as well and these approaches would certainly be part of the same framework. Please just say model/algorithm/approach, whatever is applicable. 
- new paragraph starting at “To empirically validate” in the intro.
- Before (4): “Then we rewrite the approximation error”. Eq. (4) is the ELBO, this is not an approximation error.