We propose a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables. Inspired by dropout, a popular tool for regularization and model ensemble, we assign sparse priors to the weights in deep neural networks (DNN) in order to achieve automatic "dropout" and avoid over-fitting. By alternatively sampling from posterior distribution through stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and optimizing latent variables via stochastic approximation (SA), the trajectory of the target weights is proved to converge to the true posterior distribution conditioned on optimal latent variables. This ensures a stronger regularization on the over-fitted parameter space and more accurate uncertainty quantification on the decisive variables. Simulations from large-p-small-n regressions showcase the robustness of this method when applied to models with latent variables. Additionally, its application on convolutional neural networks (CNN) leads to state-of-the-art performance on MNIST and Fashion MNIST datasets and improved resistance to adversarial attacks.
TITLE
Bayesian deep learning via stochastic gradient mcmc with a stochastic approximation adaptation
REVIEW SUMMARY
Fairly well written paper on SG-MCMC type inference in neural networks with slab and spike priors. In my view, the originality and significance is limited.
PAPER SUMMARY
The paper develops a method for sampling/optimization of a Bayesian neural network with slab and spike priors on the weights.
QUALITY
I belive the contribution is technically sound (but I have not checked all equations or the proof of Theorem 1). The empirical evaluation is not unreasonable, but also not strongly convincing.
CLARITY
The paper is fairly well written, but grammar and use of English could be slightly improved (not so important).    
ORIGINALITY
The paper builds on existing work on EM-type algorithms for slab and spike models and SG-MCMC for Bayesian inference in neural networks. The novelty of the contribution is limited: The main contribution is the combination of the two methods and some theoretical results. I am not able to judge if there is significant originality in the theoretical results (Theorem 1 + Corr 1+2) but if I am not mistaken it is more or less an application of a known result to this particular setting?
SIGNIFICANCE
While I think the proposed algorithm is reasonable and most likely useful in practice, I am not sure the contribution is substantial enough to gain large interest in the community.  
FURTHER COMMENTS
Figure 2 (d+e) are in my view not so useful for assessing the training/test performance, but I am not even completely sure what the figures shows, as there are no axis labels. I would prefer some results on the loss, perhaps averaged over multiple data sets.