Bayesian Deep Learning via Stochastic Gradient MCMC with a Stochastic Approximation Adaptation
We propose a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables. Inspired by dropout, a popular tool for regularization and model ensemble, we assign sparse priors to the weights in deep neural networks (DNN) in order to achieve automatic “dropout” and avoid over-fitting. By alternatively sampling from posterior distribution through stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and optimizing latent variables via stochastic approximation (SA), the trajectory of the target weights is proved to converge to the true posterior distribution conditioned on optimal latent variables. This ensures a stronger regularization on the over-fitted parameter space and more accurate uncertainty quantification on the decisive variables. Simulations from large-p-small-n regressions showcase the robustness of this method when applied to models with latent variables. Additionally, its application on the convolutional neural networks (CNN) leads to state-of-the-art performance on MNIST and Fashion MNIST datasets and improved resistance to adversarial attacks.
 TITLE Bayesian deep learning via stochastic gradient mcmc with a stochastic approximation adaptation REVIEW SUMMARY Fairly well written paper on SG-MCMC type inference in neural networks with slab and spike priors. In my view, the originality and significance is limited. PAPER SUMMARY The paper develops a method for sampling/optimization of a Bayesian neural network with slab and spike priors on the weights. QUALITY I belive the contribution is technically sound (but I have not checked all equations or the proof of Theorem 1). The empirical evaluation is not unreasonable, but also not strongly convincing. CLARITY The paper is fairly well written, but grammar and use of English could be slightly improved (not so important). ORIGINALITY The paper builds on existing work on EM-type algorithms for slab and spike models and SG-MCMC for Bayesian inference in neural networks. The novelty of the contribution is limited: The main contribution is the combination of the two methods and some theoretical results. I am not able to judge if there is significant originality in the theoretical results (Theorem 1 + Corr 1+2) but if I am not mistaken it is more or less an application of a known result to this particular setting? SIGNIFICANCE While I think the proposed algorithm is reasonable and most likely useful in practice, I am not sure the contribution is substantial enough to gain large interest in the community. FURTHER COMMENTS Figure 2 (d+e) are in my view not so useful for assessing the training/test performance, but I am not even completely sure what the figures shows, as there are no axis labels. I would prefer some results on the loss, perhaps averaged over multiple data sets.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Bayesian Deep Learning via Stochastic Gradient MCMC with a Stochastic Approximation Adaptation
We propose a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables. Inspired by dropout, a popular tool for regularization and model ensemble, we assign sparse priors to the weights in deep neural networks (DNN) in order to achieve automatic “dropout” and avoid over-fitting. By alternatively sampling from posterior distribution through stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and optimizing latent variables via stochastic approximation (SA), the trajectory of the target weights is proved to converge to the true posterior distribution conditioned on optimal latent variables. This ensures a stronger regularization on the over-fitted parameter space and more accurate uncertainty quantification on the decisive variables. Simulations from large-p-small-n regressions showcase the robustness of this method when applied to models with latent variables. Additionally, its application on the convolutional neural networks (CNN) leads to state-of-the-art performance on MNIST and Fashion MNIST datasets and improved resistance to adversarial attacks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 TITLE Bayesian deep learning via stochastic gradient mcmc with a stochastic approximation adaptation REVIEW SUMMARY Fairly well written paper on SG-MCMC type inference in neural networks with slab and spike priors. In my view, the originality and significance is limited. PAPER SUMMARY The paper develops a method for sampling/optimization of a Bayesian neural network with slab and spike priors on the weights. QUALITY I belive the contribution is technically sound (but I have not checked all equations or the proof of Theorem 1). The empirical evaluation is not unreasonable, but also not strongly convincing. CLARITY The paper is fairly well written, but grammar and use of English could be slightly improved (not so important). ORIGINALITY The paper builds on existing work on EM-type algorithms for slab and spike models and SG-MCMC for Bayesian inference in neural networks. The novelty of the contribution is limited: The main contribution is the combination of the two methods and some theoretical results. I am not able to judge if there is significant originality in the theoretical results (Theorem 1 + Corr 1+2) but if I am not mistaken it is more or less an application of a known result to this particular setting? SIGNIFICANCE While I think the proposed algorithm is reasonable and most likely useful in practice, I am not sure the contribution is substantial enough to gain large interest in the community. FURTHER COMMENTS Figure 2 (d+e) are in my view not so useful for assessing the training/test performance, but I am not even completely sure what the figures shows, as there are no axis labels. I would prefer some results on the loss, perhaps averaged over multiple data sets.
assistant
 TITLE Bayesian deep learning via stochastic gradient mcmc with a stochastic approximation adaptation REVIEW SUMMARY Fairly well written paper on SG-MCMC type inference in neural networks with slab and spike priors. In my view, the originality and significance is limited. PAPER SUMMARY The paper develops a method for sampling/optimization of a Bayesian neural network with slab and spike priors on the weights. QUALITY I belive the contribution is technically sound (but I have not checked all equations or the proof of Theorem 1). The empirical evaluation is not unreasonable, but also not strongly convincing. CLARITY The paper is fairly well written, but grammar and use of English could be slightly improved (not so important). ORIGINALITY The paper builds on existing work on EM-type algorithms for slab and spike models and SG-MCMC for Bayesian inference in neural networks. The novelty of the contribution is limited: The main contribution is the combination of the two methods and some theoretical results. I am not able to judge if there is significant originality in the theoretical results (Theorem 1 + Corr 1+2) but if I am not mistaken it is more or less an application of a known result to this particular setting? SIGNIFICANCE While I think the proposed algorithm is reasonable and most likely useful in practice, I am not sure the contribution is substantial enough to gain large interest in the community. FURTHER COMMENTS Figure 2 (d+e) are in my view not so useful for assessing the training/test performance, but I am not even completely sure what the figures shows, as there are no axis labels. I would prefer some results on the loss, perhaps averaged over multiple data sets.