Bayesian Inference for Large Scale Image Classification
Bayesian inference promises to ground and improve the performance of deep neural networks. It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness. Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters. Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind optimization methods for large scale deep learning tasks. We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network. ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients. We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood. We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.
This paper proposes a novel MCMC algorithm (ATMC) that estimates and samples from the posterior distribution of neural network weights. The motivation for this approach is that applying Bayesian inference to deep learning should lead to less overfitting and better uncertainty-calibrated models. Unlike previous work, the proposed method scales to large models (ResNet) and data sets (ImageNet). In addition to the main contribution, this work improves an existing numerical integrator necessary for the ATMC sampler. Moreover, in order to apply this method to ResNet, the authors use a modified version of ResNet without stochastic regularization (batch normalization and dropout). Empirical results show that the proposed method outperforms baselines on accuracy, log-likelihood, and uncertainty calibration. The single-sample posterior already yields decent results, beating a batchnorm-free version of ResNet. Overall, the paper is well-written, fluently readable, and provides a clear presentation of motivations and methods. Experiments, although not very extensive, are described in sufficient detail and corroborate the claims. Related previous work is cited throughout the paper, although there is no explicit section for it. Weaknesses: While the authors claim that the need for hyperparameter tuning is reduced, they use a cyclic step size with parameter n, a Laplace prior with parameter b, a momentum noise with parameter 0.9, and dataset-specific parameters h0, m, and c. This weakens (or even contradicts) the claim, and raises the question of how much the choice of these hyperparameters affects performance. On ImageNet, ATMC doesn*t seem to be well calibrated, and the authors do not discuss the fact that the calibration is basically on par with SGNHT. Minor: - page 4: practise -> practice - page 7: extra space after *compatible with MCMC methods* - page 7: problem specific -> problem-specific
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Bayesian Inference for Large Scale Image Classification
Bayesian inference promises to ground and improve the performance of deep neural networks. It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness. Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters. Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind optimization methods for large scale deep learning tasks. We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network. ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients. We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood. We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes a novel MCMC algorithm (ATMC) that estimates and samples from the posterior distribution of neural network weights. The motivation for this approach is that applying Bayesian inference to deep learning should lead to less overfitting and better uncertainty-calibrated models. Unlike previous work, the proposed method scales to large models (ResNet) and data sets (ImageNet). In addition to the main contribution, this work improves an existing numerical integrator necessary for the ATMC sampler. Moreover, in order to apply this method to ResNet, the authors use a modified version of ResNet without stochastic regularization (batch normalization and dropout). Empirical results show that the proposed method outperforms baselines on accuracy, log-likelihood, and uncertainty calibration. The single-sample posterior already yields decent results, beating a batchnorm-free version of ResNet. Overall, the paper is well-written, fluently readable, and provides a clear presentation of motivations and methods. Experiments, although not very extensive, are described in sufficient detail and corroborate the claims. Related previous work is cited throughout the paper, although there is no explicit section for it. Weaknesses: While the authors claim that the need for hyperparameter tuning is reduced, they use a cyclic step size with parameter n, a Laplace prior with parameter b, a momentum noise with parameter 0.9, and dataset-specific parameters h0, m, and c. This weakens (or even contradicts) the claim, and raises the question of how much the choice of these hyperparameters affects performance. On ImageNet, ATMC doesn*t seem to be well calibrated, and the authors do not discuss the fact that the calibration is basically on par with SGNHT. Minor: - page 4: practise -> practice - page 7: extra space after *compatible with MCMC methods* - page 7: problem specific -> problem-specific
assistant
This paper proposes a novel MCMC algorithm (ATMC) that estimates and samples from the posterior distribution of neural network weights. The motivation for this approach is that applying Bayesian inference to deep learning should lead to less overfitting and better uncertainty-calibrated models. Unlike previous work, the proposed method scales to large models (ResNet) and data sets (ImageNet). In addition to the main contribution, this work improves an existing numerical integrator necessary for the ATMC sampler. Moreover, in order to apply this method to ResNet, the authors use a modified version of ResNet without stochastic regularization (batch normalization and dropout). Empirical results show that the proposed method outperforms baselines on accuracy, log-likelihood, and uncertainty calibration. The single-sample posterior already yields decent results, beating a batchnorm-free version of ResNet. Overall, the paper is well-written, fluently readable, and provides a clear presentation of motivations and methods. Experiments, although not very extensive, are described in sufficient detail and corroborate the claims. Related previous work is cited throughout the paper, although there is no explicit section for it. Weaknesses: While the authors claim that the need for hyperparameter tuning is reduced, they use a cyclic step size with parameter n, a Laplace prior with parameter b, a momentum noise with parameter 0.9, and dataset-specific parameters h0, m, and c. This weakens (or even contradicts) the claim, and raises the question of how much the choice of these hyperparameters affects performance. On ImageNet, ATMC doesn*t seem to be well calibrated, and the authors do not discuss the fact that the calibration is basically on par with SGNHT. Minor: - page 4: practise -> practice - page 7: extra space after *compatible with MCMC methods* - page 7: problem specific -> problem-specific