Representing inferential uncertainty in deep neural networks through sampling
As deep neural networks (DNNs) are applied to increasingly challenging problems, they will need to be able to represent their own uncertainty. Modelling uncertainty is one of the key features of Bayesian methods. Bayesian DNNs that use dropout-based variational distributions and scale to complex tasks have recently been proposed. We evaluate Bayesian DNNs trained with Bernoulli or Gaussian multiplicative masking of either the units (dropout) or the weights (dropconnect). We compare these Bayesian DNNs ability to represent their uncertainty about their outputs through sampling during inference. We tested the calibration of these Bayesian fully connected and convolutional DNNs on two visual inference tasks (MNIST and CIFAR-10). By adding different levels of Gaussian noise to the test images, we assessed how these DNNs represented their uncertainty about regions of input space not covered by the training set. These Bayesian DNNs represented their own uncertainty more accurately than traditional DNNs with a softmax output. We find that sampling of weights, whether Gaussian or Bernoulli, led to more accurate representation of uncertainty compared to sampling of units. However, sampling units using either Gaussian or Bernoulli dropout led to increased convolutional neural network (CNN) classification accuracy. Based on these findings we use both Bernoulli dropout and Gaussian dropconnect concurrently, which approximates the use of a spike-and-slab variational distribution. We find that networks with spike-and-slab sampling combine the advantages of the other methods: they classify with high accuracy and robustly represent the uncertainty of their classifications for all tested architectures.
This paper investigates whether the variational inference interpretation of dropout, as introduced in [Gal & Ghahramani (2016), and Kingma et al (2015)], can lead to good estimates of mode uncertainty outside of the training distribution. This is an area of research that indeed warrants more experimental investigation. One very interesting finding is that MC integration leads to much calibration, thus probably much better out-of-sample prediction, than the more usual. Critique: - As explained in Kingma et al (2015), when using continuous posterior distributions over the weights, the dropout rate can be optimized, leading to better regularization. While the paper is cited in the introduction, this adaptive form of dropout is missing from experiments, without clarification. - Only the dropout rate p=0.5 was used across experiments, while the optimal rate is problem dependent, as found by earlier published work. - No new ideas are presented, and the analysis in the paper is quite limited. As it stands, this would be more appropriate for a workshop.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Representing inferential uncertainty in deep neural networks through sampling
As deep neural networks (DNNs) are applied to increasingly challenging problems, they will need to be able to represent their own uncertainty. Modelling uncertainty is one of the key features of Bayesian methods. Bayesian DNNs that use dropout-based variational distributions and scale to complex tasks have recently been proposed. We evaluate Bayesian DNNs trained with Bernoulli or Gaussian multiplicative masking of either the units (dropout) or the weights (dropconnect). We compare these Bayesian DNNs ability to represent their uncertainty about their outputs through sampling during inference. We tested the calibration of these Bayesian fully connected and convolutional DNNs on two visual inference tasks (MNIST and CIFAR-10). By adding different levels of Gaussian noise to the test images, we assessed how these DNNs represented their uncertainty about regions of input space not covered by the training set. These Bayesian DNNs represented their own uncertainty more accurately than traditional DNNs with a softmax output. We find that sampling of weights, whether Gaussian or Bernoulli, led to more accurate representation of uncertainty compared to sampling of units. However, sampling units using either Gaussian or Bernoulli dropout led to increased convolutional neural network (CNN) classification accuracy. Based on these findings we use both Bernoulli dropout and Gaussian dropconnect concurrently, which approximates the use of a spike-and-slab variational distribution. We find that networks with spike-and-slab sampling combine the advantages of the other methods: they classify with high accuracy and robustly represent the uncertainty of their classifications for all tested architectures.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper investigates whether the variational inference interpretation of dropout, as introduced in [Gal & Ghahramani (2016), and Kingma et al (2015)], can lead to good estimates of mode uncertainty outside of the training distribution. This is an area of research that indeed warrants more experimental investigation. One very interesting finding is that MC integration leads to much calibration, thus probably much better out-of-sample prediction, than the more usual. Critique: - As explained in Kingma et al (2015), when using continuous posterior distributions over the weights, the dropout rate can be optimized, leading to better regularization. While the paper is cited in the introduction, this adaptive form of dropout is missing from experiments, without clarification. - Only the dropout rate p=0.5 was used across experiments, while the optimal rate is problem dependent, as found by earlier published work. - No new ideas are presented, and the analysis in the paper is quite limited. As it stands, this would be more appropriate for a workshop.
assistant
This paper investigates whether the variational inference interpretation of dropout, as introduced in [Gal & Ghahramani (2016), and Kingma et al (2015)], can lead to good estimates of mode uncertainty outside of the training distribution. This is an area of research that indeed warrants more experimental investigation. One very interesting finding is that MC integration leads to much calibration, thus probably much better out-of-sample prediction, than the more usual. Critique: - As explained in Kingma et al (2015), when using continuous posterior distributions over the weights, the dropout rate can be optimized, leading to better regularization. While the paper is cited in the introduction, this adaptive form of dropout is missing from experiments, without clarification. - Only the dropout rate p=0.5 was used across experiments, while the optimal rate is problem dependent, as found by earlier published work. - No new ideas are presented, and the analysis in the paper is quite limited. As it stands, this would be more appropriate for a workshop.