Conservative Uncertainty Estimation By Fitting Prior Networks
Obtaining high-quality uncertainty estimates is essential for many applications of deep neural networks. In this paper, we theoretically justify a scheme for estimating uncertainties, based on sampling from a prior distribution. Crucially, the uncertainty estimates are shown to be conservative in the sense that they never underestimate a posterior uncertainty obtained by a hypothetical Bayesian algorithm. We also show concentration, implying that the uncertainty estimates converge to zero as we get more data. Uncertainty estimates obtained from random priors can be adapted to any deep network architecture and trained using standard supervised learning pipelines. We provide experimental evaluation of random priors on calibration and out-of-distribution detection on typical computer vision tasks, demonstrating that they outperform deep ensembles in practice.
The paper shows that the MSE of a deep network trained to match fixed random network is a conservative estimate of uncertainty in expectation over many such network pairs. An experiment compares this previously proposed method to other approaches for uncertainty estimation on CIFAR 10. Strengths: - Obtaining uncertainty estimates for predictions of deep neural networks is an important and open research question. - Proposition 1 is an interesting result, although the paper does not seem to discuss its significance and implications enough. Weaknesses: - Proposition 1 is described as *our uncertainty estimates are never too small*. However, as the name of the proposition suggests, it seems to only hold in expectation over models trained, which is a quite different statement. - Proposition 2 seems to simply state that a small network can be distilled into a large network. Maybe I missed part of the reasoning here? Otherwise, it should probably not be highlighted as a major contribution. - The experimental evaluation is very limited, training only on CIFAR 10. While the experiments add little value to the community, this may be acceptable for a mostly theoretical paper. Clarity: - The paper was difficult to read and unclear in explanations. It could help to define notation upfront instead of introducing shorthands on the way. - In Figure 3, the Y axis limits should be fixed across seen and unseen histograms for the same method. The current presentation is a bit misleading here, as the presented method seems to have moved the most mass under this chart scaling. Comments: - As found in prior work cited in the submission, the method tends to perform well with just one network pair. This raises the question whether the contribution of the paper that holds in expectation over many pairs and the empirical success of the approach are connected. - The marginal posterior variance sigma^2(x_star) appears in various forms with hat, tilde, and different subscripts. It maybe worth assigning different letters to these to avoid confusion.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Conservative Uncertainty Estimation By Fitting Prior Networks
Obtaining high-quality uncertainty estimates is essential for many applications of deep neural networks. In this paper, we theoretically justify a scheme for estimating uncertainties, based on sampling from a prior distribution. Crucially, the uncertainty estimates are shown to be conservative in the sense that they never underestimate a posterior uncertainty obtained by a hypothetical Bayesian algorithm. We also show concentration, implying that the uncertainty estimates converge to zero as we get more data. Uncertainty estimates obtained from random priors can be adapted to any deep network architecture and trained using standard supervised learning pipelines. We provide experimental evaluation of random priors on calibration and out-of-distribution detection on typical computer vision tasks, demonstrating that they outperform deep ensembles in practice.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper shows that the MSE of a deep network trained to match fixed random network is a conservative estimate of uncertainty in expectation over many such network pairs. An experiment compares this previously proposed method to other approaches for uncertainty estimation on CIFAR 10. Strengths: - Obtaining uncertainty estimates for predictions of deep neural networks is an important and open research question. - Proposition 1 is an interesting result, although the paper does not seem to discuss its significance and implications enough. Weaknesses: - Proposition 1 is described as *our uncertainty estimates are never too small*. However, as the name of the proposition suggests, it seems to only hold in expectation over models trained, which is a quite different statement. - Proposition 2 seems to simply state that a small network can be distilled into a large network. Maybe I missed part of the reasoning here? Otherwise, it should probably not be highlighted as a major contribution. - The experimental evaluation is very limited, training only on CIFAR 10. While the experiments add little value to the community, this may be acceptable for a mostly theoretical paper. Clarity: - The paper was difficult to read and unclear in explanations. It could help to define notation upfront instead of introducing shorthands on the way. - In Figure 3, the Y axis limits should be fixed across seen and unseen histograms for the same method. The current presentation is a bit misleading here, as the presented method seems to have moved the most mass under this chart scaling. Comments: - As found in prior work cited in the submission, the method tends to perform well with just one network pair. This raises the question whether the contribution of the paper that holds in expectation over many pairs and the empirical success of the approach are connected. - The marginal posterior variance sigma^2(x_star) appears in various forms with hat, tilde, and different subscripts. It maybe worth assigning different letters to these to avoid confusion.
assistant
The paper shows that the MSE of a deep network trained to match fixed random network is a conservative estimate of uncertainty in expectation over many such network pairs. An experiment compares this previously proposed method to other approaches for uncertainty estimation on CIFAR 10. Strengths: - Obtaining uncertainty estimates for predictions of deep neural networks is an important and open research question. - Proposition 1 is an interesting result, although the paper does not seem to discuss its significance and implications enough. Weaknesses: - Proposition 1 is described as *our uncertainty estimates are never too small*. However, as the name of the proposition suggests, it seems to only hold in expectation over models trained, which is a quite different statement. - Proposition 2 seems to simply state that a small network can be distilled into a large network. Maybe I missed part of the reasoning here? Otherwise, it should probably not be highlighted as a major contribution. - The experimental evaluation is very limited, training only on CIFAR 10. While the experiments add little value to the community, this may be acceptable for a mostly theoretical paper. Clarity: - The paper was difficult to read and unclear in explanations. It could help to define notation upfront instead of introducing shorthands on the way. - In Figure 3, the Y axis limits should be fixed across seen and unseen histograms for the same method. The current presentation is a bit misleading here, as the presented method seems to have moved the most mass under this chart scaling. Comments: - As found in prior work cited in the submission, the method tends to perform well with just one network pair. This raises the question whether the contribution of the paper that holds in expectation over many pairs and the empirical success of the approach are connected. - The marginal posterior variance sigma^2(x_star) appears in various forms with hat, tilde, and different subscripts. It maybe worth assigning different letters to these to avoid confusion.