Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel
Neural Networks (NNs) have been extensively used for a wide spectrum of real-world regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that prediction must also be estimated. Standard NNs, which are most often used in such tasks, do not provide uncertainty information. Existing approaches address this issue by combining Bayesian models with NNs, but these models are hard to implement, more expensive to train, and usually do not predict as accurately as standard NNs. In this paper, a new framework (RIO) is developed that makes it possible to estimate uncertainty in any pretrained standard NN. The behavior of the NN is captured by modeling its prediction residuals with a Gaussian Process, whose kernel includes both the NN*s input and its output. The framework is justified theoretically and evaluated in twelve real-world datasets, where it is found to (1) provide reliable estimates of uncertainty, (2) reduce the error of the point predictions, and (3) scale well to large datasets. Given that RIO can be applied to any standard NN without modifications to model architecture or training pipeline, it provides an important ingredient for building real-world NN applications.
This paper solves an interesting scientific and applied problem: can we construct an algorithm to predict uncertainties without re-training/modifying existing neural network training algos? The authors propose a novel technique (called RIO) which leverages existing neural network but use both the input as well as the output of the neural net as an input to a GP which regresses on the residual error of the neural network. The authors describe the theoretical foundations as well as show empirical results on multiple datasets. My thoughts on the paper: - The paper is well written and from section 2.1 it is clear how one could re-produce their method. - The theoretical section 2.2 feels a bit rushed, I think it would be worth sharing the high level intuition behind some of the theory first before going into the details. - Section 2.4 could be more explicit about what *large scale* means. I.o.w. from a practical point of view, the method is only limited by approximate inference for Gaussian processes. Anno 2019 this is ... - The empirical section is particularly strong and contains a wide variety of experiments with detailed analysis. As a result, I think this is a good piece of scientific work that could be interesting to the wider community. Although I did not re-run the results, the authors do share full source code for their results.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel
Neural Networks (NNs) have been extensively used for a wide spectrum of real-world regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that prediction must also be estimated. Standard NNs, which are most often used in such tasks, do not provide uncertainty information. Existing approaches address this issue by combining Bayesian models with NNs, but these models are hard to implement, more expensive to train, and usually do not predict as accurately as standard NNs. In this paper, a new framework (RIO) is developed that makes it possible to estimate uncertainty in any pretrained standard NN. The behavior of the NN is captured by modeling its prediction residuals with a Gaussian Process, whose kernel includes both the NN*s input and its output. The framework is justified theoretically and evaluated in twelve real-world datasets, where it is found to (1) provide reliable estimates of uncertainty, (2) reduce the error of the point predictions, and (3) scale well to large datasets. Given that RIO can be applied to any standard NN without modifications to model architecture or training pipeline, it provides an important ingredient for building real-world NN applications.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper solves an interesting scientific and applied problem: can we construct an algorithm to predict uncertainties without re-training/modifying existing neural network training algos? The authors propose a novel technique (called RIO) which leverages existing neural network but use both the input as well as the output of the neural net as an input to a GP which regresses on the residual error of the neural network. The authors describe the theoretical foundations as well as show empirical results on multiple datasets. My thoughts on the paper: - The paper is well written and from section 2.1 it is clear how one could re-produce their method. - The theoretical section 2.2 feels a bit rushed, I think it would be worth sharing the high level intuition behind some of the theory first before going into the details. - Section 2.4 could be more explicit about what *large scale* means. I.o.w. from a practical point of view, the method is only limited by approximate inference for Gaussian processes. Anno 2019 this is ... - The empirical section is particularly strong and contains a wide variety of experiments with detailed analysis. As a result, I think this is a good piece of scientific work that could be interesting to the wider community. Although I did not re-run the results, the authors do share full source code for their results.
assistant
This paper solves an interesting scientific and applied problem: can we construct an algorithm to predict uncertainties without re-training/modifying existing neural network training algos? The authors propose a novel technique (called RIO) which leverages existing neural network but use both the input as well as the output of the neural net as an input to a GP which regresses on the residual error of the neural network. The authors describe the theoretical foundations as well as show empirical results on multiple datasets. My thoughts on the paper: - The paper is well written and from section 2.1 it is clear how one could re-produce their method. - The theoretical section 2.2 feels a bit rushed, I think it would be worth sharing the high level intuition behind some of the theory first before going into the details. - Section 2.4 could be more explicit about what *large scale* means. I.o.w. from a practical point of view, the method is only limited by approximate inference for Gaussian processes. Anno 2019 this is ... - The empirical section is particularly strong and contains a wide variety of experiments with detailed analysis. As a result, I think this is a good piece of scientific work that could be interesting to the wider community. Although I did not re-run the results, the authors do share full source code for their results.