Neural Networks (NNs) have been extensively used for a wide spectrum of realworld regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that prediction must also be estimated. Standard NNs, which are most often used in such tasks, do not provide uncertainty information. Existing approaches address this issue by combining Bayesian models with NNs, but these models are hard to implement, more expensive to train, and usually do not predict as accurately as standard NNs. In this paper, a new framework (RIO) is developed that makes it possible to estimate uncertainty in any pretrained standard NN. The behavior of the NN is captured by modeling its prediction residuals with a Gaussian Process, whose kernel includes both the NN's input and its output. The framework is justified theoretically and evaluated in twelve real-world datasets, where it is found to (1) provide reliable estimates of uncertainty, (2) reduce the error of the point predictions, and (3) scale well to large datasets. Given that RIO can be applied to any standard NN without modifications to model architecture or training pipeline, it provides an important ingredient for building real-world NN applications.
# Summary
The authors propose a method for post-hoc correction and predictive variance estimation for neural network models. The method fits a GP to the model residuals, and learns a composite kernel that combines two kernels defined on the input space and the model’s output space (called RIO, R for residual, and IO for the input-output kernel). The authors suggest that residual means and variances at test points can then be calculated explicitly using predictive distributions from the GP. The authors run a large panel of experiments across a number of datasets, and compare to a number of methods that draw connections between neural networks and GP’s. In addition, the full method is compared to a number of methods that utilize only some components of the full RIO method. In these experiments, the RIO method generally shows strong performance in both RMSE and NLPD compared to these baselines.
# Feedback
Overall, this is a neat method. It has the flavor of a number of other composite ML methods that have worked well in the past---e.g., boosting and platt scaling---but is different enough to stand on its own. The experimental results are quite promising.
However, I am torn about the paper, because the theoretical discussion of the method is quite convoluted and seems either irrelevant or incorrect. I wish that the authors had spent more time with small demonstrations of what the procedure does in some simple settings. This would give practitioners considering the method far more intuition about when they would expect it to work and fail than the current theoretical discussion.
## Uncertainty Discussion is Lacking
The motivation and discussion sell this method as an uncertainty quantification method, but almost all of the theoretical development revolves around prediction correction. The methods properties as an uncertainty quantification tool are underdeveloped.
The only theoretical point made about uncertainty estimation is Theorem 2.7, which states that the scalar variance of the GP “nugget” is positively correlated with the variance of the NN’s residuals. Providing a scalar summary of noise is not particularly compelling for a method advertised as a point-prediction uncertainty quantification method. In addition, it is not clear what probability distribution the “correlation” is defined over. The argument made in the proof seems quite obvious: if a GP is used to model a noisier process (i.e., residuals with a larger variance), it will in some cases classify that variability as independent noise.
If the authors wanted to focus on the properties of their method as an uncertainty quantification tool, they could discuss the assumptions underlying the GP error estimates, and when they would be likely to diverge from practical properties like predictive interval coverage. For example, because the base NN predictor is treated as fixed, it seems that this method ignore uncertainty that stems from the NN fit due to random initialization. Likewise, it seems that this method would not quantify uncertainty from resampling the data and obtaining a new NN predictor. )The coverage experiments in the appendix seem to confirm this -- generally, the predictive intervals generally under-cover the predicted values.) It’s fine if the method doesn’t quantify these types of uncertainty, but discussion of these types of issues would be far more welcome than the current convoluted theory in Section 2. This discussion might not yield theorems, but it would give practitioners useful guidelines for deciding whether the particular scheme would likely work for this application.
## Problems with the Error Correction Argument
The theory section, especially 2.2, was very difficult to parse. First, as a matter of style, a sequence of Lemma and Theorem statements are given without defining most of the notation used therein, and with almost no prose providing context or intuition. In the buildup to the theorems, it is also unclear which assertions about the decompositions of y_i are assumptions about the true data generating process, and which assertions are specifications of a particular GP model.
The substance also has some issues. I think the intention in this section is to get to a rather simple variance decomposition of the labels y. The question is how much variation in y or the residual is represented in the posterior predictive mean of a particular GP. It seems reasonable that in some cases, the structure in the residual may be more amenable to modeling with a stationary GP than the structure in the raw labels y. It is not clear that all of the theoretical complexity here is necessary to make this point.
Instead, the authors make a convoluted argument that attempts to establish that the errors from the NN + GP approach will be smaller under very general circumstances. The argument is phrased somewhat ambiguously (it is not clear exactly what is being assumed, and what is corresponds to the specification of a working model), but depending on how one reads this section, the argument makes statements that are either too broad to be correct, or too narrow to be relevant.
The argument decomposes for the raw labels and the residuals into pieces that a GP can “capture” or “represent”, and parts that it cannot. The two equations are:
y_i = f(x_i) + g(x_i) + \xi_i
R_i = (y_i - h_NN(x_i)) = r_f(x_i) + r_g(x_i) + \xi_i
f(.) and r_f(.) represent the portions of the label and residual processes, respectively, that the GP "captures". It is assumed that the GP will model this portion correctly, and leave the “epsilon-indistinguishable” portion g(.) or r_g(.) untouched. The argument then assumes that f(.) and r_f(.) will have proportional kernels, and so it is possible to show that the predictions of residuals based on r_f(.) will have smaller predictive variance than predictions based on f(.) as long as the variation represented by r_g(.) is smaller than the variation represented by g(.).
On its face, this argument raises some red flags. Because h_NN(.) is allowed to be an arbitrary function, the argument here should be symmetric. Why can’t we also get a guaranteed variance reduction by adding h_NN(.) to y rather than subtracting it? Perhaps some of this is captured in the parameter \delta, which quantifies the reduction in variation represented in r_g(.) vs g(.), but the argument that the kernel of r_f(.) can be no larger than the kernel f(.) in terms of trace (that is, the proportionality constant \alpha is not greater than 1) does not make sense. If h_NN(x_i) is simply -f(x_i), then these arguments would not go through. At the very least, conditions need to be articulated about the properties of h_NN(.).
Some of the strangeness comes from the fact that this is a poor model of most prediction problems, where the main issue with fitting a GP is not “indistinguishability”, but misspecification. Consider a process y_i that is non-stationary; say g(.) has a linear trend in some component of x. A GP with a stationary covariance kernel fit to this process (such as RBF) will attempt to explain the variation due to the linear trend with a variance kernel that encodes long-range dependence. On the other hand, if this trend were removed by a base model like an NN, the residuals would have a very different structure (perhaps they would be stationary), and in this case, the GP would fit the data with a very different covariance kernel. 
Unfortunately, it does not seem like the formalism here can express a notion of misspecification at all. In the theory, it is assumed that the GP will only model the portion of the labels y_i for which it is property specified (in this case, f(.)). This generally does not occur in practice, as in the example above. It might be possible for this to apply in some circumstances, but the authors give no conditions (e.g., that the process y_i be stationary). Based on this assumption, the authors assert that the fitted GP to f(.) and r_f(.) will have the same covariance kernel parameters up to some proportionality constant \alpha. Much of their theoretical argument depends on this proportionality. But this proportionality cannot apply in general, and again, no conditions are given for when we might expect this to hold.
It would be far more compelling if the authors proposed the very standard approach to modeling data via covariance kernels, where one first models non-stationary portions of the data with a base model, then models the correlation in the residuals with something like a GP. This is the bread-and-butter approach in, say, timeseries analysis (see, e.g., the Shumway and Stoffer textbook https://www.stat.pitt.edu/stoffer/tsa4/tsa4.htm), and the approach in this paper could be framed similarly.
## Demos I Wish I Had Seen
I wish the authors had presented some demonstrations of what the GP does to the fitted values of an NN. Giving a demonstration of how the output kernel modifies predicted values, for example, would give some nice intuition the value added by this portion. I suspect that this step essentially performs something like Platt scaling, but for continuous outcomes, by shrinking predictions together so that they better match the overall distribution of observed labels. Perhaps the mechanism is different. At any rate, it would be useful to understand where the information gain is coming from, and this would be far better expressed concretely in terms of a toy data example than the theoretical arguments that are given.
## Coverage Experiments
I wish the coverage experiments evaluating predictive intervals were included in the main text. As far as uncertainty quantification evaluations go, coverage is one of the few assessments that does not rely on the model itself (unlike NPLD, which uses the model’s own log-likelihood), and can be phrased as a concrete performance guarantee.
Here, the goal for predictive intervals is to cover the true prediction value _at least as often_ as the nominal rate (95% intervals should cover the truth _at least_ 95% of the time), not merely that coverage be “close” to the nominal rate. This asymmetric evaluation gives you a concrete guarantee that the uncertainty estimate is conservative. The coverage experiments show that this method quite systematically under-covers compared to the end-to-end SVGP method, which generally satisfies this coverage property. I think this is important information to include about the model, and generally I think this behavior results from the fact that uncertainty is not propagated from the NN fit. This should be presented clearly in the main text.