We present local ensembles, a method for detecting extrapolation at test time in a pre-trained model. We focus on underdetermination as a key component of extrapolation: we aim to detect when many possible predictions are consistent with the training data and model class. Our method uses local second-order information to approximate the variance of predictions across an ensemble of models from the same class. We compute this approximation by estimating the norm of the component of a test point's gradient that aligns with the low-curvature directions of the Hessian, and provide a tractable method for estimating this quantity. Experimentally, we show that our method is capable of detecting when a pretrained model is extrapolating on test data, with applications to out-of-distribution detection, detecting spurious correlates, and active learning.
# Summary of contribution
- The paper provides a novel fast and simple approximation of second-order local parameter sensitivity of neural networks, to estimate a form of uncertainty wrt to a test sample, which is further used and tested as a novelty detector. 
- The method analyzes the most significant eigenvector/eigenvalues of the Hessian (of training loss), and use the compliment of their span to get directions of local perturbations to network parameters that affect training loss little ("ensemble subspace"). The novelty score is then based on how much the prediction is influenced by these perturbations.
- The idea of estimating "ensemble subspace" is interesting and computationally effective. Compared to other recent methods that also use second-order gradients for uncertainty, this paper is more generally applicable, and can be faster at test time. The paper demonstrates good performance on both simulated data and real data (CelebA faces with CNN, etc.).
# Decision TL;DR
I am giving a weak reject. The paper is strong in its idea, formulation, and theory, but is too similar to recent related works which this paper is reluctant to compare to (either in theory, efficiency, or performance). Since the contribution of the paper lies in the efficient approximation of local ensemble methods, readers cannot gauge how beneficial the contribution is compared to other approximations of ensembles.
# Pros
- Novel way to estimate a local neighborhood that affects training loss little (Note: not an expert in this line of research, not sure if it is completely novel) by estimating significant eigenvectors and using their compliment space.
- The paper is well-written, and relatively easy to understand, despite a few hard-to-follow spots
- Widely applicable post-hoc to any trained neural networks, and potentially faster training than ensembles / Bayesian approximated ensembles
- (Theoretical) stability compared to full Hessian inversion
# Cons
Motivation wrt other papers unclear, and a lack of comparison.
- Two of the cited papers (Gal & Ghahramani, 2016; Blundell et al., 2015) both work on local ensembles. The former uses MC dropout, the latter estimates a diagonal covariance of a Gaussian distribution of network parameters. These methods are not mentioned in the motivation or related work, which makes it hard to say this paper is well-placed in the literature.
- The reason that these methods are not compared to is insufficient. The paper only argues that they are not "post-hoc" methods. It is very unclear why in any circumstance (or use case) a post-hoc estimation of local ensemble must be (or is preferred to be) used, rather than having network parameters and local neighborhood jointly estimated. If it is for efficiency reasons, the paper does not provide any experimental comparison of the efficiency. Also, it is hard to argue that the "post-hoc" nature of this paper makes it so different from the two prior work. For the first prior work, the only time it is not post-hoc is when the original network does not have any dropout layer, and that circumstance is not very common. For the second prior work, one can easily make it post-hoc by training the network first, and estimate the diagonal covariance post-hoc using their loss. 
- The advantage of this paper is that it is more efficient and stable than alternatives, but only  the full hessian inversion is discussed. In particular, it may be necessary to discuss this paper's efficiency against MC-dropout (Gal & Ghahramani, 2016). This method can be done in mini-batches, while the proposed method has to run forward and back-propagation separately for each sample to get gθ*(x'), and it is unclear how well that scales.
Efficiency analysis lacking
- As discussed above, the proposed method seems to need to back-prop for each test sample separately without using a batch. How much this affects test efficiency is unclear.
Experimental comparison with similar methods missing.
- The paper would benefit from comparing to the two cited papers (among which MC-dropout is so easy to implement) as well as a full hessian estimation (for toy datasets at least).
- The paper poses itself as an efficient alternative, so it would be essential to gauge experimentally how fast each method is.
Others. (not crucial issues) 
- The performance of the paper's main method (LE w/ predictions) underperforms in Table 2, and a variant had to be proposed to make up for the performance drop. This suggests instability of the proposed method wrt new datasets.
- The claim in contribution "We identify underdetermination as a key factor in the unreliability of predictions" is not verified.
- Inability to scale up to large networks with larger m needed, compared to real ensemble methods or Bayesian networks with Gaussian distributions.
# Room for improvement (decreasing order of importance)
- Improve placement in the literature by discussing when this paper is more useful than prior work (Gal & Ghahramani, 2016; Blundell et al., 2015).
- Detailed theoretical or experimental analysis of efficiency against alternative approximations of ensembles.
- Performance comparison to ensemble and local Bayesian methods.
# Editorial issues
- Figure 3(c) x axis meaning unclear
- Figure 4 not mentioned in text, and unclear which experiment this refers to 
- Table 2 experiment's loss gradient version is not explained.
#############################################################
POST REBUTTAL
#############################################################
TL;DR: The rebuttal addresses some but not all of my concerns. The MC-dropout comparison especially shows the difference between some approximate ensemble methods and ensembles that specifically changes the loss little (this paper). In the end, it is a good paper in terms of theory, although the experiments is lacking in crucial places (no comparison with many existing papers that do attempt to invert hessian) and lack of analysis of claimed efficiency, which I can only hope don’t turn out to be a big deal. I am increasing the score, to marginally above borderline, but please consider the following feedback for the camera-ready version.
> MC Dropout and Bayes by Backprop effectively measure different types of uncertainty from what our method targets, and so they should not be considered competing methods.
I disagree; by the same logic we can never compare SVM with random forest because they are so different.
> However, as we discussed in 4.1, small eigenvalues make turning this representation into a proper posterior distribution difficult, because these eigenvalues need to be inverted to obtain a covariance matrix, but these inverted eigenvalues can be numerically infinite.
I agree; but the main difference is the full hessian is hard to implement while there are papers that have implemented the diagonal hessian. It would benefit the paper to compare to that and prove the issue of instability on top of arguing theoretically.
> First, we clarify that our method can in fact be performed using mini-batches of test points (both the forward and backward passes).
As far as I know, with mini-batches, the gradients averaged over all samples in the mini-batch are computed for each network parameter. But this method needs the gradient wrt each sample separately.
If the authors have some implementation trick that allows computing network parameters’ gradients wrt each sample in the mini-batch separately, please include in the implementation details. Otherwise, please mention this drawback in any “fast” claim.
I will change the score upwards due to the informative rebuttal. Please include much of the discussion in the paper or appendix.