Intuitively, unfamiliarity should lead to lack of confidence. In reality, current algorithms often make highly confident yet wrong predictions when faced with unexpected test samples from an unknown distribution different from training. Unlike domain adaptation methods, we cannot gather an "unexpected dataset" prior to test, and unlike novelty detection methods, a best-effort original task prediction is still expected. We compare a number of methods from related fields such as calibration and epistemic uncertainty modeling, as well as two proposed methods that reduce overconfident errors of samples from an unknown novel distribution without drastically increasing evaluation time: (1) G-distillation, training an ensemble of classifiers and then distill into a single model using both labeled and unlabeled examples, or (2) NCR, reducing prediction confidence based on its novelty detection score. Experimentally, we investigate the overconfidence problem and evaluate our solution by creating "familiar" and "novel" test splits, where "familiar" are identically distributed with training and "novel" are not. We discover that calibrating using temperature scaling on familiar data is the best single-model method for improving novel confidence, followed by our proposed methods. In addition, some methods' NLL performance are roughly equivalent to a regularly trained model with certain degree of smoothing. Calibrating can also reduce confident errors, for example, in gender recognition by 95% on demographic groups different from the training data.
This paper introduces two methods of adjusting the overconfidence error for predictions on novel data. The ensemble distillation approach is to penalize the distillation loss on a potentially unlabeled general dataset.  The second approach (NCR) detects the novelty first and reweigh the prediction based on the familiarity to training data. 
*stationarity*
From the statistical perspective, the overconfidence of extrapolation can kick in from two sources: a)the epistemic uncertainty.  The point estimation of softmax ignores the uncertainty of prediction at all.  A full Bayesian approach will remedy this though computationally impractical.  b) the generative distribution p(y|x) might not be identical on training data and test data.  To see the difference, if the training sample size goes to infinity, the uncertainty in a) will go to zero, but b) may still exist.  Section 3 assumes the invariant p(y|x) in novel data.  But theoretically, both methods do not require such invariance?
Slightly related here, there can be novel data for classification, and in principle, there can also be novel data for novelty detection?  That will make NCR fail.
*why the distillation helps uncertainty adjustment*
I am not convinced how the g-distillation works for this task.  In the extreme case if the ensemble model itself is totally wrong for novel data and the unlabeled general data used in training, how can I learn any extra uncertainty information from that noise? To be fair, when the temperature goes high enough, the ensemble will make uniform prediction and then the distillation loss is merely a loss function that enforces uniformity.  If I replace the ensemble softmax by a uniform prior for unlabeled general data, do I achieve the same effect?  That is essentially the same regularization as method 2, except g-distillation is on logit scale.  
*robustness-accuracy tradeoff*
The experiments do not reveal too much robustness-efficiency conflict, as the new methods still perform good enough on familiar dataset. Indeed they can be even better than the baseline in E99 loss. Does it suggest the over-confidence is even a concern for familiar data/ iid data?
In general, the paper is well-written and well-motivated. It would be more interesting to make some theoretical explanation why/when this simple approach works.   I would recommend a weak accept at this point.