Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust and efficient measures of uncertainty are crucial. While it is possible to train regression networks to output the parameters of a probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying confidence of its predictions. In this paper, we propose a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target. We accomplish this by placing evidential priors over our original Gaussian likelihood function and training our NN to infer the hyperparameters of our evidential distribution. We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output. Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters. We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.
This paper proposed deep evidential regression, a method for training neural networks to not only estimate the output but also the associated evidence in support of that output. The main idea follows the evidential deep learning work proposed in (Sensoy et al., 2018) extending it from the classification regime to the regression regime, by placing evidential priors over the Gaussian likelihood function and performing the type-II maximum likelihood estimation similar to the empirical Bayes method [1,2]. The authors demonstrated that the both the epistemic and aleatoric uncertainties could be estimated in one forward pass under the proposed framework without resorting to multiple passes and showed favorable uncertainty comparing to existing methods. Robustness against out of distribution and adversarially perturbed data is illustrated as well.
On the technical side, the novelty is incremental. The extension from the classification regime to the regression regime, from the conjugate Dirichet prior to the conjugate Normal-Inverse-Gamma prior, is quite straightforward. Besides, the presentation of the paper could be largely improved. It is not easy to follow the derivation in Section 3. The discussion of concepts and problem definitions look fragmented and incoherent. Even though the presentation largely follows (Sensoy et al., 2018) and uses terms from theory of evidence, the derivation actually is more aligned with the prior network [3] under the Bayesian framework which is missing from the references. It is really confusing that the authors talked about the variational inference when conjugate prior is used, and it is unclear how the variational distributions are used in Section 3.2 or how the "I don't know" loss term relates to the KL-divergence between the variational distribution and the prior in Section 3.3. This term was manually added as additional regularization to "prefer the evidence to shrink to zero for a sample if it cannot be correctly classified" in (Sensoy et al., 2018), and a different regularization was used to encourage distributional uncertainty in [3]. I hope that the authors could spend more efforts clarifying their ideas, especially the derivations in Section 3.2 and 3.3.
On the other hand, there is no referring to the input x in the entire derivation and problem formulation in Section 3. It took me a while to realize that the formulation in (4) actually defines the generation for a particular input, not for all the inputs. That is, the model is trying to model heteroscedastic uncertainty, not the homoscedastic counterpart. It could be better to call out the dependence on the input explicitly. 
On the quantitative side, the baseline models considered in Section 4 are mainly concerned with epistemic uncertainty estimation. So it would be good to explicitly discuss which uncertainty estimation was compared with. This work estimates both aleatoric and epistemic uncertainties, so a better comparison is to models that estimate both quantities (Kendall & Gal, 2017)[4] which has been shown to give better output estimation comparing to epistemic uncertainty estimation only (Kendall & Gal, 2017).
Other comments:
- What is the \pi in equation (8)?
- The "I don't know" loss introduced in Section 3.3 used L-p norm. What is the originality of the L-p norm here? In practice, which p value should be used? In the experiments, which p value was used?
- The RMSE results of the depth estimation presented in Table 2 are orders of magnitude smaller than those from existing work, for example Table 2(b) in (Kendall & Gal, 2017). Was a different RMSE computation used in this work?
- From the caption in Table 2, it seems that only 5 samples were used in MC-dropout, which is considerably smaller than those used in existing work (Kendall & Gal, 2017).
[1] D.J.C. MacKay. Hyperparameters: optimize, or integrate out? Maximum Entropy and Bayesian Methods, Springer 1996.
[2] B. Efron. Two modeling strategies for empirical Bayes estimation. Statistical Science, 2014.
[3] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. NeurIPS 2018.
[4] Y. Kwon, J.-H. Won, B.J. Kim, and M.C. Paik. Uncertainty quantification using Bayesian neural networks in classification: application to ischemic stroke lesion segmentation. MIDL 2018.