What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
There are two major types of uncertainty one can model. Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model -uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling epistemic vs. aleatoric uncertainty in Bayesian deep learning models for vision tasks. For this we present a Bayesian deep learning framework combining input-dependent aleatoric uncertainty together with epistemic uncertainty. We study models under the framework with per-pixel semantic segmentation and depth regression tasks. Further, our explicit uncertainty formulation leads to new loss functions for these tasks, which can be interpreted as learned attenuation. This makes the loss more robust to noisy data, also giving new state-of-the-art results on segmentation and depth regression benchmarks.
I have read the other reviews and the rebuttal.
Additional comment: The authors asked what I mean by 'numerical uncertainty'. I mean what is called 'algorithmic uncertainty' in this Wikipedia article: https://en.wikipedia.org/wiki/Uncertainty_quantification . Deep learning models are very high dimensional optimisation problems and the optimisers does not converge in almost all cases. Hence, this numerical error, i.e. the difference between the true minimum and the output of the optimiser, is a very significant source of error, which we are uncertain about (because we do not know the solution). Additionally, even rounding errors can add to the uncertainty of deep learning, as is e.g. discussed in this paper: http://proceedings.mlr.press/v37/gupta15.pdf . 
Hence, I think that the question in the title "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?" is not really answered. Although the paper is apart from this very good, this limits my rating at 'good paper, accept'. In my opinion, the authors should make clearer which kinds of uncertainty they consider (and which they ignore) or why they believe that the uncertainties considered are the dominant ones (which I find completely unclear).
% Brief Summary
The paper is concerned with evaluating which types of uncertainty are needed for computer vision. The authors restrict their analysis to aleatoric and epistemic uncertainty (leaving out numerical uncertainty). Aleatoric uncertainty includes the uncertainty from statistical noise in data. Epistemic uncertainty is usually another term for ignorance, i.e. things one could in theory know but doesn't know in practice. In this paper however, the authors use epistemic uncertainty as a synonym for model uncertainty (or structural uncertainty, i.e. ignorance over the true physical model which created the data). The paper raises the question how these two sources of uncertainty can be jointly quantified, and when which source of uncertainty is dominant.  In particular, this is discussed in the setting of computer vision with deep learning. While classically it was not possible to capture uncertainty in deep learning, Bayesian neural networks (BNN) have made it possible to capture at least epistemic uncertainty. The authors aim to jointly treat epistemic and aleatoric uncertainty in BNNs.
To expand the existing literature, the authors propose to include aleatoric uncertainty over pixel output sigma by fixing a Laplace likelihood which contains sigma, which is to be minimised with respect to the parameter vector theta. They also observe that this inclusion of aleatoric uncertainty can be interpreted as loss attenuation.
This novel method is then applied to pixel-wise depth regression and semantic segmentation. For semantic segmentation performance, a notable increase in performance is achieved by modelling both aleatoric and epistemic uncertainty.
In Chapter 5, an analysis of the effects of epistemic and aleatoric uncertainty follows. Notably, the authors experimentally validate the well-known claims that aleatoric uncertainty cannot be reduced with more data, and that only epistemic uncertainty increases for out-of-data inputs. The paper concludes by claiming that aleatoric uncertainty is particularly important for large data situations and real-time applications, while epistemic uncertainty is particularly important for safety-critical applications and small datasets. 
% Comment
I am skeptical about the use of the classifications of uncertainties which can arise in this setting in this paper. While I agree with the definition of aleatoric uncertainty, I do not agree that epistemic uncertainty is the same as model uncertainty. For me, model uncertainty is the ignorance about the true physical model which creates the data. Epistemic uncertainty is supposed to capture everything which is unknown, but is deterministic and could in theory be known. So, I think that the numerical uncertainty is missing here. In large dimensional optimisation problems (as they arise in deep learning), it is completely unclear whether the optimisation algorithm has converged. I would even expect that the numerical uncertainty could be a dominant source of error, in particular when the computation has too be performed online, e.g. when the self-driving car learns while it's driving. Hence, I would like to ask the authors on their thoughts on this.
The second issue I want to arise is the following: The authors convincingly argue that epistemic uncertainty is reduced when more data is available. In statistics, people talk about the statistical consistency in model. A model is statistically consistent when its output is a Dirac measure on the ground truth, in the limit of infinite data. It would be very interesting to see whether statistical consistency results for deep learning (if they exist) can be connected to the paper's pitch on the relative unimportance of epistemic uncertainty in big data situations.
Since I think (as explained above) that the classifications of uncertainties is missing the numerical uncertainty of the optimiser, I would argue that the paper does not completely live up to comprehensively answer the question asked in the title. While it is fine, that this is not part in the paper; I think it should be discussed in the introduction. Since, the paper is---a part from this---good, I vote for 'accept'.