Determining the source of uncertainties in the predictions of AI systems are important. It allows the users to act in an informative manner to improve the safety of such systems, applied to the real-world sensitive applications. Predictive uncertainties can originate from the uncertainty in model parameters, data uncertainty or due to distributional mismatch between training and test examples. While recently, significant progress has been made to improve the predictive uncertainty estimation of deep learning models, most of these approaches either conflate the distributional uncertainty with model uncertainty or data uncertainty. In contrast, the Dirichlet Prior Network (DPN) can model distributional uncertainty distinctly by parameterizing a prior Dirichlet over the predictive categorical distributions. However, their complex loss function by explicitly incorporating KL divergence between Dirichlet distributions often makes the error surface ill-suited to optimize for challenging datasets with multiple classes. In this paper, we present an improved DPN framework by proposing a novel loss function using the standard cross-entropy loss along with a regularization term to control the sharpness of the output Dirichlet distributions from the network. Our proposed loss function aims to improve the training efficiency of the DPN framework for challenging classification tasks with a large number of classes. In our experiments using synthetic and real datasets, we demonstrate that our DPN models can distinguish the distributional uncertainty from other uncertainty types. Our proposed approach significantly improves DPN frameworks and outperform the existing OOD detectors on CIFAR-10 and CIFAR-100 dataset while also being able to recognize distributional uncertainty distinctly.
This work studies the predictive uncertainty issue of deep learning models. In particular, this work focuses on the distributional uncertainty which is caused by distributional mismatch between training and test examples. The proposed method is developed based on the existing work called Dirichlet Prior Network (DPN). It aims to address the issue of DPN that its loss function is complicated and makes the optimization difficult. Instead, this paper proposes a new loss function for DPN, which consists of the commonly used cross-entropy loss term and a regularization term. Two loss functions are respectively defined over in-domain training examples and out-of-distribution (OOD) training examples. The final objective function is a weighted combination of the two loss functions. Experimental study is conducted on one synthetic dataset and two image datasets (CIFAR-10 and CIFAR-100) to demonstrate the properties of the proposed method and compare its performance with the relevant ones in the literature. The issue researched in this work is of significance because understanding the predictive uncertainty of a deep learning model has its both theoretical and practical value. The motivation, research issues and the proposed method are overall clearly presented. 
The current recommendation is Weak Reject because the experimental study is not convincing or comprehensive enough. 
1.	Although the goal of this work is to deal with the inefficiency issue of the objective function of existing DPN with the newly proposed one, this experimental study does not seem to conduct sufficient experiments to demonstrate the advantages (say, in terms of training efficiency & the capability in making the network scalable for more challenging dataset) of the proposed objective function over the existing one; 
2.	Table 1 compares the proposed method with ODIN. However, as indicated in this work, ODIN is trained with in-domain examples only. Is this comparison fair? Actually, ODIN's setting seems to be more practical and more challenging than the setting used by the propose methods. 
3.	The evaluation criteria shall be better explained at the beginning of the experiment, especially how they can be collectively used to verify that the proposed method can better distinguish distributional uncertainty from other uncertainty types.
4.	In addition, the experimental study can be clearer on the training and test splits. How many samples from CIFAR-10 and CIFAR-100 are used for training and test purpose, respectively? Also, since training examples are from CIFAR-10 and CIFAR-100 and the test examples are also from these two datasets, does this contradict with the motivation of “distributional mismatch between training and test examples” mentioned in the abstract?  
5.	The experimental study can have more comparison on challenging datasets with more classes since it is indicated that DPN has difficulty in dealing with a large number of classes. 
Minor:
1. Please define the \hat\theta in Eq.(5). Also, is the dirac delta estimation a good enough approximation here?
2. The \lambda_{out} < \lambda_{in} in Eq.(11) needs to be better explained. In particular, are the first terms in Eq.(10) and Eq.(11) comparable in terms of magnitude? Otherwise,  \lambda_{out} < \lambda_{in} may not make sense.
3. The novelty and significance of fine-tuning the proposed model with noisy OOD training images can be better justified.