Uncertainty estimation (UE) of model predictions is a crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, out-of-distribution detection, etc. Most of the works on modeling the uncertainty of deep neural networks evaluate these methods on image classification tasks. Little attention has been paid to UE in natural language processing. To fill this gap, we perform a vast empirical investigation of state-of-the-art UE methods for Transformer models on misclassification detection in named entity recognition and text classification tasks and propose two computationally efficient modifications, one of which approaches or even outperforms computationally intensive methods 1 . Machine learning methods are naturally prone to errors as they typically have to deal with ambiguous and incomplete data during both training and inference. Unreliable predictions hinder the application of these methods in domains, where the price of mistakes is very high, such as clinical medicine. Even in more error-tolerant domains and tasks, such as intent recognition in general-purpose chatbots, one would like to achieve a better trade-off between expressiveness of a model and its computational performance during inference. Since mistakes are inevitable, it is crucial to understand whether model predictions can be trusted or not and abstain from unreliable decisions. Uncertainty estimation (UE) of model predictions aims to solve this task. Ideally, uncertain instances should correspond to erroneous 1 The code for experiments is available online at https://github.com/AIRI-Institute/ uncertainty_transformers  Equal contribution, corresponding authors objects and help in misclassification detection . Besides misclassification detection, UE is a crucial component for active learning (Settles, 2009), adversarial attack detection (Lee et al., 2018), detection of out-of-distribution (OOD) instances (Van Amersfoort et al., 2020), etc. Some classical machine learning models, e.g. Gaussian processes (Rasmussen, 2003), have built-in UE capabilities. Modern deep neural networks (DNNs) usually take advantage of a softmax layer, which output can be considered as a prediction probability and be used for UE. However, the softmax probabilities are usually unreliable and produce overconfident predictions (Guo et al., 2017). Some previously proposed techniques such as deep ensemble (Lakshminarayanan et al., 2017) are known for producing good UE scores but require a large additional memory footprint for storing several versions of weights and multiply an amount of computation for conducting several forward passes. Reliable UE of DNN predictions that does not introduce high computational overhead is an open research question (Van Amersfoort et al., 2020). In this work, we investigate methods for UE of DNNs based on the Transformer architecture (Vaswani et al., 2017) in misclassification detection. We consider two of the most common NLP tasks: text classification and named entity recognition (NER). The latter has been overlooked in the literature on UE. To our knowledge, this work is the first to consider UE for NER. We propose two novel computationally cheap methods for UE of Transformer predictions. The first method is the modification of the Monte Carlo dropout with determinantal point process sampling of dropout masks (Shelmanov et al., 2021). We introduce an additional step for making masks more diverse, which helps to 8237 achieve substantial improvements and approach the performance of computationally-intensive methods on NER. The second method leverages Mahalanobis distance (Lee et al., 2018) but also adds a spectral normalization of the weight matrix in the classification layer (Liu et al., 2020). This method achieves the best results on most of the datasets and even outperforms computationally-intensive methods. We also investigate recently proposed regularization techniques in combination with other UE methods. The contributions of this paper are the following:  We propose two novel computationally cheap modifications of UE methods for Transformer models. The method based on Mahalanobis distance with spectral normalization approaches or even outperforms strong computationally intensive counterparts. This work is the first to investigate UE methods on the NER task. We conduct an extensive empirical evaluation, in which we investigate recently proposed regularization techniques in combination with other UE methods. It is well known that reliable uncertainty scores can be obtained simply by constructing an ensemble of decorrelated neural networks ( deep ensemble ) (Lakshminarayanan et al., 2017). However, such a straightforward approach is coupled with substantial computational and memory overhead during training an ensemble, performing inference of all its components, and storing multiple versions of weights. This overhead is a serious obstacle to deploying ensemble-based uncertainty estimation methods in practice. Uncertainty estimation is a built-in capability of Bayesian neural networks (Blundell et al., 2015). However, such models have similar issues as ensembles and also require special training procedures. Recently, it was shown by Gal and Ghahramani (2016) that dropout, a well-known regularization technique, is formally equivalent to approximate variational inference in a deep Gaussian process if it is activated during prediction. This method, known as Monte Carlo (MC) dropout, uses the approximating variational distribution with Bernoulli variables related to network units. MC dropout does not impose any overhead during training, introduces no additional parameters, and thus does not require any additional memory. The main disadvantage of this method is that it usually requires many forward-pass samplings for approximating predictive posterior, which makes it also computationally expensive. Recently, many works have investigated the approximate Bayesian inference for neural networks using deterministic approaches: Lee et al. (2018); Liu et al. (2020); Van Amersfoort et al. (2020); Mukhoti et al. (2021); Shen et al. (2021), etc. These methods do not introduce notable overhead for inference, storing weights, and usually require compatible training time. However, most of the research in this area is accomplished for computer vision tasks. For text classification, a series of works investigates UE methods for the OOD detection task (Liu et al., 2020; Podolskiy et al., 2021; Zeng et al., 2021; Hu and Khan, 2021). In this work, we focus on a more challenging task  misclassification detection. While OOD detection requires to model only the epistemic uncertainty inherent to the model and caused by a lack of training data, misclassification detection also requires to model aleatoric uncertainty caused by noise and ambiguity in data (Mukhoti et al., 2021). We consider recently proposed methods in this area that are evaluated in text processing. Three recent works propose techniques for misclassification detection based on an additive regularization of a training loss function. Zhang et al. (2019) suggest adding a penalty that reduces the Euclidean distance between training instances of the same class and increases the distance between instances of different classes. He et al. (2020) suggest using two components in the loss function that reduce the difference between outputs from two versions of a model initialized with different weights. They also use mix-up (Thulasidasan et al., 2019) to generate additional training instance representations that help to capture aleatoric uncertainty, self-ensembling, MC dropout, and a distinctiveness score to measure the epistemic uncertainty. Xin et al. (2021) introduce a regularizer that penalizes overconfident instances with high loss. In another recent work, Shelmanov et al. (2021) propose to combine MC dropout with a Determinantal Point Process (DPP) to improve the diversity of predictions by considering the correlations between neurons and sampling the 8238 diverse neurons for activation in a dropout layer. In this work, we conduct a systematic empirical investigation of UE methods on NLP tasks. We evaluate combinations of methods that have not been tested before and propose two modifications, one of which achieves the best results among computationally cheap methods. The previous work focuses on text classification tasks, while this work is the first to investigate UE also for NER. In this section, we describe the baselines and propose novel uncertainty estimation techniques. Softmax Response (SR) (Geifman and El-Yaniv, 2017) is a trivial baseline for UE that uses the probabilities generated via the output softmax layer of the neural network. SR is based on the maximum probability p ( y | x ) over classes y = c  C . The smaller this probability is, the more uncertain model is: u SR ( x ) = 1  max c  C p ( y = c | x ) . Standard Monte Carlo Dropout (MC Dropout) Consider we have conducted T stochastic forward passes with activated dropout. In this work, we use the following ways to quantify uncertainty with methods based on MC dropout:  Sampled maximum probability (SMP) is: u SMP = 1  max c  C 1 TT (cid:88) t =1 p ct , (2) where p ct is the probability of the class c for the t -th stochastic forward pass. Probability variance (PV; Gal et al. (2017); Smith and Gal (2018)) is: u PV = 1 CC (cid:88) c =1 (cid:32) 1 TT (cid:88) t =1 ( p ct  p c ) 2 (cid:33) , (3) where p c = 1 T (cid:80) t p ct is the probability for a class c averaged across T stochastic forward passes. Bayesian active learning by disagreement (BALD; Houlsby et al. (2011)) is: u BALD =  C (cid:88) c =1 p c log p c + 1 T (cid:88) c,t p ct log p ct . The two former techniques are specifically designed for estimation of the epistemic (model) uncertainty arising from the lack of knowledge and ignore the aleatoric uncertainty related to ambiguity and noise in the data, while the latter method can be seen as a measure of total uncertainty (Malinin and Gales, 2018). Transformers contain multiple dropout layers (after the embedding layer, in each attention head, and before the last classification layer). It is shown in previous work that the standard MC dropout outperforms the baseline SR only when all dropout layers are activated in a model (Shelmanov et al., 2021). Therefore, we follow this setting for experiments in this work. We note that due to activating all dropout layers, multiple stochastic predictions are required for the whole network, which introduces a large computational overhead. Similar UE scores are used in deep ensemble (Lakshminarayanan et al., 2017), where instead of multiple stochastic predictions we train and infer several model versions with different sets of weights. Diverse Determinantal Point Process Monte Carlo Dropout (DDPP MC dropout) (Ours) Determinantal point processes (DPPs; Kulesza and Taskar (2012)) are used for sampling a subset of diverse objects from a given set. Recently, Shelmanov et al. (2021) have combined the MC dropout with a determinantal point process (DPP) for sampling neurons in a dropout layer and demonstrated that using stochasticity in the last dropout layer (in a classification head of Transformer) only is enough to improve upon SR in misclassification detection. This method is less computationally expensive than the standard MC dropout since it requires multiple stochastic predictions only for the top classification layer of the network with a small number of parameters, while all other layers are inferred only once. Consider the similarity matrix C h between neurons of the h -th hidden layer (in particular, we use a correlation matrix between output values of neurons on the training set). Then one can construct the DPP-based dropout masks M DPPh using C h as a likelihood kernel for the DPP: M DPPh  DP P ( C h ) . That gives the following probability to select a set S of activations on the layer h : P (cid:104) M DPPh = S (cid:105) = det( C Sh ) det( C h + I ) , (5) 8239 where C Sh is the square submatrix of C h obtained by keeping only rows and columns indexed by the sample S . In this work, we improve this method by increasing the diversity of the sampled DPP masks. After multiple dropout masks are pre-generated via DPP in the inference step as in the original DPP MC dropout, we make an additional step, in which we select a diverse set of masks from this pre-generated pool using one of two strategies:  DDPP (+DPP) : We sample a set of diverse masks that activate different sets of neurons. For this purpose, we apply DPP sampling again to the pool of pre-generated masks. As a similarity kernel in this step, we use an RBF-similarity matrix of mask vectors. DDPP (+OOD) : We sample a set of masks that generate diverse predictions. For this purpose, we select the masks that yield the highest PV scores on the given OOD dataset. After a new set of T masks is selected, we use them as in the standard MC dropout to obtain stochastic predictions. Increasing the diversity of masks in the proposed modification is motivated by the finding of Jain et al. (2020) that improving the diversity of elements in an ensemble leads to better uncertainty estimates. We note that in masks generated with DPP, usually, less than 50% of neurons are activated, which makes predictions poorly calibrated. To mitigate this problem, for each constructed mask, we perform a temperature-scaling calibration (Guo et al., 2017) using a held-out dataset. Spectral-normalized Neural Gaussian Process (SNGP) Liu et al. (2020) suggest replacing the typical dense output layer of a network with a layer that implements a Gaussian process with an RBF kernel, whose posterior variance at a given instance is characterized by its L 2 distance from the training data in the hidden vector space constructed by underlying layers of a network. The authors propose an approximation based on random Fourier feature expansion, which enables end-to-end training and makes the inference feasible. However, this method requires hidden representations to be distance-preserving in order to make it work. While the distance between instances in the hidden space does not always have a meaningful correspondence to the distance in the input space, authors prove that to keep hidden representations distance-preserving, the transformation should satisfy the bi-Lipschitz condition. For ResNets (He et al., 2016), this requirement is satisfied if weight matrices for the nonlinear residual blocks have a spectral norm (i.e., the largest singular value) bounded from above by a constant. Therefore, to enforce the aforementioned Lipschitz constraint, they apply a spectral normalization (SN) on weight matrices. For Transformers, they normalize the matrix of the penultimate classification layer only. Mahalanobis Distance (MD) Mahalanobis distance is a generalisation of the Euclidean distance, which takes into account the spreading of instances in the training set along various directions in a feature space. Lee et al. (2018) suggest estimating uncertainty by measuring the Mahalanobis distance between a test instance and the closest class-conditional Gaussian distribution: u MD = min c  C ( h i   c ) T   1 ( h i   c ) , (6) where h i is a hidden representation of a i -th instance,  c is a centroid of a class c , and  is a covariance matrix for hidden representations of training instances. Recently, the Mahalanobis distance has been adopted for out-of-distribution detection with Transformer networks by Podolskiy et al. (2021). Mahalanobis Distance with Spectral-normalized Network (MD SN) (Ours) Since the UE method based on the Mahalanobis distance utilizes the idea of a proximity of a tested instance hidden representation to the training distribution, we expect this method to benefit from distance-preserving representations. Therefore, we propose the modification of the method of Lee et al. (2018) and Podolskiy et al. (2021) that enforces the bi-Lipschitz constraints on transformation implemented by the network. We perform spectral normalization of the weight matrix of the linear layer in the classification head of Transformer as it is suggested in SNGP (Liu et al., 2020). At each training step, a spectral norm  is estimated using the power iteration method  = (cid:107) W (cid:107) 2 , and a normalized weight matrix is obtained:  W = W . At the inference step, hidden representations are calculated using the normalized 8240 matrix  h ( x ) =  W x + b and are used for computing the Mahalanobis distance. Additive regularization is another approach to improving UE of neural network predictions. Usually, the training loss combines the original task-specific loss L task (e.g. cross-entropy) and a regularization component L reg that facilitates producing better calibrated UEs: L = L task + L reg , (7) where  is a hyperparameter that controls the regularization strength. The positive side of such techniques is that, besides SR, they can be used to improve other methods like MC dropout and deterministic methods. The drawback is that regularization affects the training procedure and can decrease the model quality. Confident Error Regularizer (CER) Xin et al. (2021) propose a regularizer that adds a penalty for an instance with a bigger loss than other instances and, at the same time, bigger confidence: L reg = k (cid:88) i,j =1  i,j 1 [ e i > e j ] , (8)  i,j = max { 0 , max c p ci  max c p cj } 2 , (9) where k is the number of instances in a batch and e i is an error of the i -th instance: e i is 1 if the prediction of the classifier matches the true label, and e i is 0 otherwise. The authors evaluate this type of regularization only in conjunction with the SR baseline. Metric Regularizer Zhang et al. (2019) propose a regularizer that aims to shorten the intra-class distance and enlarge the inter-class distance: L reg = C (cid:88) c =1 (cid:110) L intra ( c )+  (cid:88) k (cid:54) = c L inter ( c, k ) (cid:111) , (10) L intra ( c ) = 2 | S c | 2  | S c | (cid:88) i,j  S c ,i<j D ( h i , h j ) , (11) L inter ( c,k )= 1 | S c || S k | (cid:88) i  S c ,j  S k [   D ( h i ,h j )] + , (12) D ( r i , r j ) = 1 d || h i  h j || 22 , (13) where h i is a feature representation of an instance i from a penultimate layer of a model with a dimension d , S c is the set of instances from class c , | S c | is the number of elements in S c ,  and  are positive hyperparameters, [ x ] + = max (0 , x ) . (16) 8241 Method Reg.Type UEScore MRPC SST-2 CoLA CoNLL-2003(tokenlevel) CoNLL-2003(seq. level) RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  MC PV 13.97  1.16 1.68  0.09 12.90  1.92 0.82  0.11 44.35  4.90 2.06  0.16 6.32  1.66 0.10  0.02 16.05  3.78 1.93  0.43 MC BALD 14.21  1.04 1.69  0.09 12.98  1.87 0.82  0.10 45.06  4.90 2.08  0.17 6.44  1.86 0.10  0.02 16.28  4.00 1.96  0.45 MC SMP 14.38  2.07 1.76  0.19 14.00  2.20 0.91  0.15 42.95  5.98 2.01  0.15 6.04  1.03 0.09  0.02 15.79  3.34 1.80  0.35 MC CER PV 12.82  1.89 1.60  0.13 12.18  1.20 0.80  0.10 46.84  9.19 2.11  0.23 6.92  1.22 0.10  0.02 17.05  3.14 1.91  0.36 MC CER BALD 12.89  1.89 1.60  0.13 12.39  1.23 0.81  0.09 47.34  8.30 2.14  0.24 7.16  1.15 0.11  0.02 17.25  3.05 1.93  0.35 MC CER SMP 12.91  2.15 1.67  0.15 12.22  1.31 0.82  0.09 46.10  11.07 2.05  0.22 6.69  1.38 0.10  0.02 16.81  1.61 1.81  0.14 MC metric PV 14.21  1.95 1.73  0.23 12.28  1.77 0.80  0.11 42.35  0.69 2.04  0.07 6.69  0.89 0.10  0.01 17.17  1.90 1.93  0.31 MC metric BALD 14.55  2.31 1.73  0.23 12.08  1.79 0.79  0.10 43.76  0.55 2.08  0.07 6.91  1.02 0.10  0.01 17.47  1.85 1.98  0.30 MC metric SMP 13.39  1.19 1.72  0.20 13.55  1.65 0.90  0.14 40.88  1.25 2.01  0.09 6.30  0.98 0.10  0.01 16.81  1.40 1.80  0.23 DDPP(+DPP)(ours) PV 22.30  7.15 2.58  0.65 16.70  1.38 1.12  0.12 49.75  3.96 2.44  0.29 6.12  0.71 0.10  0.01 16.78  2.44 1.93  0.20 DDPP(+DPP)(ours) BALD 23.08  7.00 2.63  0.63 16.08  2.37 1.05  0.18 49.59  5.40 2.48  0.31 6.39  0.64 0.10  0.01 21.53  4.77 2.63  0.45 DDPP(+DPP)(ours) SMP 21.79  7.72 2.57  0.68 17.55  3.03 1.19  0.23 47.86  5.51 2.39  0.31 6.08  0.62 0.10  0.01 17.71  2.77 2.05  0.23 DDPP(+DPP)(ours) CER PV 15.12  2.27 2.03  0.24 13.56  1.37 0.91  0.14 54.51  8.80 2.58  0.22 6.98  0.98 0.11  0.02 19.44  1.15 2.13  0.17 DDPP(+DPP)(ours) CER BALD 15.94  3.77 2.07  0.36 14.87  2.22 0.96  0.13 55.11  7.42 2.61  0.31 7.90  1.95 0.12  0.01 26.20  6.41 3.11  0.56 DDPP(+DPP)(ours) CER SMP 14.75  1.43 2.02  0.16 14.47  1.63 0.99  0.11 54.01  9.79 2.55  0.18 6.91  1.13 0.11  0.02 20.66  1.53 2.31  0.08 DDPP(+DPP)(ours) metric PV 19.51  3.40 2.47  0.28 15.79  1.67 1.07  0.14 43.82  1.82 2.17  0.14 7.33  1.53 0.12  0.02 18.93  2.09 2.11  0.25 DDPP(+DPP)(ours) metric BALD 20.54  4.72 2.52  0.34 15.48  1.81 1.03  0.08 43.95  1.68 2.17  0.12 8.01  2.08 0.13  0.03 22.44  4.78 2.67  0.49 DDPP(+DPP)(ours) metric SMP 18.45  2.88 2.41  0.26 16.78  3.43 1.14  0.26 43.61  1.61 2.16  0.11 6.92  1.32 0.11  0.02 19.11  2.14 2.16  0.22 DDPP(+OOD)(ours) PV 22.73  7.45 2.65  0.59 19.05  2.95 1.29  0.23 51.11  12.03 2.37  0.34 6.32  0.72 0.10  0.01 16.75  2.31 1.94  0.21 DDPP(+OOD)(ours) BALD 23.85  8.39 2.69  0.58 18.27  3.05 1.22  0.23 52.59  12.08 2.42  0.34 6.59  0.69 0.11  0.01 20.56  3.09 2.50  0.26 DDPP(+OOD)(ours) SMP 22.31  7.80 2.60  0.65 19.86  3.83 1.36  0.29 50.14  9.73 2.32  0.30 6.09  0.67 0.10  0.01 17.76  2.75 2.06  0.23 DDPP(+OOD)(ours) CER PV 14.83  1.42 2.05  0.17 14.98  1.36 1.01  0.09 59.14  11.27 2.56  0.24 7.08  1.37 0.11  0.02 19.66  1.25 2.17  0.15 DDPP(+OOD)(ours) CER BALD 15.03  1.85 2.08  0.24 14.37  2.22 0.96  0.14 57.48  9.37 2.54  0.26 7.41  1.29 0.12  0.02 25.30  3.36 3.00  0.24 DDPP(+OOD)(ours) CER SMP 14.34  1.15 1.99  0.16 15.88  1.96 1.08  0.13 59.32  11.86 2.53  0.20 6.88  1.24 0.11  0.02 21.06  1.96 2.35  0.14 DDPP(+OOD)(ours) metric PV 19.03  3.97 2.41  0.34 17.75  5.20 1.10  0.17 48.54  11.38 2.23  0.24 6.92  1.32 0.11  0.02 18.36  1.90 2.05  0.26 DDPP(+OOD)(ours) metric BALD 19.33  4.78 2.41  0.40 16.71  7.13 1.02  0.20 49.31  11.87 2.24  0.25 7.21  1.49 0.11  0.02 21.35  4.47 2.54  0.45 DDPP(+OOD)(ours) metric SMP 18.55  3.06 2.42  0.27 17.08  3.78 1.14  0.26 43.67  1.77 2.15  0.11 6.71  1.18 0.10  0.02 19.01  2.30 2.16  0.25 SR CER MP 14.62  1.62 2.02  0.19 14.56  2.14 1.00  0.14 56.97  9.69 2.53  0.15 6.84  1.41 0.11  0.02 21.31  1.63 2.49  0.25 SR metric MP 18.39  2.94 2.40  0.27 16.90  3.12 1.16  0.24 44.54  2.11 2.22  0.15 6.51  1.07 0.10  0.02 20.32  1.68 2.32  0.23 SR(baseline) MP 22.32  8.08 2.58  0.65 17.93  3.84 1.22  0.28 49.48  3.71 2.35  0.25 6.08  0.62 0.10  0.01 18.81  3.35 2.21  0.29 Table 1: Results for methods based on MC dropout and regularization techniques (ELECTRA model). In the experiments, we train a model on a given dataset and perform inference on a separate test set to compute both predictions and UE scores u . We are interested in how the scores correlate with the mistakes  e of the model on the test set. For text classification, mistakes are computed in the following way:  e i = (cid:26) 1 , y i (cid:54) =  y i , 0 , y i =  y i , (14) where y i is a true label,  y i is a predicted label. For NER, we use two evaluation options: token-level and sequence-level. For the token-level evaluation, individual tokens are considered as separate instances as in the text classification. For the sequence-level evaluation, mistakes are computed in the following way:  e i = (cid:26) 1 ,  j  { 1 , . . . , n } , y ij (cid:54) =  y ij , 0 ,  j  { 1 , . . . , n } , y ij =  y ij , (15) where n is a sequence length, y ij is a true label,  y ij is a predicted label of a j -th token in a sequence. In the sequence-level evaluation, UE of a sequence is aggregated from UEs of tokens by taking maximum (for MD methods) or by summation (for others). El-Yaniv and Wiener (2010) suggest evaluating the quality of UE using the area under the risk coverage curve (RCC-AUC) . The risk coverage curve demonstrates the cumulative sum of loss due to misclassification (cumulative risk) depending on the uncertainty level used for rejection of predictions. The lower area under this curve indicates better quality of the UE method. Xin et al. (2021) propose a reversed pair proportion (RPP) metric. They note that instances with higher confidence should have a lower loss l . RPP measures how far the uncertainty estimator  u is to ideal, given the labeled dataset of size n : RP P = 1 n 2 n (cid:88) i,j =1 1 [ u ( x i ) >  u ( x j ) , l i <l j ] . This metric has an upper bound of 1; for convenience, the reported values are multiplied by 100. Similar to Xin et al. (2021), for both metrics, l is an indicator loss function. We conduct each experiment six times with different random seeds, obtaining the corresponding metric values, and report their mean and standard deviation. We also present the results using the accuracy rejection curve . This curve is drawn by varying the rejection uncertainty level (horizontal axis) and presenting the corresponding accuracy obtained when all rejected instances are labeled with an oracle (vertical axis). This emulates the work of a human expert in conjunction with a machine learning system. The higher the curve, the smaller amount of labor is needed to achieve a certain level of performance and the better is the UE method. A similar evaluation approach in a table form is used in (Zhang et al., 2019). A similar curve but without oracle labeling is used in (Lakshminarayanan et al., 2017; Filos et al., 2019). For experiments with text classification, we use three datasets from the GLUE benchmark (Wang et al., 2018) that were previously leveraged by Shelmanov et al. (2021) and Xin et al. (2021) for the same purpose: Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005), Corpus of Linguistic Acceptability (CoLA) (Warstadt et al., 2019), and Stanford Sentiment Treebank (SST-2) (Socher et al., 2013). Similar to (Shelmanov et al., 2021), we randomly subsample SST-2 to 10% to emulate a low-resource setting. The experiments with NER were performed on the widely-used CoNLL-2003 task (Tjong Kim Sang and De Meulder, 2003). For this dataset, we also subsample the training part to 10%. As an out-of-domain dataset for DDPP MC dropout, we use the IMDB binary sentiment classification dataset (Maas et al., 2011). We randomly select 5,000 instances from its test part and use them to select DPP-generated masks. For experiments, we use two modern Transformers: the pre-trained ELECTRA model (Clark et al., 2020) with 110 million parameters and DeBERTa (He et al., 2021) with 138 million parameters. They achieve higher performance on the GLUE benchmark in comparison with previous models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). The optimal hyperparameter values for each triple <Dataset, Regularization Type, Spectral Normalization Usage> are presented in Table 6 8242 Method Reg.Type UEScore MRPC SST-2 CoLA CoNLL-2003(tokenlevel) CoNLL-2003(seq. level) RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  MD MD 13.69  1.25 1.88  0.13 13.08  2.58 0.86  0.15 41.73  1.45 1.96  0.04 10.33  3.55 0.15  0.04 17.05  5.07 2.05  0.45 MD CER MD 13.61  1.82 1.87  0.22 14.10  2.69 0.96  0.16 42.50  2.65 2.00  0.07 6.82  0.90 0.10  0.01 16.92  2.51 1.87  0.23 MD metric MD 13.91  2.35 1.89  0.29 12.03  2.04 0.85  0.15 40.29  2.09 2.02  0.09 10.01  2.56 0.15  0.03 17.67  3.92 2.09  0.36 MDSN(ours) MD 13.44  1.28 1.85  0.20 11.77  1.33 0.83  0.08 40.07  3.62 1.95  0.16 7.21  1.34 0.11  0.02 17.29  3.58 2.01  0.37 MDSN(ours) CER MD 14.41  1.96 1.94  0.21 12.32  1.37 0.85  0.10 37.82  2.91 1.90  0.12 6.95  1.50 0.11  0.02 17.76  4.00 2.06  0.42 MDSN(ours) metric MD 12.04  1.33 1.56  0.12 12.05  1.42 0.84  0.07 39.37  2.00 1.97  0.15 6.90  1.21 0.11  0.02 17.02  3.39 2.01  0.40 SNGP SNGP 14.52  2.48 2.00  0.35 16.08  4.18 1.02  0.18 51.96  1.89 2.64  0.07 56.43  23.03 0.60  0.22 44.80  11.00 5.06  1.01 SRSN MP 18.83  3.89 2.46  0.46 19.02  6.07 1.21  0.35 81.25  12.56 3.40  0.33 7.46  1.39 0.12  0.02 20.13  3.50 2.30  0.26 SR CER MP 14.62  1.62 2.02  0.19 14.56  2.14 1.00  0.14 56.97  9.69 2.53  0.15 6.84  1.41 0.11  0.02 21.31  1.63 2.49  0.25 SR metric MP 18.39  2.94 2.40  0.27 16.90  3.12 1.16  0.24 44.54  2.11 2.22  0.15 6.51  1.07 0.10  0.02 20.32  1.68 2.32  0.23 SR(baseline) MP 22.32  8.08 2.58  0.65 17.93  3.84 1.22  0.28 49.48  3.71 2.35  0.25 6.08  0.62 0.10  0.01 18.81  3.35 2.21  0.29 Table 2: Results of deterministic methods with different types of regularization (ELECTRA model). in Appendix A. For the optimal hyperparameter search, we split the original training data into training and validation subsets in a ratio of 80 to 20 and apply Bayesian optimization with early stopping. For text classification, we use accuracy as an objective metric, and for sequence tagging, we use span-based F1-score (Tjong Kim Sang and De Meulder, 2003). Sets of pre-defined values for each hyperparameter are given in the caption of Table 6. After the hyperparameter search is completed, we train the model on the original training set using the optimal values. The hyperparameters for UE methods are presented in Table 9 in Appendix A. The values for the DDPP MC dropout and MD SN are chosen using a grid search, while validating on the held-out validation dataset with RCC-AUC as an objective. For deep ensemble, we use random subsampling of the training set with a fixed ratio of 90%. The results of methods based on MC dropout and loss regularization are presented in Table 1 (for ELECTRA). The standard computationally intensive MC dropout achieves big improvements over the SR baseline on all text classification datasets and the sequence-level CoNLL-2003 benchmark. For token-level CoNLL-2003, none of the considered methods substantially outperform the baseline. Uncertainty estimation scores BALD and PV have similar results, outperforming SMP on SST-2, while SMP has a slight advantage over them on CoLA and CoNLL-2003. The DDPP MC dropout method does not outperform the MC dropout. However, DDPP (+DDPP) demonstrates a notable advantage over the SR baseline on text classification datasets SST2 and CoLA, while both DDPP (+DDPP) and DDPP (+OOD) outperform the baseline on the sequence-level CoNLL-2003 benchmark. The main advantage of the proposed DDPP MC dropout method consists in its much faster inference compared to the computationally expensive standard MC dropout. The DDPP MC dropout has the same computational overhead during inference as the original DPP MC dropout, which is only less than 0.5% of the overhead introduced by the standard MC dropout (Shelmanov et al., 2021). We conduct an ablation study of the proposed modifications for the original DPP MC dropout. The experimental results of this study presented in Table 12 in Appendix C demonstrate the benefits of using calibration and introducing diversity in mask generation. Both metric regularization and CER achieve a substantial advantage over the baseline on text classification datasets SST-2 and MRPC. However, regularization appears to be malignant for NER. Adding loss regularization to MC dropout usually helps to achieve better results on text classification. The best results on SST-2 and CoLA are achieved using metric regularization, while the best result for MRPC is obtained using CER. Regularization and DDPP MC dropout usually complement each other, the results of their combination are slightly better than when they are applied individually for all datasets except CoNLL-2003. The results for deterministic methods are presented in Table 2 (for ELECTRA). SNGP gives substantial improvements on the text classification datasets MRPC and SST-2 but significantly falls behind the trivial baseline on CoNLL-2003. The low performance of SNGP for NER can be attributed to the fact that it is initially designed for classification 8243 Method Reg.Type UEScore MRPC SST-2 CoLA CoNLL-2003(tokenlevel) CoNLL-2003(seq. level) RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  RCC-AUC  RPP  MC SMP 14.38  2.07 1.76  0.19 14.00  2.20 0.91  0.15 42.95  5.98 2.01  0.15 6.04  1.03 0.09  0.02 15.79  3.34 1.80  0.35 MC CER PV 12.82  1.89 1.60  0.13 12.18  1.20 0.80  0.10 46.84  9.19 2.11  0.23 6.92  1.22 0.10  0.02 17.05  3.14 1.91  0.36 MC metric BALD 14.55  2.31 1.73  0.23 12.08  1.79 0.79  0.10 43.76  0.55 2.08  0.07 6.91  1.02 0.10  0.01 17.47  1.85 1.98  0.30 MC metric SMP 13.39  1.19 1.72  0.20 13.55  1.65 0.90  0.14 40.88  1.25 2.01  0.09 6.30  0.98 0.10  0.01 16.81  1.40 1.80  0.23 DeepEnsemble PV 20.70  4.24 2.10  0.35 12.02  1.63 0.71  0.07 50.15  5.57 2.21  0.19 4.02  1.24 0.06  0.02 13.18  4.60 1.54  0.57 DeepEnsemble SMP 13.01  2.57 1.68  0.27 12.13  1.27 0.79  0.08 43.73  4.25 2.05  0.19 4.16  1.37 0.06  0.02 13.93  4.88 1.57  0.58 MSD MSD DS 12.70  1.61 1.74  0.25 11.17  1.03 0.78  0.06 39.21  2.18 1.90  0.12 12.34  4.19 0.18  0.05 16.83  3.92 1.94  0.25 DDPP(+DPP)(ours) PV 22.30  7.15 2.58  0.65 16.70  1.38 1.12  0.12 49.75  3.96 2.44  0.29 6.12  0.71 0.10  0.01 16.78  2.44 1.93  0.20 DDPP(+DPP)(ours) SMP 21.79  7.72 2.57  0.68 17.55  3.03 1.19  0.23 47.86  5.51 2.39  0.31 6.08  0.62 0.10  0.01 17.71  2.77 2.05  0.23 DDPP(+DPP)(ours) CER PV 15.12  2.27 2.03  0.24 13.56  1.37 0.91  0.14 54.51  8.80 2.58  0.22 6.98  0.98 0.11  0.02 19.44  1.15 2.13  0.17 DDPP(+DPP)(ours) CER SMP 14.75  1.43 2.02  0.16 14.47  1.63 0.99  0.11 54.01  9.79 2.55  0.18 6.91  1.13 0.11  0.02 20.66  1.53 2.31  0.08 DDPP(+DPP)(ours) metric SMP 18.45  2.88 2.41  0.26 16.78  3.43 1.14  0.26 43.61  1.61 2.16  0.11 6.92  1.32 0.11  0.02 19.11  2.14 2.16  0.22 DDPP(+OOD)(ours) PV 22.73  7.45 2.65  0.59 19.05  2.95 1.29  0.23 51.11  12.03 2.37  0.34 6.32  0.72 0.10  0.01 16.75  2.31 1.94  0.21 DDPP(+OOD)(ours) SMP 22.31  7.80 2.60  0.65 19.86  3.83 1.36  0.29 50.14  9.73 2.32  0.30 6.09  0.67 0.10  0.01 17.76  2.75 2.06  0.23 DDPP(+OOD)(ours) CER BALD 15.03  1.85 2.08  0.24 14.37  2.22 0.96  0.14 57.48  9.37 2.54  0.26 7.41  1.29 0.12  0.02 25.30  3.36 3.00  0.24 DDPP(+OOD)(ours) CER SMP 14.34  1.15 1.99  0.16 15.88  1.96 1.08  0.13 59.32  11.86 2.53  0.20 6.88  1.24 0.11  0.02 21.06  1.96 2.35  0.14 DDPP(+OOD)(ours) metric SMP 18.55  3.06 2.42  0.27 17.08  3.78 1.14  0.26 43.67  1.77 2.15  0.11 6.71  1.18 0.10  0.02 19.01  2.30 2.16  0.25 MD CER MD 13.61  1.82 1.87  0.22 14.10  2.69 0.96  0.16 42.50  2.65 2.00  0.07 6.82  0.90 0.10  0.01 16.92  2.51 1.87  0.23 MD metric MD 13.91  2.35 1.89  0.29 12.03  2.04 0.85  0.15 40.29  2.09 2.02  0.09 10.01  2.56 0.15  0.03 17.67  3.92 2.09  0.36 MDSN(ours) MD 13.44  1.28 1.85  0.20 11.77  1.33 0.83  0.08 40.07  3.62 1.95  0.16 7.21  1.34 0.11  0.02 17.29  3.58 2.01  0.37 MDSN(ours) CER MD 14.41  1.96 1.94  0.21 12.32  1.37 0.85  0.10 37.82  2.91 1.90  0.12 6.95  1.50 0.11  0.02 17.76  4.00 2.06  0.42 MDSN(ours) metric MD 12.04  1.33 1.56  0.12 12.05  1.42 0.84  0.07 39.37  2.00 1.97  0.15 6.90  1.21 0.11  0.02 17.02  3.39 2.01  0.40 SR CER MP 14.62  1.62 2.02  0.19 14.56  2.14 1.00  0.14 56.97  9.69 2.53  0.15 6.84  1.41 0.11  0.02 21.31  1.63 2.49  0.25 SR metric MP 18.39  2.94 2.40  0.27 16.90  3.12 1.16  0.24 44.54  2.11 2.22  0.15 6.51  1.07 0.10  0.02 20.32  1.68 2.32  0.23 SR(baseline) MP 22.32  8.08 2.58  0.65 17.93  3.84 1.22  0.28 49.48  3.71 2.35  0.25 6.08  0.62 0.10  0.01 18.81  3.35 2.21  0.29 Table 3: Comparison of the best results for all methods (ELECTRA model). tasks rather than sequence tagging. MD yields much bigger improvements over the SR baseline on all datasets and significantly outperforms SNGP. MD SN is able to improve the misclassification detection performance even further for MRPC, SST-2, and CoLA. We also conduct an ablation study (Table 2), in which we use the spectral normalization without MD. We see that SN on its own, as expected, mostly does not improve the UE performance; the results usually are even slightly worse than the baseline. Regularization also helps to improve the results of methods based on the Mahalanobis distance. For both MD and MD SN, regularization helps on CoLA and CoNLL-2003. For MD, it also helps on SST-2, while for MD SN, regularization improves the results on MRPC. We note that regularization reduces the gap between MD and MD SN on text classification datasets and even gives a slight advantage to MD over MD SN on CoNLL-2003. The best results across all deterministic methods for text classification datasets are achieved by MDSN. The biggest improvements are obtained on MRPC, where regularized MD SN reduces RCC-AUC by more than 46% compared to the baseline. Table 3 and Figure 1 compare results of the best methods in each group for ELECTRA. Table 11 and Figure 3 in Appendix B show the best results for DeBERTa. In these tables and figures, we also present the results of deep ensemble (Lakshminarayanan et al., 2017), which is a strong yet computationally intensive baseline (Ashukha et al., 2020), and results of another recently proposed computationally intensive method called MSD (He et al., 2020) that leverage mix-up (Thulasidasan et al., 2019), self-ensembling, MD, 8244 0.2 0.4 0.6 0.8 1.0 Rejection rate 0.90 0.92 0.94 0.96 0.98 1.00 A cc u r a c y s c o r e Deep Ensemble, SMP DDPP (+OOD), CER, SMP (ours) MC, CER, PV MD, CER MD SN, metric (ours) SR (baseline) Figure 2: Median values of accuracy rejection curves for selected methods on MRPC (ELECTRA model). and the MC dropout (all layers are activated). We can see that it is possible to substantially improve misclassification detection performance and achieve even better results than MC dropout, deep ensemble, or MSD almost with no overhead in terms of memory consumption and amount of computation. For text classification and for both models, computationally cheap methods are either better or on par with the expensive counterparts. However, for NER, we see that the latter methods seriously fall behind deep ensemble and MC dropout. On the token-level CoNLL-2003 benchmark, only deep ensemble substantially outperforms the SR baseline. On the sequence-level CoNLL-2003 benchmark, MD with CER, DDPP (+DDP) PV, and DDPP (+OOD) PV improve upon SR, but only approach the performance of computationally intensive methods. The proposed in this work MD SN method outperforms all other computationally efficient alternatives on text classification datasets. For both models, it even substantially outperforms all computationally expensive methods on the CoLA dataset, while on other text classification datasets it is on par with them. Another method proposed in this work, DDPP MC dropout, empowered with regularization techniques, is able to substantially reduce the gap between the SR baseline and computationally intensive UE methods, while introducing only a fraction of their overhead. Figure 2 also presents accuracy rejection curves for selected methods on MRPC. The figure shows that if we reject 20% of instances using UE obtained with MC dropout and ask human experts to label these uncertain objects, the accuracy score of such a human-machine hybrid system will increase from 88.4% to 96.0%, which is 1.3% better than the SR baseline. Such an additional gain over the SR baseline can be crucial for safe-critical applications. Deep ensemble and MD SN are close to each other and achieve 95.6% and 95.2% of accuracy correspondingly. Rejecting 40% of most uncertain instances gives 98.2% of accuracy for the computationally-intensive deep ensemble, while the proposed cheap MD SN method yields even better result with 98.5% of accuracy, which is 1.7% higher than the result of the SR baseline. Our extensive empirical investigation on text classification and NER tasks shows that computationally cheap UE methods are able to substantially improve misclassification detection for Transformers, performing on par or even better than computationally intensive MC dropout and deep ensemble. The proposed in this work method based on the Mahalanobis distance and spectral normalization of a weight matrix (MD SN) achieves the best results among other computationally cheap methods on text classification datasets and is on par with expensive methods. This method does not require seriously modifying a model architecture, extra memory storage, and introduces only a little amount of additional computation during inference. We also show that our modification of DPP MC dropout that leverages the diversity of generated dropout masks, which is also a computationally cheap method, is able to outperform the softmax response baseline and approach the computationally intensive methods on NER. Finally, we find that regularization can slightly improve the results of methods based on MC dropout and the Mahalanobis distance in text classification. The spectral normalization is theoretically proven to ensure bi-Lipschitz constraint on the transformation defined by the standard residual connection network (Liu et al., 2020). However, the self-attention blocks in Transformers have a more complicated architecture than the layers of standard ResNets, which means that the theoretical guarantees for them do not hold in general. In future work, we are looking forward to investigating other techniques to ensure bi-Lipschitz constraint on self-attention blocks, which might further improve deterministic methods for uncertainty estimation of Transformers. We thank anonymous reviewers for their insightful suggestions to improve this paper. The work was supported by a grant for research centers in the field of artificial intelligence (agreement identifier 000000D730321P5Q0002 dated November 2, 2021 No. 70-2021-00142 with ISP RAS).
abstain abstain abstain objective abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain objective objective abstain result abstain abstain objective objective objective objective objective other other other other other other other other other other other other other method other objective other other other other other other method objective objective objective other other other method method other other other other other method method method other other other method other other method method objective method method objective objective other method method other other other other other other other other other other method method method other other other other other other other other other other method method abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain method abstain method method method method abstain other method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method result abstain abstain abstain method abstain abstain abstain abstain other abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result objective abstain result result abstain abstain objective other other