We address the problem of calibrating prediction confidence for output entities of interest in natural language processing (NLP) applications. It is important that NLP applications such as named entity recognition and question answering produce calibrated confidence scores for their predictions, especially if the applications are to be deployed in a safety-critical domain such as healthcare. However, the output space of such structured prediction models is often too large to adapt binary or multi-class calibration methods directly. In this study, we propose a general calibration scheme for output entities of interest in neural network based structured prediction models. Our proposed method can be used with any binary class calibration scheme and a neural network model. Additionally, we show that our calibration method can also be used as an uncertainty-aware, entity-specific decoding step to improve the performance of the underlying model at no additional training cost or data requirements. We show that our method outperforms current calibration techniques for named-entity-recognition, part-of-speech and question answering. We also improve our model's performance from our decoding step across several tasks and benchmark datasets. Our method improves the calibration and model performance on out-of-domain test scenarios as well. Several modern machine-learning based Natural Language Processing (NLP) systems can provide a confidence score with their output predictions. This score can be used as a measure of predictor confidence. A well-calibrated confidence score is a probability measure that is closely correlated with the likelihood of model output's correctness. As a result, NLP systems with calibrated confidence can predict when their predictions are likely to be incorrect and therefore, should not be trusted. This property is necessary for the responsible deployment of NLP systems in safety-critical domains such as healthcare and finance. Calibration of predictors is a well-studied problem in machine learning (Guo et al., 2017; Platt, 2000); however, widely used methods in this domain are often defined as binary or multi-class problems(Naeini et al., 2015; Nguyen and O'Connor, 2015). The structured output schemes of NLP tasks such as information extraction (IE) (Sang and De Meulder, 2003) and extractive question answering (Rajpurkar et al., 2018) have an output space that is often too large for standard multi-class calibration schemes. Formally, we study NLP models that provide conditional probabilities p  ( y | x ) for a structured output y given input x . The output can be a label sequence in case of part-of-speech (POS) or named entity recognition (NER) tasks, or a span prediction in case of extractive question answering (QA) tasks, or a relation prediction in case of relation extraction task. p  ( y | x ) can be used as a score of the model's confidence in its prediction. However, p  ( y | x ) is often a poor estimate of model confidence for the output y . The output space of the model in sequence-labelling tasks is often large, and therefore p  ( y | x ) for any output instance y will be small. For instance, in a sequence labelling task with C number of classes and a sequence length of L , the possible events in output space will be of the order of CL . Additionally, recent efforts (Guo et al., 2017; Nguyen and O'Connor, 2015; Dong et al., 2018; Kumar and Sarawagi, 2019) at calibrating machine learning models have shown that they are poorly calibrated. Empirical results from Guo et al. (2017) show that techniques used in deep neural networks such as dropout and their large architecture size can negatively affect the calibration of their outputs in binary and multi-class classification tasks. Parallelly, large neural network architectures based on contextual embeddings (Devlin et al., 2018; Peters et al., 2018) have shown state-of-the-art performance across several NLP tasks (Andrew and Gao, 2007; Wang et al., 2019) . They are being rapidly adopted for information extraction and other NLP tasks in safety-critical applications (Zhu et al., 2018; Sarabadani, 2019; Li et al., 2019; Lee et al., 2019). Studying the miss-calibration in such models and efficiently calibrating them is imperative for their safe deployment in the real world. In this study, we demonstrate that neural network models show high calibration errors for NLP tasks such as POS, NER and QA. We extend the work by Kuleshov and Liang (2015) to define well-calibrated forecasters for output entities of interest in structured prediction of NLP tasks. We provide a novel calibration method that applies to a wide variety of NLP tasks and can be used to produce model confidences for specific output entities instead of the complete label sequence prediction. We provide a general scheme for designing manageable and relevant output spaces for such problems. We show that our methods lead to improved calibration performance on a variety of benchmark NLP datasets. Our method also leads to improved out-of-domain calibration performance as compared to the baseline, suggesting that our calibration methods can generalize well. Lastly, we propose a procedure to use our calibrated confidence scores to re-score the predictions in our defined output event space. This procedure can be interpreted as a scheme to combine model uncertainty scores and entity-specific features with decoding methods like Viterbi. We show that this re-scoring leads to consistent improvement in model performance across several tasks at no additional training or data requirements. Structured Prediction refers to the task of predicting a structured output y = [ y 1 , y 2 , ...y L ] for an input x . In NLP, a wide array of tasks including parsing, information extraction, and extractive question answering fall within this category. Recent approaches towards solving such tasks are commonly based on neural networks that are trained by minimizing the following objective : L (  |D ) =  |D| (cid:88) i =0 log ( p  ( y ( i ) | x ( i ) )) + R (  ) (1) where  is the parameter vector of the neural network and R is the regularization penalty and D is the dataset { ( y ( i ) , x ( i ) ) } |D| i =0 . The trained model p  can then be used to produce the output  y = argmax y Y p  ( y | x ) . Here, the corresponding model probability p  ( y | x ) is the uncalibrated confidence score. In binary class classification, the output space Y is [0 , 1] . The confidence score for such classi-fiers can then be calibrated by training a forecaster F y : [0 , 1]  [0 , 1] which takes in the model confidence F y ( P  ( y | x )) to produce a recalibrated score (Platt, 2000). A widely used method for binary class calibration is Platt scaling where F y is a logistic regression model. Similar methods have also been defined for multi-class classification (Guo et al., 2017). However, extending this to structured prediction in NLP settings is non-trivial since the output space |Y| is often too large for us to calibrate the output probabilities of all events. Calibration methods for binary/multi class classification has been widely studied in related literature (Brocker, 2009; Guo et al., 2017). Recent efforts at confidence modeling for NLP has focused on several tasks like co-reference, (Nguyen and O'Connor, 2015), semantic parsing (Dong et al., 2018) and neural machine translation (Kumar and Sarawagi, 2019). In this section, we define the calibration framework by Kuleshov and Liang (2015) in the context of structured prediction problems in NLP. The model p  denotes the neural network that produces an conditional probability p  ( y | x ) given an ( x, y ) tuple. In a multi/binary class setting, a function F y is used to map the output p  ( y | x ) to a calibrated confidence score for all y  Y . In a structured prediction setting, since the cardinality of Y is usually large, we instead focus on the event of interest set I ( x ) . I ( x ) contains events of interest E that are defined using the output events relevant to the deployment requirements of a model. The event E is a subset of Y . There can be several different schemes to define I ( x ) . In later sections, we discuss related work on calibration that can be understood as applications of different I ( x ) schemes. In this work, we define a general framework for constructing I ( x ) for NLP tasks which allows us to maximize calibration performance on output entities of interest. We define F y ( E, x, p  ) to be a function, that takes the event E , the input feature x and p  to produce a confidence score between [0 , 1] . We refer to this calibration function as the forecaster and use F y ( E, x ) as a shorthand since it is implicit that F y depends on outputs of p  . We would like to find the forecaster that minimizes the discrepancy between F y ( E , x ) and P ( y  E | x ) for ( x, y ) sampled from P ( x, y ) and E uniformly sampled from I ( x ) . A commonly used methodology for constructing a forecaster for p  is to train it on a held-out dataset D dev . A forecaster for a binary classifier is perfectly calibrated if P ( y = 1 | F y ( x ) = p ) = p. (2) It is trained on samples from { ( x, I ( y = 1) : ( x, y )  D dev } . The main contributions of this paper stem from our proposed schemes for constructing the aformen-tioned I ( x ) sets for NLP applications. Entities of Interest : In the interest of brevity, let us define Entities of interest  ( x ) as the set of all entity predictions that can be queried from p  for a sample x . For instance, in the case of answer span prediction for QA, the  ( x ) may contain the MAP prediction of the best answer span (answer start and end indexes). In a parsing or sequence labeling task,  ( x ) may contain the top-k label sequences obtained from viterbi decoding. In a relation or named-entity extraction task,  ( x ) contains the relation or named entity span predictions respectively. Each entity s in  ( x ) corresponds to a event set E that is defined by all outputs in Y that contain the entity s . I ( x ) contains set E for all entities in  ( x ) . Positive Entities and Events : We are interested in providing a calibrated probability for y  E corresponding to an s for all s in  ( x ) . Here y is the correct label sequence for the input x . If y lies in the set E for an entity s , we refer to s as a positive entity and the event as a positive event. In the example of named entity recognition, s may refer to a predicted entity span, E refers to all possible sequences in Y that contain the predicted span. The corresponding event is positive if the correct label sequence y contains the span prediction s . Schemes for construction of I ( x ) : While constructing the set  ( x ) we should ensure that it is limited to a relatively small number of output entities, while still covering as many positive events in I ( x ) as possible. To explain this consideration, let us take the example of a parsing task such as syntax or semantic parsing. Two possible schemes for defining I ( x ) are : 1. Scheme 1:  ( x ) contains the MAP label sequence prediction. I ( x ) contains the event corresponding to whether the label sequence y (cid:48) = argmax y p  ( y | x ) is correct. 2. Scheme 2:  ( x ) contains all possible label sequences. I ( x ) contains a event corresponding to whether the label sequence y (cid:48) is correct, for all y (cid:48)  Y Calibration of model confidence by Dong et al. (2018) can be viewed as Scheme 1, where the entity of interest is the MAP label sequence prediction. Whereas, using Platt Scaling in a one-vs-all setting for multi-class classification (Guo et al., 2017) can be seen as an implementation of Scheme 2 where the entity of interest is the presence of class label. As discussed in previous sections, Scheme 2 is too computationally expensive for our purposes due to large value of |Y| . Scheme 1 is computationally cheaper, but it has lower coverage of positive events. For instance, a sequence labelling model with a 60% accuracy at sentence level means that only 60 % of positive events are covered by the set corresponding to argmax y p  ( y | x ) predictions. In other words, only 60 % of the correct outputs of model p  will be used for constructing the forecaster. This can limit the positive events in I ( x ) . Including the topk predictions in  ( x ) may increase the coverage of positive events and therefore increase the positive training data for the forecaster. The optimum choice of k involves a trade-off. A larger value of k implies broader coverage of positive events and more positive training data for the forecaster training. However, it may also lead to Calibration BERT BERT +CRF Distil BERT Platt 15.90  .03 15.56  .23 12.30  .13 Calibrated Mean 2.55  .34 2.31  .35 2.02  .16 +Var 2.11  .32 2.55  .32 2.73  .40 Platt+top2 11.4  .07 14.21  .16 11.03  .31 Calibrated Mean+top2 2.94  .29 4.82  .15 3.61  .17 +Var+top2 2.17  .35 4.26  .10 2.43  .16 +Rank+top2 2.43  .30 2.43  .45 2.21  .09 +Rank+Var+top2 1.81  .12 2.29  .27 1.97  .14 Platt+top3 17.46  .13 18.11  .16 12.84  .37 +Rank+Var+top3 3.18  .12 3.71  .25 2.05  .06 Table 1: ECE percentages on Penn Treebank for different models and calibration methods. Task specific details about  ( x ) are provided in the later sections. For the purposes of this paper, topk refers to the top k MAP sequence predictions, also referred to as argmax ( k ) . Here we provide a summary of the steps involved in Forecaster construction. Remaining details are in the Appendix. We train the neural network model p  on the training data split for a task and use the validation data for monitoring the loss and early stopping. After the training is complete, this validation data is re-purposed to create the forecaster training data. We use an MC-Dropout(Gal and Ghahramani, 2016) average of (n=10) samples to get a low variance estimate of logit outputs from the neural networks. This average is fed into the decoding step of the model p  to obtain topk label sequence predictions. We then collect the relevant entities in  ( x ) , along with the I ( y  E) labels to form the training data for the forecaster. We use gradient boosted decision trees (Friedman, 2001) as our region-based (Dong et al., 2018; Kuleshov and Liang, 2015) forecaster model. Choice of the hyperparameter k : We limit our choice of k to { 2 , 3 } . We train our forecasters on training data constructed through top2 and top3 extraction each. These two models are then evaluated on top1 extraction training data, and the best value of k is used for evaluation on test. This heuristic for k selection is based on the fact that the top1 training data for a good predictor p  , is a positive-event rich dataset. Therefore, this dataset can be used to reject a larger k if it leads to reduced performance on positive events. We refer to the value of k obtained from this heuristic as as heuristick . Model and Model Uncertainty based features contain the mean probability obtained by averaging over the marginal probability of the entity of interest obtained from 10 MC-dropout samples of p  . Average of marginal probabilities acts as a reduced variance estimate of un-calibrated model confidence. Our experiments use the pre-trained contextual word embedding architectures as the backbone networks. We obtain MC-Dropout samples by enabling dropout sampling for all dropout layers of the networks. We also provide 10 th and 90 th percentile values from the MC-Dropout samples, to provide model uncertainty information to the forecaster. Since our forecaster training data contains entity predictions from topk MAP predictions, we also include the rank k as a feature. We refer to these two features as Var and Rank in our models. Entity of interest based features contain the length of the entity span if the output task is named entity. We only use this feature in the NER experiments and refer to it as ln. Data Uncertainty based features: Dong et al. (2018) propose the use of language modelling (LM) Calibration BERT BERT +CRF Distil BERT Baseline 60.30  .12 62.31  .11 60.17  .08 +Rank+Var+top2 60.30  .23 62.31  .09 60.13  .11 +Rank+Var+top3 59.84  .16 61.06  .14 58.95  .08 Table 2: Micro-avg f-score for POS datasets using the baseline and our best proposed calibration method. and OOV-word-based features as a proxy for data uncertainty estimation. The use of word-pieces and large pre-training corpora in contextual word embedding models like BERT may affect the ef-ficacy of LM based features. Nevertheless, we use LM perplexity (referred to as lm) in the QA task to investigate its effectiveness as an indicator of the distributional shift in data. Essentially, our analysis focuses on LM perplexity as a proxy for distributional uncertainty (Malinin and Gales, 2018) in our out-of-domain experiments. The use of word-pieces in models like BERT reduces the negative effect of OOV words on model prediction. Therefore, we do not include OOV features in our experiments. We use BERT -base (Devlin et al., 2018) and distil BERT (Sanh et al., 2019) network architecture for our experiments. Validation split for each dataset was used for early stopping BERT fine-tuning and as training data for forecaster training. POS and NER experiments are evaluated on Penn Treebank and CoNLL 2003 (Sang and De Meulder, 2003), MADE 1.0 (Jagannatha et al., 2019) respectively. QA experiments are evaluated on SQuAD1.1 (Ra-jpurkar et al., 2018) and EMRQA (Pampari et al., 2018) corpus. We also investigate the performance of our forecasters on an out-of-domain QA corpus constructed by applying EMRQA QA data generation scheme (Pampari et al., 2018) on the MADE 1.0 named entity and relations corpus. Details for these datasets are provided in their relevant sections. We use the expected calibration error (ECE) metric defined by Naeini et al. (2015) with N = 20 bins (Guo et al., 2017) to evaluate the calibration of our models. ECE is defined as an estimate of the expected difference between the model confidence and accuracy. ECE has been used in several related works (Guo et al., 2017; Maddox et al., 2019; Kumar et al., 2018; Vaicenavicius et al., 2019) to estimate model calibration. We use Platt scaling as the baseline calibration model. It uses the length-normalized probability averaged across 10 MC-Dropout samples as the input. The lower variance and length invariance of this input feature make Platt Scaling a strong baseline. We also use a Calibrated Mean baseline using Gradient Boosted Decision Trees as our estimator with the same input feature as Platt. Part-of-speech (POS) is a sequence labelling task where the input is a text sentence, and the output is a sequence of syntactic tags. We evaluate our method on the Penn Treebank dataset (Marcus et al., 1994). We can define either the token prediction or the complete sequence prediction as the entity of interest. Since using a token level entity of interest effectively reduces the calibration problem to that of calibrating a multi-class classifier, we instead study the case where the predicted label sequence of the entire sentence forms the entity of interest set. The event of interest set is defined by the events y = MAP k ( x ) which denote whether each topk sentence level MAP prediction is correct. We use three choice of p  models, namely BERT , BERT-CRF and distil BERT . We use model uncertainty and rank based features for our POS experiments. Table 1 shows the ECE values for our baseline, proposed and ablated models. The value of heuristick is 2 for all +Rank+Var+topk forecasters across all PTB models. top k  in Table 1 refers to forecasters trained with additional topk predictions. Our methods outperform both baselines by a large margin. Both Rank and Var features help in improving model calibration. Inclusion of top2 prediction sequences also improve the calibration performance significantly. Table 1 also shows the performance of our full feature model +Rank+Var+top k  for the sub-optimal value of Calibration CoNLL MADE 1.0 ( BERT ) (bio BERT ) Platt 2.00  .12 4.00  .07 Calibrated Mean 2.29  .33 3.07  .18 +Var 2.43  .36 3.05  .17 +Var+ln 2.24  .14 2.92  .24 Platt+top3 16.64  .48 2.14  .18 Calibrated Mean+top3 17.06  .50 2.22  .31 +Var+top3 17.10  .24 2.17  .39 +Rank+Var+top3 2.01  .33 2.34  .15 +Rank+Var+ln+top3 1.91  .29 2.12  .24 Table 3: ECE percentages for the two named entity datasets and calibration methods. k = 3 . It has lower performance than k = 2 across all models. Therefore for the subsequent experimental sections, we only report top k calibration performance using the heuristick value only. We use the confidence predictions of our full-feature model +Rank+Var+top k to re-rank the top-k predictions in the test set. Table 2 shows the sentence-level (entity of interest) accuracy for our re-ranked top prediction and the original model prediction. For Named Entity (NE) Recognition experiments, we use two NE annotated datasets, namely CoNLL 2003 and MADE 1.0. named entities such as Person, Location etc. as Medication, Indication and Adverse effects. The entity of interest for NER is the named entity span prediction. We define  ( x ) as predicted entity spans in argmax ( k ) label sequences predictions for x . We use BERT -base with token-level softmax output and marginal likelihood based training. The model uncertainty estimates for Var feature are computed by estimating the variance of length normalized MC-dropout samples of span marginals. Due to the similar trends in behavior of BERT and BERT +CRF model in POS experiments, we only use BERT model for NER. However, the span marginal computation can be easily extended to linear-chain CRF models. We also use the length of the predicted named entity as the feature ln in this experiment. Complete details about forecaster and baselines are in the Appendix. Value of heuristick is 3 for all +Rank+Var+topk forecasters. We show ablation and baseline results for k = 3 only. However, no other forecasters for any k  { 2 , 3 } outperform our best forecasters in Table 3. We use the confidence predictions of our +Rank+Var+top3 models to re-score the confidence predictions for all spans predicted in top3 MAP predictions for samples in the test set. A threshold of 0.5 was used to remove span predictions with low confidence scores. Table 4 shows the Named Entity level (entity of interest) Micro-F score for our re-ranked top prediction and the original model prediction. We see that re-ranked predictions from our models consistently improve the model f-score. We use three datasets for evaluation of our calibration methods on the QA task. Our QA tasks are modeled as extractive QA methods with a single span answer predictions. We use three datasets to construct experiments for QA calibration. SQuAD1.1 and EMRQA (Pampari et al., 2018) are open-domain and clinical-domain QA datasets, respectively. We process the EMRQA dataset by restricting the passage length and removing unanswerable questions. We also design an out-of-domain evaluation of calibration using clinical QA datasets. We follow the guidelines from Pampari et al. (2018) to create a QA dataset Calibration SQuAD1.1 EMRQA MADE 1.0 MADE 1.0(OOD) ( BERT ) (bio BERT ) (bio BERT ) (bio BERT ) Platt 3.69  .16 5.07  .37 3.64  .17 15.20  .16 Calibrated Mean 2.95  .26 2.28  .18 2.50  .31 13.26  .94 +Var 2.92  .28 2.74  .15 2.71  .32 12.41  .95 Platt+top3 7.71  .28 5.42  .25 11.87  .19 16.36  .26 Calibrated Mean+top3 3.52  .35 2.11  .19 9.21  .25 12.11  .24 +Var+top3 3.56  .29 2.20  .20 9.26  .27 11.67  .27 +Var+lm+top3 3.54  .21 2.12  .19 6.07  .26 12.42  .32 +Rank+Var+top3 2.47  .18 1.98  .10 1.77  .23 12.69  .20 +Rank+Var+lm+top3 2.79  .32 2.24  .29 1.66  .27 12.60  .28 Table 5: ECE percentages for QA tasks SQuAD1.1, EMRQA and MADE 1.0. from MADE 1.0 (Jagannatha et al., 2019). This allows us to have two QA datasets with common question forms, but different text distributions. In this experimental setup we can mimic the evaluation of calibration methods in a real-world scenario, where the task specifications may remain the same but the underlying text source changes. Details about dataset pre-processing and construction are provided in the Appendix. The entity of interest for QA is the topk answer span predictions. We use the lm perplexity as a feature in this experiment to analyze its behaviour in out-of-domain evaluations. We use a 2 layer unidirectional LSTM to train a next word language model on the EMRQA passages. This language model is then used to compute the perplexity of a sentence for the lm input feature to the forecaster. We use the same baselines as the previous two tasks. Based on Table 5, our methods outperform the baselines by a large margin in both in-domain and out-of-domain experiments. Value of heuristick is 3 for all +Rank+Var+topk forecasters. We show ablation and baseline results for k = 3 only. However, no other forecasters for any k  { 2 , 3 } outperform our best forecasters in Table 5 . Our models are evaluated on SQuAD1.1 dev set, and test sets from EMRQA and MADE 1.0. They show consistent improvements in ECE and Exact Match Accuracy. Our proposed methods outperform the baselines in most tasks and are almost as competitive in others. Features and topk samples: The inclusion of topk features improve the performance in almost all tasks when the rank of the prediction is included. We see large increases in calibration error when the topk prediction samples are included in forecaster training without including the rank information in tasks such as CoNLL NER and MADE 1.0 QA. This may be because the k = 1 , 2 , 3 predictions Figure 1: Modified reliability plots ( Accuracy Confidence vs Confidence ) on MADE 1.0 QA test. may have similar model confidence and uncertainty values. Therefore a more discriminative signal such as rank is needed to prioritize them. For instance, the difference between probabilities of k = 1 and k = 2 MAP predictions for POS tagging may differ by only one or two tokens. In a sentence of length 10 or more, this difference in probability when normalized by length would account to very small shifts in the overall model confidence score. Therefore an additional input of rank k leads to a substantial gain in performance for all models in POS. Our task-agnostic scheme of Rank+Var+topk based forecasters consistently outperform or stay competitive to other forecasting methods. However, results from task-specific features such as lm and len show that use of task-specific features can further reduce the calibration error. Our domain shift experimental setup has the same set of questions in both in-domain and out-of-domain datasets. Only the data distribution for the answer passage is different. However, we do not observe an improvement in out-of-domain performance by using lm feature. A more detailed analysis of task-specific features in QA with both data and question shifts is required. We leave further investigations of such schemes as our future work. Choice of k is important : The optimal choice of k seems to be strongly dependent on the inherent properties of the tasks and its output event set. In all our experiments, for a specific task all Figure 2: An example of named entity span from CoNLL dataset. +Rank+Var+top k forecasters exhibit consistent behaviours with respect to the choice of k . In POS experiments, heuristick = 2 . In all other tasks, heuristick = 3 . Our heuristick models are the best performing models, suggesting that the heuristic described in Section 2.5 may generalize to other tasks as well. Re-scoring : We show that using our forecaster confidence to re-rank the entities of interest leads to a modest boost in model performance for the NER and QA tasks. In POS no appreciable gain or drop in performance was observed for k = 2 . We believe this may be due to the already high token level accuracy (above 97%) on Penn Treebank data. Nevertheless, this suggests that our re-scoring does not lead to a degradation in model performance in cases where it is not effective. Our forecaster re-scores the topk entity confidence scores based on model uncertainty score and entity-level features such as entity lengths. Intuitively, we want to prioritize predictions that have low uncertainty over high uncertainty predictions, if their uncalibrated confidence scores are similar. We provide an example of such re-ranking in Figure 2. It shows a named entity span predictions for the correct span Such. The model p  produces two entity predictions off-spinner Such and Such. The un-calibrated confidence score of off-spinner Such is higher than Such, but the variance of its prediction is higher as well. Therefore the +Rank+Var+ln+top3 re-ranks the second (and correct) prediction higher. It is important to note here that the variance of off-spinner Such may be higher just because it involves two token predictions as compared to only one token prediction in Such. This along with the ln feature in +Rank+Var+ln+top3 may mean that the forecaster is also using length information along with uncertainty to make this prediction. However, we see similar improvements in QA tasks, where the ln feature is not used, and all entity predictions involve two predictions (span start and end index predictions). These results suggest that use of uncertainty features are useful in both calibration and re-ranking of predicted structured output entities. Out-of-domain Performance : Our experiments testing the performance of calibrated QA systems on out-of-domain data suggest that our methods result in improved calibration on unseen data as well. Additionally, our methods also lead to an improvement in system accuracy on out-of-domain data, suggesting that the mapping learned by the forecaster model is not specific to a dataset. However, there is still a large gap between the calibration error for within domain and out-of-domain testing. This can be seen in the reliability plot shown in Figure 1. The number of samples in each bin are denoted by the radius of the scatter point. The calibrated models shown in the figure corresponds to +Rank+Var+lm+top3' forecaster calibrated using both in-domain and out-of-domain validation datasets for forecaster training. We see that out-of-domain forecasters are over-confident and this behaviour is not mitigated by using data-uncertainty aware features like lm. This is likely due to a shift in model's prediction error when applied to a new dataset. Re-calibration of the forecaster using a validation set from the out-of-domain data seems to bridge the gap. However, we can see that the sharpness (Kuleshov and Liang, 2015) of out-of-domain trained, in-domain calibrated model is much lower than that of in-domain trained, in-domain calibrated one. Additionally, a validation dataset is often not available in the real world. Mitigating the loss in calibration and sharpness induced by out-of-domain evaluation is an important avenue for future research. Uncertainty Estimation : We use MC-Dropout as a model ( epistemic ) uncertainty estimation method in our experiments. However, our method is not specific to MC-Dropout, and is compatible with any method that can provide a predictive distribution over token level outputs. As a result any bayesian or ensemble based uncertainity estimation method (Welling and Teh, 2011; Lakshmi-narayanan et al., 2017; Ritter et al., 2018) can be used with our scheme. In this work, we do not investigate the use of aleatoric uncertainty for calibration. Our use of language model features is aimed at accounting for distributional uncertainty instead of aleatoric uncertainty (Gal, 2016; Malinin and Gales, 2018). Investigating the use of different types of uncertainty for calibration remains as our future work. We show a new calibration and confidence based re-scoring scheme for structured output entities in NLP. We show that our calibration methods outperform competitive baselines on several NLP tasks. Our task-agnostic methods can provide calibrated model outputs of specific entities instead of the entire label sequence prediction. We also show that our calibration method can provide improvements to the trained model's accuracy at no additional training or data cost. Our method is compatible with modern NLP architectures like BERT . Lastly, we show that our calibration does not over-fit on in-domain data and is capable of generalizing the calibration to out-of-domain datasets. Research reported in this publication was supported by the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health under Award Number R01HL125089.
abstain abstain abstain objective objective result result result result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective method result result objective abstain result other other other other other other other other other objective other other method abstain abstain method abstain abstain abstain abstain method method method abstain abstain abstain objective method abstain abstain abstain abstain abstain method abstain method abstain abstain method method abstain abstain objective objective objective abstain objective abstain abstain abstain abstain abstain abstain abstain other abstain objective method abstain method abstain method abstain method method method method abstain abstain abstain result abstain abstain method result method abstain abstain abstain method other abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain method abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain result abstain abstain method method result abstain abstain abstain objective result method result abstain result other