Most classification models work by first predicting a posterior probability distribution over all classes and then selecting that class with the largest estimated probability. In many settings however, the quality of posterior probability itself (e.g., 65% chance having diabetes), gives more reliable information than the final predicted class alone. When these methods are shown to be poorly calibrated, most fixes to date have relied on posterior calibration, which rescales the predicted probabilities but often has little impact on final classifications. Here we propose an end-to-end training procedure called posterior calibrated (PosCal) training that directly optimizes the objective while minimizing the difference between the predicted and empirical posterior probabilities. We show that PosCal not only helps reduce the calibration error but also improve task performance by penalizing drops in performance of both objectives. Our PosCal achieves about 2.5 % of task performance gain and 16.1 % of calibration error reduction on GLUE (Wang et al., 2018) compared to the baseline. We achieved the comparable task performance with 13.2 % calibration error reduction on xSLUE (Kang and Hovy, 2019), but not outperforming the two-stage calibration baseline. PosCal training can be easily extendable to any types of classification tasks as a form of regularization term. Also, PosCal has the advantage that it incrementally tracks needed statistics for the calibration objective during the training process, making efficient use of large training sets 1 . Classification systems, from simple logistic regression to complex neural network, typically predict posterior probabilities over classes and decide the final class with the maximum probability. The 1 Code is publicly available at https://github.com/ THEEJUNG/PosCal/ model's performance is then evaluated by how accurate the predicted classes are with respect to out-of-sample, ground-truth labels. In some cases, however, the quality of posterior estimates themselves must be carefully considered as such estimates are often interpreted as a measure of confidence in the final prediction. For instance, a well-predicted posterior can help assess the fairness of a recidivism prediction instrument (Chouldechova, 2017) or select the optimal number of labels in a diagnosis code prediction (Kavuluru et al., 2015). Guo et al. (2017) showed that a model with high classification accuracy does not guarantee good posterior estimation quality. In order to correct the poorly calibrated posterior probability, existing calibration methods (Zadrozny and Elkan, 2001; Platt et al., 1999; Guo et al., 2017; Kumar et al., 2019) generally rescale the posterior distribution predicted from the classifier after training. Such post-processing calibration methods re-learn an appropriate distribution from a held-out validation set and then apply it to an unseen test set, causing a severe discrepancy in distributions across the data splits. The fixed split of the data sets makes the post-calibration very limited and static with respect to the classifier's performance. We propose a simple but effective training technique called Posterior Calibrated (PosCal) training that optimizes the task objective while calibrating the posterior distribution in training. Unlike the post-processing calibration methods, PosCal directly penalizes the difference between the predicted and the true (empirical) posterior probabilities dynamically over the training steps. PosCal is not a simple substitute of the postprocessing calibration methods. Our experiment shows that PosCal can not only reduce the calibration error but also increase the task performance on the classification benchmarks: compared to the baseline MLE (maximum likelihood estimation) training method, PosCal achieves 2.5% performance improvements on GLUE (Wang et al., 2018) and 0.5% on xSLUE (Kang and Hovy, 2019), and at the same time 16.1% posterior error reduction on GLUE and 13.2% on xSLUE. Our work is primarily motivated by previous analyses of posterior calibration on modern neural networks. Guo et al. (2017) pointed out that in some cases, as the classification performance of neural networks improves, its posterior output becomes poorly calibrated. There are a few attempts to investigate the effect of posterior calibration on natural language processing (NLP) tasks: Nguyen and O'Connor (2015) empirically tested how classifiers on NLP tasks (e.g., sequence tagging) are calibrated. For instance, compared to the Naive Bayes classifier, logistic regression outputs well-calibrated posteriors in sentiment classification task. Card and Smith (2018) also mentioned the importance of calibration when generating a training corpus for NLP tasks. As noted above, numerous post-processing calibration techniques have been developed: traditional binning methods (Zadrozny and Elkan, 2001, 2002) set up bins based on the predicted posterior  p , recalculate calibrated posteriors  q per each bin on a validation set, and then update every  p with  q if  p falls into the certain bin. On the other hand, scaling methods (Platt et al., 1999; Guo et al., 2017; Kull et al., 2019) re-scale the predicted posterior  p from the softmax layer trained on a validation set. Recently, Kumar et al. (2019) pointed out that such re-scaling methods do not actually produce well-calibrated probabilities as reported since the true posterior probability distribution can not be captured with the often low number of samples in the validation set 2 . To address the issue, the authors proposed a scaling-binning calibrator, but still rely on the validation set. In a broad sense, our end-to-end training with the calibration reduction loss can be seen as sort of regularization designed to mitigate over-fitting. Just as classical explicit regularization techniques such as the lasso (Tibshirani, 1996) penalize models large weights, here we penalize models with posterior outputs that differ substantially from the estimated true posterior. In general, most of existing classification models are designed to maximize the likelihood estimates (MLE). Its objective is then to minimize the cross-entropy (Xent) loss between the predicted probability and the true probability over k different classes. During training time, PosCal minimizes the cross-entropy as well as the calibration error as a multi-task setup. While the former is a task-specific objective, the latter is a statistical objective to make the model to be statistically well-calibrated from its data distribution. Such data-oriented calibration makes the task-oriented model more reliable in terms of its data distribution. Compared to the prior post-calibration methods with a fixed (and often small) validation set, PosCal dynamically estimates the required statistics for calibration from the train set during training iterations. Given a training set D = { ( x 1 , y 1 ) .. ( x n , y n ) } where x i is a p-dimensional vector of input features and y i is a k-dimensional one-hot vector corresponding to its true label (with k classes), our training minimizes the following loss: LP os C al = L xent +  L cal (1) where L xent is the cross-entropy loss for task objective (i.e., classification) and L cal is the calibration loss on the cross-validation set. is a weighting value for a calibration loss L cal . In practice, the optimal value of  can be chosen via cross-validation. More details are given in  4. L xent =  n (cid:88) i =1 k (cid:88) j =1 y ( j ) i log ( p ( j ) i ) (2) L cal = n (cid:88) i =1 k (cid:88) j =1 d ( p ( j ) i , q ( j ) i ) (3) where L xent is a typical cross-entropy loss with  p as an updated predicted probability while training. L cal is our proposed loss for minimizing the calibration loss: q is an true (empirical) probability and d is an function to measure the difference (e.g., mean squared error or Kullback-Leibler divergence) between the updated  p and true posterior q probabilities. The empirical probability q can be calculated by measuring the ratio of true labels per each bin split by the predicted posterior  p from each update. We sum up the losses from every class j  { 1 , 2 ..k } . We show a detailed training procedure of PosCal in Algorithm 1. While training, we update the model parameters (i.e., weight matrices in the classifier) as well as the empirical posterior probabilities by calculating the predicted posterior with the recently updated parameters. For Q , we exactly calculate a label frequency per bin B . Since it is time-consuming to update Q at every step, we set up the number of Q updates per each epoch so as to only update Q at each batch. Task: NLP classification benchmarks. We test our models on two different benchmarks on NLP classification tasks: GLUE (Wang et al., 2018) and xSLUE (Kang and Hovy, 2019). GLUE contains different types of general-purpose natural language understanding tasks such as question-answering, sentiment analysis and text entailment. Since true labels on the test set are not given from the GLUE benchmark, we use the validation set as the test set, and randomly sample 1% of train set as a validation set. xSLUE (Kang and Hovy, 2019) is yet another classification benchmark but on different types of styles such as a level of humor, formality and even demographics of authors. For the details of each dataset, refer to the original papers. Metrics. In order to measure the task performance, we use different evaluation metrics for each task. For GLUE tasks, we report F1 for MRPC, Matthews correlation for CoLA, and accuracy for other tasks followed by Wang et al. (2018). For xSLUE, we use F1 score. To measure the calibration error, we follow the metric used in the previous work (Guo et al., 2017); Expected Calibration Error ( ECE ) by measuring how the predicted posterior probability is different from the empirical posterior probability: ECE = 1 K (cid:80) Kk =1 (cid:80) Bb =1 | B kb | n | q kb   p kb | , where  p kb is an averaged predicted posterior probability for label k in bin b , q kb is a calculated empirical probability for label k in bin b , B kb is a size of bin b in label k , and n is a total sample size. The lower ECE , the better the calibration quality. Models. We train the classifiers with three different training methods: MLE , L1 , and PosCal . MLE is a basic maximum likelihood estimation training by minimizing the cross-entropy loss, L1 is MLE training with L 1 regularizer, and PosCal is our proposed training by minimizing LP os C al (Eq 1). For PosCal training, we use Kullback-Leibler divergence to measure L cal . We also report ECE with a temperature scaling (Guo et al., 2017) ( tScal ), which is considered the state-of-the-art post-calibration method. For our classifiers, we fine-tuned the pre-trained BERT classifier (Devlin et al., 2019). Details on the hyper-parameters used are given in Appendix A. Task Perf. Results. Table 1 and 2 show task performance and calibration error on two benchmarks: GLUE and xSLUE, respectively. In general, PosCal outperforms the MLE training and MLE with L 1 regularization in GLUE for both task performance and calibration, though not in xSLUE. Compared to the tScal, PosCal shows a stable improvement over different tasks on calibration reduction, while tScal sometimes produces a poorly calibrated result (e.g., CoLA, MRPC). Analysis. We visually check the statistical effect of PosCal with respect to calibration. Figure 1 shows how predicted posterior distribution of PosCal is different from MLE . We choose two datasets where PosCal improves both accuracy and calibration quality compared with the basic MLE: RTE from GLUE and Stanford's politeness dataset from xSLUE. We then draw two different histograms: a histogram of  p frequencies (top) and a calibration histogram,  p versus the empirical posterior probability q (bottom). Figure 1(c,d) show that PosCal spreads out the extremely predicted posterior probabilities (0 or 1) from MLE to be more well calibrated over different bins. The well-calibrated posteriors also help correct the skewed predictions in Figure 1(a,b). To better understand in which case PosCal helps correct the wrong predictions from MLE, we analyze how prediction  p is different between MLE and PosCal in test set. Table 3 shows the number of correct/incorrect predictions and its correspond-Data Sentence True label MLE  p PosCal  p RTE (S1) Researchers at the Harvard School of Public Health say that people who drink coffee may be doing a lot more than keeping themselves awake this kind of consumption apparently also can help reduce the risk of diseases. ing label distributions grouped by the two models. For example, COR by MLE and INCOR by PosCal in the fourth row of Table 3 means that there are three test samples that MLE correctly predicts while PosCal not. We find that in most of cases, PosCal corrects the wrong predictions from MLE by re-scaling  p in a certain direction. In RTE, most inconsistent predictions between MLE and PosCal have their posterior predictions near to the decision boundary (i.e., 50% for binary classification) with an averaged predicted probability about 40%. This is mainly because PosCal does not change the majority of the predictions but helps correct the controversial predictions near to the decision boundary. PosCal improves 3.3% of accuracy but only sacrifices 1.1% by correctly predicting the samples predicted as 'not entailment' by MLE to 'entailment'. On the other hand, SPolite has more extreme distribution of  p from MLE than RTE. We find a fair trade-off between two models (-9.5%, +10.6%) but still PosCal outperforms MLE. Table 4 shows examples that only PosCal predicts correctly, with corresponding  p of true label from MLE and PosCal (INCOR  COR cases in Table 3). The predicted probability  p should be greater than 50% if models predict the true label. In the first example of RTE dataset, two expressions from S1 and S2 (e.g, reduce the risk of dis-ease in S1 and health benefits in S2) make MLE confusing to predict, so  p of true label becomes slightly less than the borderline probability (e.g.,  p = 49 . 7% < 50% ), making incorrect prediction. Another example of RTE shows how the MLE fails to predict the true label since the model cannot learn the connection between the location of newspaper (e.g., Norway) and its name (e.g., Verden Gang). In the two cases from SPolite dataset, the level of politeness indicated on phrases (e.g., Not at all in the first case and Could you in the second case) is not captured well by MLE, so the model predicts the incorrect label. From our manual investigation above, we find that statistical knowledge about posterior probability helps correct  p while training PosCal, so making  p switch its prediction. For further analysis, we provide more examples in Appendix C. 5 Conclusion and Future Directions We propose a simple yet effective training technique called PosCal for better posterior calibration. Our experiments empirically show that PosCal can improve both the performance of classifiers and the quality of predicted posterior output compared to MLE-based classifiers. The theoretical underpinnings of our PosCal idea are not explored in detail here, but developing formal statistical support for these ideas constitutes interesting future work. Currently, we fix the bin size at 10 and then estimate q by calculating accuracy of p per bin. Estimating q with adaptive binning can be a potential alternative for the fixed binning. We thank Matt Gormley and the anonymous reviewers for their helpful comments and discussion.
abstain abstain abstain objective objective result result abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective abstain abstain result objective other other other other other other other other abstain objective other other other other other other other other other other other other other other objective other abstain other abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain other