Abstract Automatic dialogue coherence evaluation has attracted increasing attention and is crucial for developing promising dialogue systems. However, existing metrics have two major limitations: (a) they are mostly trained in a simpli-fied two-level setting (coherent vs. incoherent), while humans give Likert-type multi-level coherence scores, dubbed as quantifiable; (b) their predicted coherence scores cannot align with the actual human rating standards due to the absence of human guidance during training. To address these limitations, we propose Quanti fiable D ialogue C oherence E valuation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating standards. Specifically, QuantiDCE includes two training stages, Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) fine-tuning. During MLR pre-training, a new MLR loss is proposed for enabling the model to learn the coarse judgement of coherence degrees. Then, during KD fine-tuning, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated data. To advocate the generalizability even with limited finetuning data, a novel KD regularization is introduced to retain the knowledge learned at the pre-training stage. Experimental results show that the model trained by QuantiDCE presents stronger correlations with human judgements than the other state-of-the-art metrics. 1 1 Introduction Dialogue coherence, which requires a response to be fluent, consistent and context-related, is an essential property for developing promising dialogue  Corresponding Author. systems (Cervone et al., 2018). However, it is still challenging to evaluate the coherence of a response generated by a dialogue system. Although human evaluation is always considered as the most accurate way to evaluate the coherence, it is expensive and high-latency, which cannot meet the evaluation demand of the frequent development of dialogue systems. Therefore, automatic evaluation metrics are developed to serve as human proxies that can rapidly compute the dialogue coherence and return relatively accurate results. The current widely used metrics measure the lexical word-overlap between generated responses and reference responses, such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). However, they have been demonstrated to be biased and correlate poorly with human judgements since no semantic information is considered (Liu et al., 2016; Novikova et al., 2017). To overcome this issue, researchers turned to develop learnable metrics based on neural networks that incorporate the semantic information, such as RUBER (Tao et al., 2018), BERT-RUBER (Ghazarian et al., 2019) and GRADE (Huang et al., 2020). However, these metrics deviate from the actual human rating due to two limitations. First, they simplify the coherence evaluation task in a two-level setting, i.e., coherent or incoherent, by maximizing the differences between the positive coherent dialogues and the negative incoherent ones obtained by some negative sampling strategies. In contrast, humans usually adopt Likert scaling and give coherence scores from multiple levels like 1 to 5, as shown in Figure 1. Second, to avoid relying on large-scale human-annotated data, they are mostly trained in a purely unsupervised manner and cannot align with the human rating due to the absence of introducing the actual human rating standards during training. To address the above limitations, we propose a novel dialogue coherence metric training framework, named as Quanti fiable D ialogue C oherence E valuation (QuantiDCE). This framework consists of two training stages: Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) finetuning. At the MLR pre-training stage, a new multilevel ranking (MLR) loss is proposed for learning the coarse judgement of coherence degrees. Specifically, the MLR loss separates the context-response pairs with different coherence levels and compacts the pairs within the same level in one-dimensional score space. As a result, the pretrained model is able to distinguish different coherence-level dialogue responses for a given context and predicts more accurate coherence scores. At the KD finetuning stage, the pretrained model is further finetuned to learn the actual human rating standards with only very few human-annotated coherence scores. To mitigate overfitting into the scarce annotated data during fine-tuning, a novel knowledge distillation regularization loss is introduced to retain the knowledge learned at the pre-training stage, where the pretrained model (teacher) provides the soft targets for the model during fine-tuning (stu-dent). Experimental results show that the metric trained by our QuantiDCE obviously outperforms the other state-of-the-art metrics in terms of the Pearson, Spearman and Kendall correlations with human judgements by around 5% points on average. To summarize our contributions: 1) We propose QuantiDCE, a novel quantifiable training framework for dialogue coherence evaluation, which aims to align the automatic scores with the actual human rating standards via MLR pre-training and KD fine-tuning. To the best of our knowledge, it is the first attempt to consider the quantifiable problem for dialogue coherence evaluation. 2) Extensive experiments demonstrate the effectiveness of our QuantiDCE, which enables the trained metric to have obviously stronger correlations with human judgements than the other state-of-the-art metrics. Automatic Coherence Evaluation. The widely used automatic metrics, such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and ROUGE (Lin, 2004), use statistical rules to measure the degree of lexical word-overlap between generated responses and reference responses. However, these metrics have been demonstrated to correlate poorly with human judgments due to the absence of semantic information (Liu et al., 2016; Novikova et al., 2017). Therefore, the subsequent metrics are considered to incorporate the semantic information. For instance, BERTScore (Zhang et al., 2020) turns to measure the soft semantic word-overlap rather than the hard lexical word-overlap like BLEU. Moreover, learnable metrics encoding the semantic information have been attracting interests recently, which are trained in a supervised manner with large-scale human-annotated data, such as ADEM (Lowe et al., 2017), or trained in an unsupervised manner with automatically constructed data, such as RUBER (Tao et al., 2018) and BERT-RUBER (Ghazarian et al., 2019). Furthermore, the recently proposed coherence metric, GRADE (Huang et al., 2020), introduces the graph information of dialogue topic transitions and achieves the current state-of-the-art results. Note that these learnable metrics are trained in a two-level training objective to separate the coherent dialogues from the incoherent ones, while our QuantiDCE models the task in a multi-level setting which is closer to the actual human rating. Knowledge Distillation. Knowledge distillation (KD) is a method that transfers the knowledge from a large trained teacher model to a smaller student model by using the soft targets provided by the teacher (Hinton et al., 2015). In recent years, KD has been applied to many specific tasks (Sun et al., 2020; Wei et al., 2019; Kim and Rush, 2016; Sourty et al., 2020). Unlike these previous works, we use KD to retain knowledge learned at the pre-training Stage One: MLR Pre-training   BERTMLPBERTMLP Human rating Stage Two: KD Fine-tuning     Teacher model KD Student model Level-1 Centroid Score Level-2 Centroid Score Level-3 Centroid Score   Level-1 Scores   Dialogue Example Context Level-1 Level-2 Level-3 Response separation compactness Level-2 Scores Level-3 Scores  kd_   Separation loss Compactness Loss Ordering Loss Concat    Concat Concat Figure 2: The overall pipeline of our QuantiDCE, consisting of two training stages which are marked by the blue and the black one-way arrows. In this section, we present QuantiDCE, a two-stage framework for dialogue coherence metric learning, consisting of Multi-Level Ranking (MLR) pre-training and Knowledge Distillation (KD) finetuning. As illustrated in Figure 2, given a metric model M (Section 3.1), QuantiDCE enables M to learn multi-level representations for context-response pairs with different levels of coherence degrees during the pre-training stage (Section 3.2), and further to learn the rating standards of humans with only a fraction of data during the fine-tuning stage (Section 3.3). After these two training stages, the quantifiable gap between automatic metrics and humans can be obviously reduced. In our QuantiDCE framework, the metric model M is composed of: (1) an encoder network for encoding the input context-response pairs into features and (2) a predictor network for transforming the encoded features into coherence scores. Specifically, we adopt BERT (Devlin et al., 2019) as the encoder network and a multi-layer perceptron (MLP) as the predictor network. Given a context c = { c 1 ,    , c m } and a response r = { r 1 ,    , r n } where c i and r i are tokens of the context and the response respectively, the c and r are concatenated as { [CLS] , c 1 ,    , c m , [SEP] , r 1 ,    , r n , [SEP] } , denoted as [ c ; r ] . Then the coherence score  s of the response r w.r.t. the context c is predicted by:  s = MLP ( BERT ([ c ; r ])) , (1) where MLP is a three-layer fully-connected network in which the activation functions of the three layers are two exponential linear units (Clevert et al., 2016) and a sigmoid function, respectively. For learning the coarse judgement of coherence degrees without the direct supervision of score annotations, the model M is first pretrained by minimizing a new multi-level ranking (MLR) loss on a large-scale dialogue dataset. Concretely, the MLR loss is composed of a separation loss, a compactness loss and an ordering loss. R i = { ( r ji, 1 ,    , r ji,K ) } Lj =1 is a response set with L coherence levels 2 and K responses for each level, the model M is trained by minimizing the following MLR loss: where (cid:96) sepi , (cid:96) comi , and (cid:96) ordi refer to the separation loss, the compactness loss and the ordering loss of the i th example, respectively. The separation loss aims to separate the features of context-response pairs with different coherence levels by separating the coherence scores of the different pairs 3 . Moreover, to efficiently compute the loss, we first compute the centroids of the context-response pairs belonging to the same coherence level for the i th dialogue example, i.e., e i = { e ji = (cid:80) Kk =1  s j i,k | j  [1 , L ] , e j i  R } where  s ji,k is the coherence score of the context-response pair ( c i , r ji,k ), and the separation loss between the centroids is then computed as follows: (cid:96) sepi = L  1 (cid:88) j =1 L (cid:88) l = j +1 max (0 , w    d( e ji , e li )) , (3) where d(  ) is the L1 distance,  is the lower bound for the distance between two centroids, and w = l  j is the distance weight used for amplifying the lower bound w.r.t. the coherence-level gap. The compactness loss aims to compact the pairs within the same level, which served as a regularization role to avoid the occurrence of outliers for each coherence level. Specifically, the coherence score  s j i,k is forced to be closer to the corresponding centroid e ji as follows: (cid:96) comi = L (cid:88) j =1 K (cid:88) k =1 max (0 , d( e ji ,  s ji,k )   ) , (4) where  is the upper bound for the distance between the centroid of a certain coherence level and the score within this level. 2 The coherence level is in ascending order, i.e., the response in a higher level is more coherent than the lower one. 3 We also tried to directly restrict the features of differentlevel pairs to be separated, but the performance dropped compared with restricting the scores. The ordering loss is finally introduced to assure that the rank order of the predicted scores satisfies the pre-defined order of coherence degrees, i.e.,  s ji,k <  s j +1 i,k , j  [1 , L  1] , k  [1 , K ] . It is critical since the separation loss only restricts the scores of the pairs from different coherence levels to be separated and this restriction is also satisfied when the scores of the highest level are lower than the scores of the lowest level. Similar to the separation loss, the ordering loss is also computed between each two centroids as follows: (cid:96) ordi = L  1 (cid:88) j =1 L (cid:88) l = j +1 max (0 , e li  e ji ) . The model M pretrained by the MLR loss is further trained at the KD fine-tuning stage to directly learn the actual human rating standards with only a fraction of annotated data. Formally, given a training dataset D ft = { ( c i , r i , s i ) } N 2 i =1 where c i , r i and s i are the dialogue context, the corresponding response and the human-annotated coherence score of r i w.r.t. c i respectively, the previous fine-tuning approach for the scoring task usually optimizes the model M with an MSE loss between the predicted score  s i and the human score s i : (cid:96) msei = ( s i   s i ) 2 . However, by minimizing (cid:96) msei for each example, the model M will be easily over-fitting on the very few annotated data, and thus the model generalizability will be dramatically reduced. To overcome this issue, a novel knowledge distillation (KD) regularization is introduced for retaining the knowledge learned at the MLR pre-training stage. Concretely, the pretrained model M is treated as the teacher model that provides the soft targets for the student model  M which is entirely copied from M . And we adopt the distillation objectives of TinyBERT (Jiao et al., 2020), including the distillations of the embedding layer, the Transformer layers and the prediction layer. The KD loss is then formulated as: (cid:96) kdi = T +1 (cid:88) t =0 || O ti   O ti || 22 + T (cid:88) t =1 || A ti   A ti || 22 , (7) where ||  || 22 indicates the squared L2 norm, T is the number of the Transformer layers, O ti and  O ti are Algorithm 1 Training Procedure of QuantiDCE Input: training datasets D pt and D ft , metric model M Output: student model  M 1: initialize M with BERTBASE 2: for all ( c i , R i )  D pt do 3: S i = M ( c i , R i ) 4: compute the centroids e i for S i 5: compute (cid:96) sepi and (cid:96) ordi for e i 6: compute (cid:96) comi between e i and S i 7: compute L mlr 8: update M to minimize L mlr 9: end for 10: initialize  M with M 11: for all ( c i , r i , s i )  D ft do 12: O i , A i = M ( c i , r i ) 13:  s i ,  O i ,  A i =  M ( c i , r i ) 14: compute (cid:96) msei between s i and  s i 15: compute (cid:96) kdi between O i , A i and  O i ,  A i 16: compute L kd mse 17: update  M to minimize L kd mse 18: end for 19: return student model  M the t th layer outputs of M and  M respectively, A ti and  A ti are the attention matrices of the t th transformer layer. Note that the layer 0 and the layer T+1 refer to the embedding layer and the prediction layer respectively. Overall, the loss function for KD fine-tuning, named as KD-MSE loss, is the weighted sum of (cid:96) msei and (cid:96) kdi across the whole training dataset D ft : L kd mse = 1 N 2 N 2 (cid:88) i =1 (   (cid:96) msei +   (cid:96) kdi ) , (8) where  and  are hyperparameters, and we empirically found that  = 1 and  = 5 performs well. The overall training procedure is summarized in Algorithm 1. 4 Experiments 4.1 Experimental Setup Baseline Metrics. We compare the metric model trained by our QuantiDCE with eight popular automatic dialogue metrics, including three lexical word-overlap metrics: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005), one semantic word-overlap metric, BERTScore (Zhang et al., 2020), and four learnable metrics: ADEM (Lowe et al., 2017), BERT-RUBER (Ghazarian et al., 2019), BLEURT (Sellam et al., 2020) and GRADE (Huang et al., 2020). Evaluation. Our QuantiDCE and the baselines are evaluated by computing the correlations between the model-predicted scores and the human-rated scores. Specifically, we adopt Pearson, Spearman and Kendall as the correlation measures and a large-scale human judgement benchmark (Huang et al., 2020) to provide the human-rated scores. This benchmark contains 1,200 unique (context, response, human-rated score) triplets for metric evaluation where the contexts were randomly selected from the test set of three chit-chat datasets including DailyDialog (Li et al., 2017), ConvAI2 (Dinan et al., 2019) and EmpatheticDialogues (Rashkin et al., 2019), and the responses were produced by both the retrieval-based dialogue models and the generation-based ones to assure response diversity. Training Datasets. We use two datasets, Daily-Dialog++ 4 and DailyDialogEVAL 5 , to support the pre-training and fine-tuning of QuantiDCE, respectively. The DailyDialog++ dataset (Sai et al., 2020) contains over 11K conversations, which augments the original DailyDialog dataset with multiple responses of different quality levels including five golden reference responses, five adversarial irrelevant responses and five random selected responses for each context. Therefore, in this work, we set the number of coherence levels L = 3 where the pairs containing the random responses, the adversarial responses and the reference responses respectively belong to the levels from 1 to 3. As to the fine-tuning data, we use the DailyDialog human judgement dataset, denoted as DailyDialogEVAL, which is a subset of the adopted evaluation benchmark (Huang et al., 2020), with 300 human rating data in total, and randomly split the data into training (90%) and validation (10%) sets. Implementation Details. We use BERTBASE to initialize the encoder network, which is in line with the current SOTA metric, GRADE. For the MLR pre-training, we pretrain our model for 5 epochs with batch size 3 and learning rate 2e-5 where the lower bound for the separation loss  = 0.3 and the upper bound for the compactness loss 4 https://github.com/iitmnlp/ Dialogue-Evaluation-with-BERT 5 https://github.com/li3cmz/GRADE Metric Pearson Spearman Kendall Average ConvAI2 BLEU 0.003 * 0.128 0.088 0.073 ROUGE 0.136 0.140 0.097 0.124 METEOR 0.145 0.181 0.123 0.15 BERTScore 0.225 0.225 0.154 0.201 ADEM 0.026 * 0.037 * 0.049 * 0.037 BERT-RUBER 0.266 0.266 0.185 0.239 BLEURT 0.152 0.149 0.103 0.135 GRADE 0.496 0.503 0.356 0.452 QuantiDCE 0.554 0.554 0.395 0.501 EmpatheticDialogues BLEU -0.051 * 0.002 * 0.005 * -0.015 ROUGE 0.029 * -0.013 * -0.010 * 0.002 METEOR 0.118 0.055 * 0.04 * 0.071 BERTScore 0.046 * 0.033 * 0.021 * 0.033 ADEM 0.007 * 0.009 * 0.040 * 0.019 BERT-RUBER -0.022 * -0.040 * -0.029 * -0.030 BLEURT 0.203 0.192 0.13 0.175 GRADE 0.350 0.344 0.243 0.312 QuantiDCE 0.412 0.393 0.274 0.360 Table 1: Correlations between automatic evaluation metrics and human judgements on two datasets (Con-vAI2 and EmpatheticDialogues). = 0.1. For the KD fine-tuning, we further fine-tune the pretrained model for 20 epochs with batch size 10 and learning rate 5e-6. For all the training, BERTAdam is used as the optimizer with  1 = 0 . 9 and  2 = 0 . 999 . For the Transformer-layer distillation, we distill all the Transformer layers since the model architectures of the teacher and the student are exactly the same. Metric Performance. The correlation results of QuantiDCE and the other baseline metrics on the large-scale human judgement benchmark are presented in Table 1, including the ConvAI2 and the EmpatheticDialogues datasets. 6 For a fair comparison, the learnable baseline metrics, ADEM, BERT-RUBER and GRADE, are trained on the training dataset we adopted, i.e., DailyDialog++. 7 Generally, QuantiDCE achieves an absolute averaged correlation improvement by around 5% points over the current SOTA, GRADE. Besides, all the results of QuantiDCE are statistically significant with p-value < 0.01. evaluation since we used it for fine-tuning. 7 BLEURT was not trained on DailyDialog++ since this dataset is not suitable for the BLEURT pre-training strategy. Instead, we trained BLEURT with the fine-tuning data we used. The training details of these baseline metrics are provided in Appendix A. Loss Pearson Spearman Kendall Average ConvAI2 BCE 0.505 0.505 0.361 0.457 Ranking 0.507 0.504 0.360 0.457 SupCon 0.495 0.523 0.367 0.462 FAT 0.516 0.521 0.371 0.469 Vanilla MLR 0.522 0.536 0.379 0.479 MLR (ours) 0.554 0.554 0.395 0.501 EmpatheticDialogues BCE 0.354 0.353 0.243 0.317 Ranking 0.399 0.389 0.272 0.353 SupCon 0.332 0.315 0.22 0.289 FAT 0.381 0.358 0.245 0.328 Vanilla MLR 0.403 0.387 0.267 0.352 MLR (ours) 0.412 0.393 0.274 0.360 Table 2: Correlations between human judgements and the metric models trained with different losses during pre-training and the same KD-MSE loss during finetuning. Pre-Training Objective. To verify the superiority of our pre-training objective, namely the MLR loss, we investigated the performance of several existing loss functions for pre-training compared with ours. Specifically, two categories of loss functions used for metric training are adopted, including (a) the two-level setting and (b) the multi-level setting. The binary cross entropy (BCE) loss and the margin ranking loss are adopted for the two-level setting, while another three loss functions are adopted for the multi-level setting, including the supervised contrastive (SupCon) loss (Khosla et al., 2020), the fast-approximated triplet (FAT) loss (Yuan et al., 2019) and the vanilla MLR loss (Lin et al., 2020) 8 . As shown in Table 2, the performance of our MLR loss is the best among all the pre-training objectives. And we also found that the multi-level setting losses perform better than the two-level ones, especially on the ConvAI2 dataset. Moreover, in order to more intuitively analyze the performances of these pre-training objectives, we also visualize the encoded features and the predicted scores of the model M after being pretrained by the above loss functions on the DailyDialog++ dataset without fine-tuning. 9 As shown in Figure 3, (a) the BCE loss cannot separate the level-1 scores from the level-2 ones and the corresponding features are also mixed; (b) the FAT loss, on the other hand, separates the features of different levels well, but does not consider the relative gaps where the distances between the level-1 and level-3 features are 8 The details of these pre-training loss fucntions are provided in Appendix B. 9 The visualization results of the ranking loss, SupCon loss and Vanilla MLR loss are provided in Appendix C. not larger than those between level-1 and level-2; (c) in contrast, our MLR loss separates both the features and the scores well and also considers the relative gaps between different levels. Fine-Tuning Objective. Furthermore, we also verified the effectiveness of our KD-MSE loss during fine-tuning by comparing with other fine-tuning losses, including the pure MSE loss without KD regularization as shown in Equation 6 and the same MSE loss except for freezing the encoder network and only finetuning the predictor network i.e. the MLP, denoted as MSE (fix encoder). As the results shown in Table 3, compared with the other two losses, the model finetuned by our KD-MSE loss has the highest correlation results on both ConvAI2 and EmpatheticDialogues. Moreover, by compar-0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Score 3 2 1 L e v e l Figure 4: Score visualization on the dailydialog++ dataset where the scores are predicted by our QuantiDCE after KD fine-tuning. ing the results of MSE and KD-MSE, we can find that introducing KD regularization leads to obvious averaged correlation improvements by 20.2% points on ConvAI2 and 11.3% points on EmpatheticDialogues, which verifies the effectiveness of the KD loss. Besides, we also reported the last-epoch correlation results on the training dataset, DailyDialogEVAL. And the results of MSE and MSE (fix encoder) indicate the phenomena of overfitting and under-fitting into DailyDialogEVAL respectively, which explain the reasons of their low performance on the two evaluation datasets. In contrast, our KD-MSE loss enables the model to learn the actual human rating standards from the scarce annotated data and avoid overfitting it si-Metric Pearson Spearman Kendall Average QuantiDCE 0.554 0.554 0.395 0.501 w/o MLR pre-training 0.373 0.357 0.246 0.325 w/o separation loss 0.388 0.416 0.289 0.364 w/o compactness loss 0.526 0.550 0.390 0.489 w/o ordering loss -0.494 -0.522 -0.371 -0.462 w/o KD fine-tuning 0.531 0.540 0.381 0.484 Table 4: Ablation studies on the ConvAI2 dataset by removing one of the component in QuantiDCE, including the MLR loss (w/o MLR pre-training), the KD+MSE loss (w/o KD fine-tuning), and three secondary losses of the MLR loss. multaneously. Finally, in Figure 4, we present the visualization of the scores predicted by our QuantiDCE after KD fine-tuning. Compared with the score distributions before fine-tuning in Figure 3(c), the finetuned score distributions of the level-1 and level-3 are wider and partly overlap with the level-2 distribution. It is predictable as the judgements of coherence are always subjective and humans tend to give vague and middle scores instead of extremely high or low scores. Component Analysis. To verify the contributions of the core components in our QuantiDCE, we further conducted ablation studies on the ConvAI2 dataset. As shown in Table 4, both the MLR pretraining and KD fine-tuning contribute to the better performance of QuantiDCE. Besides, we also conducted ablations by removing one of the secondary loss during MLR pre-training, including the separation loss, the compactness loss and the ordering loss. The results show that the performance ben-efits from all these losses in which the separation loss and the ordering loss are crucial for training a metric with strong and positive human correlations. Number of Data for Fine-Tuning. Moreover, we also investigated how the scale of data for finetuning effects the model performance by increasing the number of fine-tuning data 5% each time from zero. The trend of the model performance is presented in Figure 5. We observed that minimizing our KD-MSE loss made the correlation results have a gradually increasing trend after an initial decrease. 10 More specifically, the result achieved the standard before fine-tuning at around the 70% data scale and continued increasing until 100% with a final improvement by around 2% points. For comparison, the performance trends of MSE and MSE (fix encoder) are also provided. And the results present overall decreasing trends of the model performance, which indicates that the model trained by MSE or MSE (fix encoder) cannot benefit from the increasing of data scale, due to the severe overfitting or under-fitting. Therefore, to effectively utilize the limited data, it is important to enable the update of the entire network and add some constraints to avoid over-fitting, such as our proposed KD regularization. To illustrate the performance of QuantiDCE, two representative examples are shown in Table 5 . The first example shows the strength of QuantiDCE where the coherence score given by ours is closer to the human rating score compared with the extremely high score given by GRADE. However, in the second example, both our QuantiDCE and GRADE deviate from the human score, possibly because the number of coherence levels we adopted in this work ( L = 3) is insufficient as humans usually consider more levels of dialogue coherence. 10 The initial decrease probably attributes to the randomness of data sampling where the smaller the sampling ratio is, the higher the probability that noisy samples dominate the sampled data will be. And overfitting into the noisy samples leads to the performance decrease. In this paper, we propose QuantiDCE, a novel training framework aiming to bridge the gap between the training objective and the actual human rating and train a quantifiable dialogue coherence metric. In general, QuantiDCE includes two training stages, MLR pre-training for learning the coarse human judgements of dialogue coherence degrees, and KD fine-tuning for learning the actual human rating standards. Experimental results show that the metric trained by QuantiDCE presents strong correlations with human judgements. For future work, it is interesting to investigate a more efficient way to obtain multi-level data and extend the multilevel setting into the general evaluation for natural language generation. We thank all anonymous reviewers for their constructive comments. This work was supported in part by National Key R&D Program of China under Grant No. 2020AAA0109700, National Natural Science Foundation of China (NSFC) under Grant No.U19A2073 and No.61976233, Guangdong Province Basic and Applied Basic Research (Regional Joint Fund-Key) Grant No.2019B1515120039, Shenzhen Fundamental Research Program (Project No. RCYX20200714114642083, No. JCYJ20190807154211365), Zhijiang Lab's Open Fund (No. 2020AA3AB14) and CSIG Young Fellow Support Fund.
abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain result objective objective objective other other other other other other other objective other other other method method abstain abstain method method other abstain abstain abstain abstain abstain abstain other abstain other abstain method abstain abstain abstain abstain other abstain abstain abstain abstain other abstain other abstain method abstain method method other abstain method abstain abstain method abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain other abstain result abstain abstain abstain abstain abstain abstain abstain result result abstain result result abstain other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain objective abstain abstain abstain other other