Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality (Liu et al., 2016). Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem. We present an evaluation model (ADEM) that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model’s predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. We also show that ADEM can generalize to evaluating dialogue models unseen during training, an important step for automatic dialogue evaluation.
1 Introduction
Building systems that can naturally and meaningfully converse with humans has been a central goal of artificial intelligence since the formulation of the Turing test (Turing, 1950). Research on one type of such systems, sometimes referred to as non-taskoriented dialogue systems, goes back to the mid60s with Weizenbaum’s famous program ELIZA: a rule-based system mimicking a Rogerian psychotherapist by persistently either rephrasing statements or asking questions (Weizenbaum, 1966). Recently, there has been a surge of interest in the
research community towards building large-scale non-task-oriented dialogue systems using neural networks (Sordoni et al., 2015b; Shang et al., 2015; Vinyals and Le, 2015; Serban et al., 2016a; Li et al., 2015). These models are trained in an end-to-end manner to optimize a single objective, usually the likelihood of generating the responses from a fixed corpus. Such models have already had a substantial impact in industry, including Google’s Smart Reply system (Kannan et al., 2016), and Microsoft’s Xiaoice chatbot (Markoff and Mozur, 2015), which has over 20 million users.
One of the challenges when developing such systems is to have a good way of measuring progress, in this case the performance of the chatbot. The Turing test provides one solution to the evaluation of dialogue systems, but there are limitations with its original formulation. The test requires live human interactions, which is expensive and difficult to scale up. Furthermore, the test requires carefully designing the instructions to the human interlocutors, in order to balance their behaviour and expectations so that different systems may be ranked accurately by performance. Although unavoidable, these instructions introduce bias into the evaluation measure. The more common approach of having humans evaluate the quality of dialogue system
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
responses, rather than distinguish them from human responses, induces similar drawbacks in terms of time, expense, and lack of scalability. In the case of chatbots designed for specific conversation domains, it may also be difficult to find sufficient human evaluators with appropriate background in the topic (e.g. Lowe et al. (2015)).
Despite advances in neural network-based models, evaluating the quality of dialogue responses automatically remains a challenging and understudied problem in the non-task-oriented setting. The most widely used metric for evaluating such dialogue systems is BLEU (Papineni et al., 2002), a metric measuring word overlaps originally developed for machine translation. However, it has been shown that BLEU and other word-overlap metrics are biased and correlate poorly with human judgements of response quality (Liu et al., 2016). There are many obvious cases where these metrics fail, as they are often incapable of considering the semantic similarity between responses (see Figure 1). Despite this, many researchers still use BLEU to evaluate their dialogue models (Ritter et al., 2011; Sordoni et al., 2015b; Li et al., 2015; Galley et al., 2015; Li et al., 2016a), as there are few alternatives available that correlate with human judgements. While human evaluation should always be used to evaluate dialogue models, it is often too expensive and time-consuming to do this for every model specification (for example, for every combination of model hyperparameters). Therefore, having an accurate model that can evaluate dialogue response quality automatically — what could be considered an automatic Turing test — is critical in the quest for building human-like dialogue agents.
To make progress towards this goal, we make the simplifying assumption that a ‘good’ chatbot is one whose responses are scored highly on appropriateness by human evaluators. We believe this is sufficient for making progress as current dialogue systems often generate inappropriate responses. We also find empirically that asking evaluators for other metrics results in either low inter-annotator agreement, or the scores are highly correlated with appropriateness (see supp. material). Thus, we collect a dataset of appropriateness scores to various dialogue responses, and we use this dataset to train an automatic dialogue evaluation model (ADEM). The model is trained in a semi-supervised manner using a hierarchical recurrent neural network (RNN) to predict human scores.
# Examples 4104 # Contexts 1026 # Training examples 2,872 # Validation examples 616 # Test examples 616 κ score (inter-annotator 0.63 correlation)
Table 1: Statistics of the dialogue response evaluation dataset. Each example is in the form (context, model response, reference response, human score).
We show that ADEM scores correlate significantly with human judgement at both the utterance-level and system-level. We also show that ADEM can often generalize to evaluating new models, whose responses were unseen during training, making ADEM a strong first step towards effective automatic dialogue response evaluation.1
2 Data Collection
To train a model to predict human scores to dialogue responses, we first collect a dataset of human judgements (scores) of Twitter responses using the crowdsourcing platform Amazon Mechanical Turk (AMT).2 The aim is to have accurate human scores for a variety of conversational responses — conditioned on dialogue contexts — which span the full range of response qualities. For example, the responses should include both relevant and irrelevant responses, both coherent and non-coherent responses and so on. To achieve this variety, we use candidate responses from several different models. Following (Liu et al., 2016), we use the following 4 sources of candidate responses: (1) a response selected by a TF-IDF retrieval-based model, (2) a response selected by the Dual Encoder (DE) (Lowe et al., 2015), (3) a response generated using the hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a), and (4) human-generated responses. It should be noted that the humangenerated candidate responses are not the reference responses from a fixed corpus, but novel human responses that are different from the reference. In addition to increasing response variety, this is necessary because we want our evaluation model to learn to compare the reference responses to the candidate responses. We provide the details of our AMT experiments in the supplemental material,
1We will provide open-source implementations of the model upon publication.
2All data collection was conducted in accordance with the policies of the host institutions’ ethics board.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
including additional experiments suggesting that several other metrics are currently unlikely to be useful for building evaluation models.
To train evaluation models on human judgements, it is crucial that we obtain scores of responses that lie near the distribution produced by advanced models. This is why we use the Twitter Corpus (Ritter et al., 2011), as such models are pre-trained and readily available. Further, the set of topics discussed is quite broad — as opposed to the very specific Ubuntu Dialogue Corpus — and therefore the model may also be suited to other chit-chat domains. Finally, since it does not require domain specific knowledge (e.g. technical knowledge), it should be easy for AMT workers to annotate.
3 Technical Background
3.1 Recurrent Neural Networks
Recurrent neural networks (RNNs) are a type of neural network with time-delayed connections between the internal units. This leads to the formation of a hidden state ht, which is updated for every input: ht = f(Whhht−1 +Wihxt), where Whh and Wih are parameter matrices, f is a non-linear activation function such as tanh, and xt is the input at time t. The hidden state allows for RNNs to better model sequential data, such as language.
In this paper, we consider RNNs augmented with long-short term memory (LSTM) units (Hochreiter and Schmidhuber, 1997). LSTMs add a set of gates to the RNN that allow it to learn how much to update the hidden state. LSTMs are one of the most well-established methods for dealing with the vanishing gradient problem in recurrent networks (Hochreiter, 1991; Bengio et al., 1994).
3.2 Word-Overlap Metrics
One of the most popular approaches for automatically evaluating the quality of dialogue responses is by computing their word overlap with the reference response. In particular, the most popular metrics are the BLEU and METEOR scores used for machine translation, and the ROUGE score used for automatic summarization. While these metrics tend to correlate with human judgements in their target domains, they have recently been shown to highly biased and correlate very poorly with human judgements for dialogue response evaluation (Liu et al., 2016). We briefly describe BLEU here, and provide a more detailed summary of word-overlap metrics in the supplemental material.
BLEU BLEU (Papineni et al., 2002) analyzes the co-occurrences of n-grams in the reference and the proposed responses. It computes the n-gram precision for the whole dataset, which is then multiplied by a brevity penalty to penalize short translations. For BLEU-N , N denotes the largest value of ngrams considered (usually N = 4).
Drawbacks One of the major drawbacks of word-overlap metrics is their failure in capturing the semantic similarity between the model and reference responses when there are few or no common words. This problem is less critical for machine translation; since the set of reasonable translations of a given sentence or document is rather small, one can reasonably infer the quality of a translated sentence by only measuring the word-overlap between it and one (or a few) reference translations. However, in dialogue, the set of appropriate responses given a context is much larger (Artstein et al., 2009); in other words, there is a very high response diversity that is unlikely to be captured by word-overlap comparison to a single response.
Further, word-overlap scores are computed directly between the model and reference responses. As such, they do not consider the context of the conversation. While this may be a reasonable assumption in machine translation, it is not the case for dialogue; whether a model response is an adequate substitute for the reference response is clearly context-dependent. For example, the two responses in Figure 1 are equally appropriate given the context. However, if we simply change the context to: “Have you heard of any good movies recently?”, the model response is no longer relevant while the reference response remains valid.
4 An Automatic Dialogue Evaluation
Model (ADEM)
To overcome the problems of evaluation with wordoverlap metrics, we aim to construct a dialogue evaluation model that: (1) captures semantic similarity beyond word overlap statistics, and (2) exploits both the context and the reference response to calculate its score for the model response. We call this evaluation model ADEM.
ADEM learns distributed representations of the context, model response, and reference response using a hierarchical RNN encoder. Given the dialogue context c, reference response r, and model response r̂, ADEM first encodes each of them into vectors (c, r̂, and r, respectively) using the RNN
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
encoder. Then, ADEM computes the score using a dot-product between the vector representations of c, r, and r̂ in a linearly transformed space: :
score(c, r, r̂) = (cTM r̂+ rTN r̂− α)/β (1)
where M,N ∈ Rn are learned matrices initialized to the identity, and α, β are scalar constants used to initialize the model’s predictions in the range [1, 5]. The model is shown in Figure 2.
The matrices M and N can be interpreted as linear projections that map the model response r̂ into the space of contexts and reference responses, respectively. The model gives high scores to responses that have similar vector representations to the context and reference response after this projection. The model is end-to-end differentiable; all the parameters can be learned by backpropagation. In our implementation, the parameters θ = {M,N} of the model are trained to minimize the squared error between the model predictions and the human score, with L2-regularization:
L = ∑
i=1:K
[score(ci, ri, r̂i)− humani]2 + γ||θ||2
(2) where γ is a scalar constant. The simplicity of our model leads to both accurate predictions and fast evaluation (see supp. material), which is important to allow rapid prototyping of dialogue systems.
The hierarchical RNN encoder in our model consists of two layers of RNNs (El Hihi and Bengio, 1995; Sordoni et al., 2015a). The lower-level RNN, the utterance-level encoder, takes as input words from the dialogue, and produces a vector output at the end of each utterance. The context-level encoder takes the representation of each utterance as input and outputs a vector representation of the context. This hierarchical structure is useful for incorporating information from early utterances in the context (Serban et al., 2016a). Following previous work, we take the last hidden state of the
context-level encoder as the vector representation of the input utterance or context.
An important point is that the ADEM procedure above is not a dialogue retrieval model: the fundamental difference is that ADEM has access to the reference response. Thus, ADEM can compare a model’s response to a known good response, which is significantly easier than inferring response quality from solely the context.
Pre-training with VHRED We would like an evaluation model that can make accurate predictions from few labeled examples, since these examples are expensive to obtain. We therefore employ semi-supervised learning, and use a pre-training procedure to learn the parameters of the encoder. In particular, we train the encoder as part of a neural dialogue model; we attach a third decoder RNN that takes the output of the encoder as input, and train it to predict the next utterance of a dialogue conditioned on the context.
The dialogue model we employ for pre-training is the latent variable hierarchical recurrent encoderdecoder (VHRED) model (Serban et al., 2016b). The VHRED model is an extension of the original hierarchical recurrent encoder-decoder (HRED) model (Serban et al., 2016a) with a turn-level stochastic latent variable. The dialogue context is encoded into a vector using our hierarchical encoder, and the VHRED then samples a Gaussian variable that is used to condition the decoder (see supplemental material for further details). After training VHRED, we use the last hidden state of the context-level encoder, when c, r, and r̂ are fed as input, as the vector representations for c, r, and r̂, respectively. We use representations from the VHRED model as it produces more diverse and coherent responses compared to HRED.
Maximizing the likelihood of generating the next utterance in a dialogue is not only a convenient way
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
of training the encoder parameters; it is also an objective that is consistent with learning useful representations of the dialogue utterances. Two context vectors produced by the VHRED encoder are similar if the contexts induce a similar distribution over subsequent responses; this is consistent with the formulation of the evaluation model, which assigns high scores to responses that have similar vector representations to the context. VHRED is also closely related to the skip-thought-vector model (Kiros et al., 2015), which has been shown to learn useful representations of sentences for many tasks, including semantic relatedness and paraphrase detection. The skip-thought-vector model takes as input a single sentence and predicts the previous sentence and next sentence. On the other hand, VHRED takes as input several consecutive sentences and predicts the next sentence. This makes it particularly suitable for learning long-term context representations.
5 Experiments
5.1 Experimental Procedure
In order to reduce the effective vocabulary size, we use byte pair encoding (BPE) (Gage, 1994; Sennrich et al., 2015), which splits each word into sub-words or characters. We also use layer normalization (Ba et al., 2016) for the hierarchical encoder, which we found worked better at the task of dialogue generation than the related recurrent batch normalization (Ioffe and Szegedy, 2015; Cooijmans et al., 2016). To train the VHRED model, we employed several of the same techniques found in (Serban et al., 2016b) and (Bowman et al., 2016): we drop words in the decoder with a fixed rate of 25%, and we anneal the KL-divergence term linearly from 0 to 1 over the first 60,000 batches. We use Adam as our optimizer (Kingma and Ba, 2014).
When training ADEM, we also employ a subsampling procedure based on the model response length. In particular, we divide the training examples into bins based on the number of words in a response and the score of that response. We then over-sample from bins across the same score to ensure that ADEM does not use response length to predict the score. This is because humans have a tendency to give a higher rating to shorter responses than to longer responses (Serban et al., 2016b), as shorter responses are often more generic and thus are more likely to be suitable to the context. Indeed, the test set Pearson correlation between response
length and human score is 0.27.
For training VHRED, we use a context embedding size of 2000. However, we found the ADEM model learned more effectively when this embedding size was reduced. Thus, after training VHRED, we use principal component analysis (PCA) (Pearson, 1901) to reduce the dimensionality of the context, model response, and reference response embeddings to n. We found experimentally that n = 50 provided the best performance.
When training our models, we conduct early stopping on a separate validation set. For the evaluation dataset, we split the train/ validation/ test sets such that there is no context overlap (i.e. the contexts in the test set are unseen during training).
5.2 Results
Utterance-level correlations We first present new utterance-level correlation results3 for existing word-overlap metrics, in addition to results with embedding baselines and ADEM, in Table 2. The baseline metrics are evaluated on the entire dataset of 4,104 responses.4 We measure the correlation for ADEM on the validation and test sets.
We also conduct an analysis of the response data from (Liu et al., 2016), where the pre-processing is standardized by removing ‘<first speaker>’ tokens at the beginning of each utterance. The results are detailed in the supplemental material. We can observe from both this data, and the new data in Table 2, that the correlations for the word-overlap metrics are even lower than estimated in previous studies (Liu et al., 2016; Galley et al., 2015). In particular, this is the case for BLEU-4, which has frequently been used for dialogue response evaluation (Ritter et al., 2011; Sordoni et al., 2015b; Li et al., 2015; Galley et al., 2015; Li et al., 2016a).
We can see from Table 2 that ADEM correlates far better with human judgement than the wordoverlap baselines. This is further illustrated by the scatterplots in Figure 3. We also compare with ADEM using tweet2vec embeddings for c, r, and r̂, which are computed at the character-level with a bidirectional GRU (Dhingra et al., 2016), and ob-
3We present both the Spearman correlation (computed on ranks, depicts monotonic relationships) and Pearson correlation (computed on true values, depicts linear relationships) scores.
4Note that our word-overlap correlation results in Table 2 are also lower than those presented in (Galley et al., 2015). This is because Galley et al. measure corpus-level correlation, i.e. correlation averaged across different subsets (of size 100) of the data, and pre-filter for high-quality reference responses.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
(a) BLEU-2 (b) ROUGE (c) ADEM
Figure 3: Scatter plot showing model against human scores, for BLEU-2 and ROUGE on the full dataset, and ADEM on the test set. We add Gaussian noise drawn from N (0, 0.3) to the integer human scores to better visualize the density of points, at the expense of appearing less correlated.
Full dataset Test set Metric Spearman Pearson Spearman Pearson BLEU-2 0.039 (0.013) 0.081 (<0.001) 0.051 (0.254) 0.120 (<0.001) BLEU-4 0.051 (0.001) 0.025 (0.113) 0.063 (0.156) 0.073 (0.103) ROUGE 0.062 (<0.001) 0.114 (<0.001) 0.096 (0.031) 0.147 (<0.001) METEOR 0.021 (0.189) 0.022 (0.165) 0.013 (0.745) 0.021 (0.601) T2V 0.140 (<0.001) 0.141 (<0.001) 0.140 (<0.001) 0.141 (<0.001) VHRED -0.035 (0.062) -0.030 (0.106) -0.091 (0.023) -0.010 (0.805) Validation set Test set C-ADEM 0.338 (<0.001) 0.355 (<0.001) 0.366 (<0.001) 0.363 (<0.001) R-ADEM 0.404 (<0.001) 0.404 (<0.001) 0.352 (<0.001) 0.360 (<0.001) ADEM (T2V) 0.252 (<0.001) 0.265 (<0.001) 0.280 (<0.001) 0.287 (<0.001) ADEM 0.410 (<0.001) 0.418 (<0.001) 0.428 (<0.001) 0.436 (<0.001)
Table 2: Correlation between metrics and human judgements, with p-values shown in brackets. ‘ADEM (T2V)’ indicates ADEM with tweet2vec embeddings (Dhingra et al., 2016), and ‘VHRED’ indicates the dot product of VHRED embeddings (i.e. ADEM at initialization). C- and R-ADEM represent the ADEM model trained to only compare the model response to the context or reference response, respectively.
tain reasonable but inferior performance compared to using VHRED embeddings.
System-level correlations We show the systemlevel correlations for various metrics in Table 3, and present it visually in Figure 4. Each point in the scatterplots represents a dialogue model; humans give low scores to TFIDF and DE responses, higher scores to HRED and the highest scores to other human responses. It is clear that existing word-overlap metrics are incapable of capturing this relationship for even 4 models. This renders them completely deficient for dialogue evaluation. However, ADEM produces almost the same model ranking as humans, achieving a significant Pearson correlation of 0.954.5 Thus, ADEM correlates well with humans both at the response and system level.
Generalization to previously unseen models When ADEM is used in practice, it will take as input responses from a new model that it has not
5For comparison, BLEU achieves a system-level correlation of 0.99 on 5 models in the translation domain (Papineni et al., 2002).
seen during training. Thus, it is crucial that ADEM correlates with human judgements for new models. We test ADEM’s generalization ability by performing a leave-one-out evaluation. For each dialogue model that was the source of response data for training ADEM (TF-IDF, Dual Encoder, HRED, humans), we conduct an experiment where we train on all model responses except those from the chosen model, and test only on the model that was unseen during training.
The results are given in Table 4. We observe that the ADEM model is able to generalize for all models except the Dual Encoder. This is particularly surprising for the HRED model; in this case, ADEM was trained only on responses that were written by humans (from retrieval models or human-generated), but is able to generalize to responses produced by a generative neural network model. When testing on the entire test set, the model achieves comparable correlations to the ADEM model that was trained on 25% less data selected at random.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 4: Scatterplots depicting the system-level correlation results for ADEM, BLEU-2, BLEU-4,and ROUGE on the test set. Each point represents the average scores for the responses from a dialogue model (TFIDF, DE, HRED, human). Human scores are shown on the horizontal axis, with normalized metric scores on the vertical axis. The ideal metric has a perfectly linear relationship.
Metric Pearson BLEU-1 -0.079 (0.921) BLEU-2 0.308 (0.692) BLEU-3 -0.537 (0.463) BLEU-4 -0.536 (0.464) ROUGE 0.268 (0.732) ADEM 0.954 (0.046)
Table 3: System-level correlation, with the p-value in brackets.
Qualitative Analysis To illustrate some strengths and weaknesses of ADEM, we show human and ADEM scores for each of the responses to various contexts in Table 5. There are several instances where ADEM predicts accurately: in particular, ADEM is often very good at assigning low scores to poor responses. This seen in the first two contexts, where most of the responses given a score of 1 from humans are given scores less than 2 by ADEM. The single exception in response (4) for the second context seems somewhat appropriate and should perhaps have been scored higher by the human evaluator. There are also several instances where the model assigns high scores to suitable responses, as in the first two contexts.
One drawback we observed is that ADEM tends to be too conservative when predicting response scores. This is the case in the third context, where the model assigns low scores to most of the responses that a human rated highly. This behaviour is likely due to the squared error loss used to train ADEM; since the model receives a large penalty for incorrectly predicting an extreme value, it learns to predict scores closer to the average human score. We provide many more experiments, including a failure analysis, in the supplemental material.
6 Related Work
Related to our approach is the literature on novel methods for the evaluation of machine translation systems, especially through the WMT evaluation task (Callison-Burch et al., 2011; Machácek and
Bojar, 2014; Stanojevic et al., 2015). In particular, (Albrecht and Hwa, 2007; Gupta et al., 2015) have proposed to evaluate machine translation systems using Regression and Tree-LSTMs respectively. Their approach differs from ours as, in the dialogue domain, we must additionally condition our score on the context of the conversation, which is not necessary in translation.
Several recent approaches use hand-crafted reward features to train dialogue models using reinforcement learning (RL). For example, (Li et al., 2016b) use features related to ease of answering and information flow, and (Yu et al., 2016) use metrics related to turn-level appropriateness and conversational depth. These metrics are based on hand-crafted features, which only capture a small set of relevant aspects; this inevitably leads to suboptimal performance, and it is unclear whether such objectives are preferable over retrieval-based crossentropy or word-level maximum log-likelihood objectives. Furthermore, many of these metrics are computed at the conversation-level, and are not available for evaluating single dialogue responses. The metrics that can be computed at the responselevel could be incorporated into our framework, for example by adding a term to equation 1 consisting of a dot product between these features and a vector of learned parameters.
There has been significant work on evaluation methods for task-oriented dialogue systems, which attempt to solve a user’s task such as finding a restaurant. These methods include the PARADISE framework (Walker et al., 1997) and MeMo (Möller et al., 2006), which consider a task completion signal. Our models do not attempt to model task completion, and thus fall outside this domain.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Test on full dataset Test on removed model responses Data Removed Spearman Pearson Spearman Pearson TF-IDF 0.406 (<0.001) 0.409 (<0.001) 0.186 (0.021) 0.196 (0.015) Dual Encoder 0.364 (<0.001) 0.373 (<0.001) 0.026 (0.749) 0.027 (0.736) HRED 0.393 (<0.001) 0.396 (<0.001) 0.151 (0.060) 0.148 (<0.070) Human 0.292 (<0.001) 0.298 (<0.001) 0.216 (<0.010) 0.148 (<0.070) Average 0.364 0.369 0.145 0.130 25% at random 0.378 (<0.001) 0.384 (<0.001) — —
Table 4: Correlation for ADEM when various model responses are removed from the training set. The left two columns show performance on the entire test set, and the right two columns show performance on responses only from the dialogue model not seen during training. The last row (25% at random) corresponds to the ADEM model trained on all model responses, but with the same amount of training data as the model above (i.e. 25% less data than the full training set).
Context Reference response Model responses Human score ADEM score photo to see my television debut go to - some. some on <url> - hehe <url>→ it really was you? i thought ppl were recognizing someone who looked like you! were the oysters worth the wait? yeah it was me . haha i ’d kinda forgotten about it it was filmed a while ago 1) i’m not sure. i just don’t know what to do with it. 2) you heard the horsepower productions remix of lee scratch perry’s ‘exercising’ off his ‘mighty upsetter’ album? 3) you wont chug a fuzzy peach navel 4) they were! 3 1 1 5 1.602 1.513 1.744 3.274 just beat call of duty!! → want a cookie? → yes!! → come get it im in kenmore at the moment
1) i’m gonna get a new phone some moro 2) no way man. 3) wow i just got a free pizza coupon! get yours
before theres no more! <url> 4) i’m going to go to the mall.
1 5 1
1
1.848 4.265 0.921
2.634 am i out of twitter jail yet? testing → yeah. i posted bail → thanks. i am a right chatter tweetbox on sundays. same happened last sunday lol
any news on meeting our user ? i go to the us on friday and i don ’t want to miss anything arranged 1) i’m not sure if i’m going to be able to get it. 2) good to see another mac user in the leadership ranks 3) awww poor baby hope u get to feeling better soon. maybe some many work days at piedmont 4) did you tweet too much? 3 4 2 5
1.912 1.417
1.123 2.539
Table 5: Examples of scores given by the ADEM model.
7 Discussion
We use the Twitter Corpus to train our models as it contains a broad range of non-task-oriented conversations and has has been used to train many stateof-the-art models. However, our model could easily be extended to other general-purpose datasets, such as Reddit, once similar pre-trained models become publicly available. Such models are necessary even for creating a test set in a new domain, which will help us determine if ADEM generalizes to related dialogue domains. We leave investigating the domain transfer ability of ADEM for future work.
The evaluation model proposed in this paper favours dialogue models that generate responses that are rated as highly appropriate by humans. It is likely that this property does not fully capture the desired end-goal of chatbot systems. For example, one issue with building models to approximate human judgements of response quality is the problem of generic responses. Since humans often provide high scores to generic responses due to their appropriateness for many given contexts, a model trained to predict these scores will exhibit the same behaviour. An important direction for future work
is modifying ADEM such that it is not subject to this bias. This could be done, for example, by censoring ADEM’s representations (Edwards and Storkey, 2016) such that they do not contain any information about length. Alternatively, one can combine this with an adversarial evaluation model (Kannan and Vinyals, 2017; Li et al., 2017) that assigns a score based on how easy it is to distinguish the dialogue model responses from human responses. In this case, a model that generates generic responses will easily be distinguishable and obtain a low score.
An important direction of future research is building models that can evaluate the capability of a dialogue system to have an engaging and meaningful interaction with a human. Compared to evaluating a single response, this evaluation is arguably closer to the end-goal of chatbots. However, such an evaluation is extremely challenging to do in a completely automatic way. We view the evaluation procedure presented in this paper as an important step towards this goal; current dialogue systems are incapable of generating responses that are rated as highly appropriate by humans, and we believe our evaluation model will be useful for measuring and facilitating progress in this direction.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: This paper proposes an evaluation metric for automatically evaluating the quality of dialogue responses in non-task-oriented dialogue. The metric operates on continuous vector space representations obtained by using RNNs and it comprises two components: one that compares the context and the given response and the other that compares a reference response and the given response. The comparisons are conducted by means of dot product after projecting the response into corresponding context and reference response spaces. These projection matrices are learned by minimizing the squared error between the model predictions and human annotations.  I think this work gives a remarkable step forward towards the evaluation of non-task-oriented dialogue systems. Different from previous works in this area, where pure semantic similarity was pursued, the authors are going beyond pure semantic similarity in a very elegant manner by learning projection matrices that transform the response vector into both context and reference space representations. I am very curious on how your projection matrices M and N differ from the original identity initialization after training the models. I think the paper will be more valuable if further discussion on this is introduced, rather than focusing so much on resulting correlations.   - Weaknesses:  The paper also leaves lots questions related to the implementation. For instance, it is not clear whether the human scores used to train and evaluate the system were single AMT annotations or the resulting average of few annotations. Also, it is not clear how the dataset was split into train dev test and whether n-fold cross validation was conducted or not. Also, it would be nice to better explain why in table 2 correlation for ADEM related scores are presented for the validation and test sets, while for the other scores they are presented for the full dataset and test set. The section on pre-training with VHRED is also very clumsy and confusing, probably it is better to give less technical details but a better high level explanation of the pre-training strategy and its advantages.  - General Discussion:   There are many obvious cases where these metrics fail, as they are often incapable of considering the semantic similarity between responses (see Figure 1). d Be careful with statements like this one. This is not a problem of semantic similarity! Opposite to it, the problem is that completely different semantic cues might constitute pragmatically valid responses. Then, semantic similarity itself is not enough to evaluate a dialogue system response. Dialogue system response evaluation must go beyond semantics (This is actually what your M and N matrices are helping to do!!!)    an accurate model that can evaluate dialogue response quality automatically   what could be considered an automatic Turing test    The original intention of Turing test was to be a proxy to identify define intelligent behaviour. It actually proposes a test on intelligence based on an  intelligent d machine capability to imitate human behaviour in such a way that it would be difficult for a common human to distinguish between such a machine responses and actual human responses. It is of course related to dialogue system performance, but I think it is not correct to say that automatically evaluating dialogue response quality is an automatic Turing test. Actually, the title itself  Towards an Automatic Turing Test d is somehow misleading!   the simplifying assumption that a  good  chatbot is one whose responses are scored highly on appropriateness by human evaluators. d This is certainly the correct angle to introduce the problem of non-task-oriented dialogue systems, rather than  Turing Test d. Regarding this, there has been related work you might like to take a look at, as well as to make reference to, in the WOCHAT workshop series (see the shared task description and corresponding annotation guidelines).  In the discussion session:  and has has been used d to  and it has been used d


Exploring Macro Discourse Structure with Macro-micro Unified Primary-secondary Relationship
Discourse structure analysis is helpful for the machine to identify different types of discourse writing styles, and lays the foundation for the study of discourse automatic generation. In this paper, after studying the difference and the relationship between micro and macro discourse structures, we explore the macro discourse structure, and put forward a Macro Chinese Discourse Treebank (MCDTB) on the top of existing Chinese Discourse Treebank (CDTB), by unifying micro and macro discourse structures. Specifically, at the micro level, we put forward the primary-secondary relationship from the logical semantic perspective, while at the macro level, we put forward the primarysecondary relationship from the pragmatic function perspective. Preliminary experimentation shows that our macro-micro unified schema is appropriate for the discourse structure analysis.
1 Introduction
Discourse rarely exists isolated, so the interpretation of a discourse requires understanding of its structure and semantics. (De Beaugrande, 1981) As an instance, the discourse primary-secondary relationship is the relation between the primary content and secondary content within a discourse, or the relation between the primary and secondary aspects of a discourse and the others. Here, the primary content refers to the dominant position in the discourse, and plays a decisive role in the part, while the secondary content refers to the auxiliary position, and does not play a decisive role in the discourse.
In principle, the primary-secondary relationship
plays a critical role in natural language processing, since the recognition of the discourse primarysecondary relationship not only helps to understand the discourse structure and semantics, but also provides strong support for deep natural language processing applications. Despite of its potential in statistical machine translation (Meyer and Popescu-Belis, 2012; Guzmán et al., 2014; Peldszus and Stede, 2015), automatic text summarization (Atkinson and Munoz, 2013; Ferreira et al., 2014; Cohan and Goharian, 2015), question answering (Liakata et al., 2013), information extraction (Presutti et al., 2012; Zou et al., 2014), sentiment analysis (Mukherjee et al., 2012; Mittal et al., 2013; Bhatia et al., 2015) etc. recognition of the discourse primary-secondary relationship has become the bottleneck in discourse structure analysis recently, largely due to the ignorance of its critical role in discourse structure analysis by viewing it only as a dispensable component in the analysis of the discourse rhetorical structure.
Generally speaking, there exist two hierarchical levels of discourse structures: micro level and macro level. While micro structure refers to the relationship between the internal structures in a sentence or two consecutive sentences, the macro structure refers to the relationship between sentences, paragraphs and chapters, at a higher level.
At the micro level, the Rhetorical Structure Theory (RST) (Mann and Thompson, 1987), one of the most influential discourse theories, represents the discourse structure as a tree with phrases or clauses as elementary units(EDUs), where adjacent EDUs are connected by rhetorical relations, forming larger discourse units. Although RST establishes two different types of units, where nucleus are considered as the central parts, and satellites as peripheral ones. The nuclearity defined in RST is considered from the particular rhetorical relation, instead of the overall discourse, and with-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
189
190
191
192
193
194
195
196
197
198
199
out considering the overall discourse intention, the nuclearity is difficult to distinguish.
At the macro level, Van Dijk (1980) proposes the macrostructure theory, indicating that a text not only has local or microstructure relationship between subsequent sentences, but also has overall structure that defines their global coherence and organization. Although Van Dijk explores the macrostructure from both semantics and pragmatics, he focuses on the semantic aspect of the macrostructure, emphasizing that without the semantic macrostructure, it is unable to explain the meaning of the overall discourse and understand the partial coherence in the macrostructure. Unfortunately, no matter whether at micro level or at macro level, existing theories focus on logical semantics, with few from the pragmatic function perspective.
In this paper, we regard the recognition of primary-secondary relationship as an independent task from the discourse rhetorical structure analysis, and integrate the overall discourse with the primary-secondary relationship as the unified carrier. For this purpose, a macro discourse structure representation schema is proposed. At the micro level, we lay on the logical semantic relationship between the discourse units based on the rhetorical structure, while at the macro level, we lay on the pragmatic function of the content based on the schematic structure. Specifically, our work expands the discourse analysis from intra-paragraph to the overall discourse. Guided by the representation schema, we annotate 97 news wire articles both from the perspective of micro semantics and macro pragmatics. Table 1 compares the macro and micro structures. Preliminary experiments on the recognition of discourse relation and primarysecondary relation show that our macro-micro unified representation schema is appropriate for the discourse structure analysis tasks.
2 Related work
In this section, we first summarize theories wildly employed in discourse structure, then we present existing approaches to discourse structure analysis.
2.1 Discourse Theory
From micro perspective, discourse structure theories mainly include Cohesion Theory(Halliday, 1971), Hobbs Model(Hobbs, 1993, 1979), Rhetor-
ical Structure Theory(Mann and Thompson, 1986, 1987; Mann et al., 1992), Intention Structure Theory (Grosz and Sidner, 1986; Grosz et al., 1995), Chinese Complex Sentences Theory (Xing, 2001; Yao, 2006), Connective-driven Dependency Tree Theory (Li, 2015; Li et al., 2014) etc. From macro perspective, the theories of discourse structure are relatively few, including Super-theme Theory (Martin and Rose, 2003), Macrostructures Theory (Van Dijk, 1980, 1985, 2013) etc.
Cohesion Theory: Halliday (1971) pointed out in cohesion theory, “cohesion considers the interaction of cohesion with other aspects of text organization.” When Grimes (1975) depended the cohesion theory, he identified two types of functional relations: 1) paratactic, where all parts of a relation are equally prominent in terms of their discoursal functions; and 2) hypotactic, where one part of a relation is more prominent than the other part.
Rhetorical Structure Theory: RST (Mann and Thompson, 1986) provides a general way to describe the relations among the elementary discourse units, and establishes two different types of units, nucleus and satellite. Nuclei are considered as the most important parts of text whereas satellites contribute to the nuclei and are secondary. Figure 1 shows the rhetorical structure of three types of rhetorical relation: (a) evidence; (b) circumstance; and (c) sequence. The arrows point to the nuclei in the structure.
Connective-driven Dependency Tree: Li et al. (2014) proposed a Connective-driven Dependency Tree(CDT) scheme to represent discourse rhetorical structure, with EDUs as leaf nodes and connectives as non-leaf nodes, to accommodate the special characteristics of Chinese language. A discourse can be expressed to a complete discourse structure tree based on the CDT representation schema.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Perspective Structure Concern Description Micro Rhetorical Structure Logical semantics Structural analysis from the form Macro Schematic Structure Functional Pragmatics Functional analysis from the information
Table 1: Comparison of micro and macro structures
Macrostructure Theory: The point of macrostructures is that texts not only have local or micro structural relations between subsequent sentences, but also have overall structures that define their global coherence and organization (Van Dijk, 1980). Figure 2 shows Van Dijk’s macrostructure theory. In this marcostructure, Pi represents a lower layer proposition, and Mkj represents a marcostructure unit, in which the superscript k represents the hierarchy, while the subscript j indicates the unit order on current layer.
M1 n
M1 n-1 M2 n-1 M3 n-1
M1 n-2 M2 n-2 M3 n-2
M1 1
P1 P2 P3
Figure 2: Macrostructure Theory
2.2 Computational Models
Since Marcu (2000) first developed a rule-based discourse parser, several algorithms for discourse parsing have been proposed, both statistical and rule-based, especially since the release of English discourse corpora such as Penn Discourse Treebank (PDTB) and the Rhetorical Structure Theory Discourse Treebank (RST-DT)
For the research in RST-DT, SPADE(Soricut and Marcu, 2003) is a sentence level discourse parser. Two probabilistic models were employed that use syntactic and lexical information to segment and parse text. LeThanh et al. (2004) proposed a multi-step algorithm to segment text and organize the spans into trees for each successive level of text organization. Hernault et al. (2010) presented HILDA, an implemented discourse parser by combining adjacent spans with
greedy bottom-up tree building approach, they built discourse trees in linear time complexity with respect to the length of the input text. Joty et al. (2013) built a discourse tree by applying an optimal parsing algorithm to probabilities inferred from two Conditional Random Fields: one for intra-sentential parsing and the other for multisentential parsing. The performance of the algorithms we mentioned above is shown in Table 2.
On the contrast of the researches at the micro level, very few corpus resources and computational models are proposed at the macro level.
3 Macro Discourse Representation Schema
The overall discourse structure is relevant to the discourse genre and discourse pattern, thus discourse structures vary if the genres are different. For example, the news articles are commonly described in summary-story structure, and academic papers are consist of abstract, introduction, related work, experimentation, conclusion etc. while the court documents are recorded in the structure of in what way, for what reason, where, according to what inference etc.
There are also different ways of discourse expression for the same genre, that is, different discourse patterns. The discourse pattern is the macro structure of text organization, and related to the specific environment and use habits, after repeated use. Various genres formed their own particular and stylized discourse organization structure and characteristics, and became
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
the rules obeyed by the same domain. According to linguists, the common discourse patterns include Problem-Response Pattern, QuestionAnswer Pattern, Claim-Counterclaim Pattern, General-Particular Pattern etc. (Hoey, 2001)
News Report
Summary Story
Headline Lead Situation Comments
Episode Background
ConsequencesMain Events Context History
Circumstances Previous Events
Verbal Reactions Conclusions
EvaluationsExpectations
Figure 3: Schematic Structure on News
Figure 3 is the schematic structure of news genre(Van Dijk, 2013). There are two main parts in the structure: summary and story. Summary contains headline and lead, while story includes situation and comments, and all these parts together constitute the macrostructure of news discourse through a top-down hierarchical structure. Such a schematic structure allows readers to quickly analyze and understand the discourse function in the communicative situation.
Inspired by Van Dijk’s schematic structure, we unify the primary-secondary relationship which defined in RST (nuclearity) and macrostructure separately, and combine the micro and macro discourse structure together, and then explore a Macro Discourse structure representation schema. In this representation schema, discourse structure tree is represented the hierarchical discourse(as shown in Figure 4). We construct a top-down structure with multiple layers, composed of title, chapter, paragraph, microstructure, and elementary discourse unit. In general, there is only one layer of the title and the chapter, while the other structural layers are composed of multiple layers. The arrow direction indicates the primary unit of discourse relation. For the microstructure, we follow the definition of CDT, which not expanded in the Figure 4.
We expand the discourse analysis from intraparagraph to the overall discourse on the basis of original discourse structure analysis. At the micro level, we analyze the logical semantic relationship between the discourse units based on the rhetorical structure; at the macro level, we ana-
lyze the function of the discourse content based on the schematic structure. Guided by this representation, we take the genre of news wire as the main research object, and annotate 97 news wire articles both from the perspective of micro semantics and macro pragmatics.
For example, in the case of magazine article chtb 0596 from Chinese Treebank 8.0, the macro schematic structure and the micro rhetorical structure of our annotation are shown in figure 5 and figure 6. In schematic structure, we annotate the function of each paragraph and unit group, while in rhetorical structure we focus on the rhetorical relation between discourse units.
3.1 Leaf Nodes
Unlike the definition in microstructure (the elementary units are treated as leaf nodes), on the macro layer, we directly treat the paragraphs which are natural segmented in the discourses as leaf nodes.
Take the chtb 0019 for example, which is a typical news wire article from CTB 8.0. There are five paragraphs in the news Significant achievements in the construction of Ningbo Free Trade Zone, so we treat the five paragraphs(1,2,3,4,5) as leaf nodes directly, and the rhetorical structure of this discourse is shown as Figure 7. Limited to the length of this paper, the full discourse text of this example is not included, please refer to the corpus CTB 8.0.
3.2 Non-leaf Nodes
Discourse relations connect discourse units, which are treated as non-leaf nodes in our macro discourse structure. In our representation scheme, we divide the discourse relations into three categories, fifteen subcategories which listed in Table 3. As shown in Figure 7, the relations Elaboration, Background, and Joint are non-leaf nodes in the structure tree.
3.3 Primary-secondary Relationship
A discourse relation generally includes two or more discourse units, these discourse units belong to the same relation layer. If one of the discourse units can generalize the intention and content of the relation layer, and can represent the relation layer connect to other layers, this discourse unit is a primary unit, while others are secondary ones. If the two or more discourse units are equally important, this relation is a multi-primary relation.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Discourse title
Chapter
Paragraph (multi-layers)
Micro-structure (multi-layers)
Elementary Discourse unit
Title
Chapter 2 Relation Chapter 1 Relaiton
Chapter n Relation
Relation Relation
P5
Micro-structure
EDU EDUEDU
Micro-structure
EDU EDU
P1 P2 P3 P4
Relation
Relation
P1 P2
Relation
P3 P4
Relation
P5 P6
Figure 4: Macro Discourse Structure
Detailed
Subtitle 6Summary
BackgroundEpisode
Subtitle 12 Subtitle 15
Magazine article
Summary Situation
12 13-142-41
Question Answer
5
Summary
7
Situation
Episode 1 Episode 2
9-118
Comment
20-21
Episode 1 Episode 2
17-1916
Situation
Figure 5: Macro Schematic Structure of chtb 0596
Joint
Elaboration
Progression
Joint
Background
Joint
Elaboration
Sequence
Elaboration
Joint
Elaboration
ElaborationSequence
Elaboration
Joint
Evaluation
Elaboration
1 2 5 63 4 7 8 11 129 10 13 14 18 1916 1715 20 21
Elaboration
Figure 6: Micro Rhetorical Structure of chtb 0596
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Elaboration
Background
1 2 3 4 5
Joint
Figure 7: Micro Rhetorical structure of chtb 0019
Categories Subcategories
Coordination Joint Sequence Progression Contrast Supplement
Causality
Cause-Result Result-Cause Behavior-Purpose Purpose-Behavior Background
Elaboration
Elaboration Statement-Illustration Illustration-Statement Summary Evaluation
Table 3: Micro Relationship in our corpus
We define three types of primary-secondary relations:(1) P-S, the former unit is primary, and the latter unit is secondary; (2) S-P, the former unit is secondary, and the latter unit is primary; (3) MP, multi-primary. For instance, in statementillustration relation, one of the discourse units is statement, while the other is illustration, the illustration unit is a service for the statement, so the statement-illustration relation is P-S relation; another example, in joint relation, there can be two or more discourse units, one or more discourse units may act as primary parts, so joint relation may be a P-S or S-P or MP relation.
From the micro perspective, the primarysecondary relationship represents the relation between sentences or sentence groups, while from the macro perspective, the primary-secondary relationship represents the relation between paragraphs, chapters or discourses.
In the discourse structure tree shown in Figure 7, the arrows point to the primary units in the discourse. In this discourse, paragraph 2 introduces the approval and development situation of Ningbo free trade zone, which is the background of the event Ningbo free trade zone has achieved fruitful results mentioned in paragraph
1. These two paragraphs constitute a background relation, in which paragraph 1 is a primary unit, and paragraph 2 is a secondary unit according to the content. Paragraphs 3, 4 and 5 respectively from the three aspects expounds the fruitful results mentioned in paragraph 1, the discourse relation among them is joint, and all the three paragraphs constitute a whole unit, explain the overall structure constituted by paragraph 1 and 2, the two paragraph groups form the relation of elaboration, in which the paragraph 1 and 2 is more important. In the overall discourse, paragraph 1 can best express the discourse topic Significant achievements in the construction of Ningbo Free Trade Zone (also the discourse title), paragraph 1 is therefore the most important paragraph among all these discourse units. Based on the arrow pointing in the discourse structure tree can also get the same conclusion.
3.4 Macro Schematic Structure
In the representation schema, the discourse is organized as a tree structure, in which paragraphs appear in the leaf nodes and the discourse relations appear in the non-leaf nodes. The tree structure is an appropriate representation of discourse structure, which not only expresses the hierarchical relationship of the discourse, but also expresses the primary-secondary relationship between the discourse units. Essentially, the depth of the hierarchical structure indicates the depth of the corresponding discourse semantic.
Based on the micro rhetorical discourse structure, we convert the logical semantic relationship to pragmatic function relationship, and establish the schematic structure. We analyze and understand the discourse from a global perspective, and add the pragmatic function to both the leaf nodes and non-leaf nodes, which means all the discourse units on each layer. The functions of every discourse unit form a completed schematic structure tree. Figure 5 and Figure 6 respectively show the macro schematic structure and micro rhetorical structure.
4 Building a Macro Chinese Discourse Treebank
It is necessary to construct a micro and macro unified discourse corpus for discourse structure analysis. Annotation research of macro and micro unified discourse structure, will not only lay the foun-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
dation for the research of discourse structure analysis, but also provide strong support for deeper NLP applications.
4.1 Annotation Overview
Guided by the Macro Discourse Structure, we annotate a Macro Chinese Discourse Treebank(MCDTB) consist of 97 Xinhua news wire articles. Preliminary experiments show that the presentation schema and corpus resource are appropriate to the macro discourse structure analysis.
Because the discourse units are not isolated from the overall discourse, its difficult to judge whether the discourse units are important or not simply from the units themselves. It is necessary to have a comprehensive understanding of the overall article when judging the primarysecondary relations and functional roles.
In determining primary-secondary relations, we should have a full understanding of the discourse intention and main content, and grasp the principle of local and global. Local principle refers to the primary units could summarize the main content and intention of its relation layer, and represent the relation layer relating to the context; global principle refers to the primary discourse units should conform to the main intention of the overall article.
We annotate the discourse topic, lead, abstract, paragraph topics, discourse relations, primarysecondary relations, paragraph segments, and explore the relationship between microstructure and macrostructure.
4.2 Annotation Strategy
We employ a combination of top-down and bottom-up strategy in the annotation work. (1) We determine the overall level first and then analysis goes on step by step to the individual discourse units. Such a top-down strategy can easily grasp the overall discourse structure, which consistent with the reading habit of human beings. (2) Meanwhile, we determine whether the lower discourse units need to be combined first according to the similarity of their forms and contents, and combine them together as a whole unit to contact with other parts. Our annotation work shows the strategy is effective.
4.3 Quality Assurance
To ensure the quality of our corpus, we adopt the annotator consistency using agreement and kappa
on 20 documents(chtb 0011-chtb 0030). Table 4 illustrates the annotation consistency in detail.
As shown in Table 4 we measure the agreement and Kappa of discourse structure, discourse relation, and primary-secondary relation. Its very difficult to achieve high consistence because the judgments of relation and structure are very subjective. Our measurement data is only taken on the layer of leaf nodes.
4.4 Corpus Statistics
Our corpus consists of 97 news wire articles from Chinese Treebank 8.0. There are 533 paragraphs with 438 discourse relations, and each article with an average of 5.49 paragraphs. Detailed statistical data are shown in Table 5.
Count of documents 97 Count of paragraphs 533 Average paragraphs in document 5.49 Maximal of paragraphs 13 Minimal of paragraphs 2 Count of sentences 1339 Average paragraph length (sentences/ paragraph) 2.51 Amount of macro discourse relations 438
Table 5: Corpus Statistics
We also count the primary-secondary data for each subcategory (shown as Table 6). Among the 438 relations, 151 are joint and all of them are multi-nucleus; 121 are elaboration and all of them have a nucleus in the former discourse unit. The statistics shows the relationship between discourse relation and primary-secondary relation is strong dependence, we can make further exploration on this phenomenon later.
5 Preliminary Experimentation
Based on the corpus resource we built, we can do the following analysis: discourse relation discovery, discourse relation recognition, primarysecondary relation recognition, and discourse
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Discourse relation Total P-S S-P MP Joint 151 0 0 151 Sequence 17 1 1 15 Progression 2 0 1 1 Contrast 2 1 0 1 Supplement 30 30 0 0 Cause-Result 13 4 7 2 Result-Cause 10 8 0 2 Behavior-Purpose 4 3 0 1 Purpose-Behavior 1 1 0 0 Background 47 47 0 0 Elaboration 121 121 0 0 Statement-Illustration 9 9 0 0 Illustration-Statement 3 0 3 0 Summary 5 2 3 0 Evaluation 23 15 5 3 Total 438 242 20 176
Table 6: Corpus statistics of discourse relations and primary-secondary relations. P-S: PrimarySecondary; S-P: Secondary-Primary; MP: MultiPrimary
structure tree construction. In this section, we evaluated our Macro Chinese Discourse Treebank with the tasks of recognition of discourse relation and primary-secondary relation.
We solve the imbalance data problem of primary-secondary relation recognition by resampling strategy, that is, re-sampling the data set of nucleus in latter to ten times. A 10 fold cross validation method is adopted, in which sample set is divided into 10 parts of equal size, 9 of them as training set, the remaining one as test set, and different part is selected as test set each time, repeat the exercise 10 times, the average of the 10 results is used to estimate the model performance.
All our classifiers are trained using nltk package with default parameters, and experimented with maximum entropy classifier. Table 7 summarizes the features used in our preliminary experiments.
Text structural features Number of paragraphs in Arg1 (Arg2) Text content features Word pairs (top-10 TF-IDF words in Arg1 and Arg2) Words and POSs (first three words in Arg1 and Arg2) Semantic features Semantic similarity of Arg1 (Arg2) and title Relation structure features Hierarchical characteristics Beginning and end position of Arg1 (Arg2)
Table 7: Features used in our experiments
Table 8 and Table 9 presents the performances of primary-secondary relation recognition and discourse relation recognition. On our corpus,
primary-secondary relation recognition achieves F-score of 79.3%, 88.9%, 50.0% absolutely in the three types of nucleus in former, nucleus in latter and multi-nucleus, and discourse relation recognition achieves F-score of 57.1%, 66.7%, 50.0%, 28.6%, 40.0% on the main types of discourse relation. In general, the errors are produced by two causes: 1) imbalanced distribution of the relations; 2) semantic similarity between the relations. The experimental results show that our corpus is helpful to the discourse structure analysis tasks.
Primary-secondary relation Precision Recall F-score Primary-Secondary 72.4% 87.5% 79.3% Secondary-Primary 80.0% 100.0% 88.9% Multi-Primary 85.7% 35.3% 50.0%
Table 8: Results of primary-secondary relation recognition in MCDTB
Discourse relation Precision Recall F-score Joint 61.5% 53.3% 57.1% Elaboration 59.1% 76.5% 66.7% Sequence 100.0% 33.3% 50.0% Background 25.0% 33.3% 28.6% Cause-Result 50.0% 33.3% 40.0%
Table 9: Results of discourse relation recognition in MCDTB
6 Conclusion
In this paper, we focus on the problem of discourse primary-secondary relationship whose importance is totally ignored in recent studies. We explore to recognize the primary-secondary relationship which plays a critical role in discourse structure analysis, and view it as an independent task from the discourse rhetorical structure analysis.
In particular, we expand the discourse structure analysis from intra-paragraph to the overall discourse, and propose a macro-micro unified discourse structure representation scheme, and describe the scheme in detail. We also annotate 97 news wire articles based on the representation schema we defined. Evaluation of the corpus resource on discourse relation recognition and primary-secondary relation recognition justifies the effectiveness of the representation schema and corpus resource. In the future work, we will enlarge the scale of the corpus and explore the various discourse patterns.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposed a macro discourse structure scheme. The authors carried out a pilot study annotating a corpus consisting of 97 news articles from Chinese treebank 8.0. They then built a model to recognize the primary-secondary relations and 5 discourse relations (joint, elaboration, sequence, background, cause-result) in this corpus.  The paper is poorly written and I have difficulties to follow it. I strongly suggest that the authors should find a native English speaker to carefully proofread the paper. Regarding the content, I have several concerns:   1 The logic of the paper is not clear and justifiable:  1) what are  logical semantics  and  pragmatic function (line 115-116)? I'd prefer the authors to define them properly.  2) macro discourse structure: there are some conflicts of the definition between macro structure and micro structure. Figure 4 demonstrates the combination of macro discourse structure and micro discourse structure. There, the micro discourse structure is presented *within paragraphs*. However, in the specific example of micro discourse structure shown in Figure 6, the micro-level discourse structure is *beyond the paragraph boundary* and captures the discourse relations across paragraphs. This kind of micro-level discourse structure is indeed similar to the macro structure proposed by the authors in Figure 5, and it's also genre independent. So, why can't we just use the structure in Figure 6? What's the advantage of macro discourse structure proposed in Figure 5? For me, it's genre dependent and doesn't provide richer information compared to Figure 6.  By the way, why sentence 6 and sentence 15 are missing in Figure 5? Is it because they are subtitles? But sentence 12 which is a subtitle is present there.  2 Corpus construction (section 4) is not informative enough: without a detailed example, it's hard to know the meaning of  discourse topic, lead, abstract, paragraph topics (line 627-629) . And you were saying you  explore the relationships between micro-structure and macro-structure , but I can't find the correspondent part.  Table 4 is about agreement study The authors claimed  Its very difficult to achieve high consistence because the judgments of relation and structure are very subjective. Our measurement data is only taken on the layer of leaf nodes. -------to First, what are the leaf nodes? In the macro-level, they are paragraphs; in the micro-level, they are EDUs. Should we report the agreement study for macro-level and micro-level separately? Second, it seems for me that the authors only take a subset of data to measure the agreement. This doesn't reflect the overall quality of the whole corpus, i.e., high agreement on the leaf nodes annotation doesn't ensure that we will get high agreement on the non-leaf nodes annotation.  Some other unclear parts in section 4:  Table 4:  discourse structure, discourse relation  are not clear, what is discourse structure and what is discourse relation?  Table 5:  amount of macro discourse relations , still not clear to me, you mean the discourse relations between paragraphs? But in Figure 6, these relations can exist both between sentences and between paragraphs.  3 Experiments: since the main purpose of the paper is to provide richer discourse structure (both on macro and micro level), I would expect to see some initial results in this direction. The current experiment is not very convincing: a) no strong baselines; b) features are not clearly described and motivated; c) I don't understand why only a sub set of discourse relations from Table 6 is chosen to perform the experiment of discourse relation recognition.  In general, I think the paper needs major improvement and currently it is not ready for acceptance.