How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non–linearly encoded in the embedding space, which questions the cosine–based, unsupervised, evaluation methods. All results and analysis scripts are available online.
Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning).
In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non–linearly encoded in the embedding space, which questions the cosine–based, unsupervised, evaluation methods. All results and analysis scripts are available online.
1 Introduction
Using word embeddings remains a standard practice in modern NLP systems, both in shallow and deep architectures [Goldberg, 2015]. By encoding information about words in a relatively simple algebraic structure [Arora et al., 2016] they enable
fast transfer to the task of interest1. The importance of word representation learning has lead to developing multiple algorithms, but lack of principled evaluation hinders moving the field forward, which motivates developing more principled ways of evaluating word representations. Word embeddings are not only hard to evaluate, but also challenging to train. Recent practice shows that one often needs to tune algorithm, corpus and hyperparameters towards the target task [Lai et al., 2016, Sharp et al., 2016b], which challenges the promise of broad applicability of unsupervised pretraining.
Evaluation methods of word embeddings can be roughly divided into two groups: extrinsic and intrinsic [Schnabel and Labutov, 2015]. In the former approach embeddings are used in a downstream task (eg. POS tagging), while in the latter embeddings are tested directly for preserving syntactic of semantic relations. The most popular intrinsic task is Word Similarity (WS) which evaluates how well dot product between two vectors reproduce score assigned by human annotators. Intrinsic evaluations always assume a very specific model for recovering given property.
Despite popularity of word embeddings, there is no clear consensus what evaluation methods should be used, and both intrinsic and downstream evaluations are criticized [Tsvetkov et al., 2015a, Faruqui et al., 2016]. On top of that, different evaluation schemes usually lead to different rankings of embeddings [Schnabel and Labutov, 2015]. For instance, it has been shown [Baroni and Dinu, 2014] that neural-based word embeddings perform consistently better then count-based models and later, using WS and WA tasks, it was argued otherwise [Levy et al., 2015]. Recent research in evaluation methods focuses on representative or inter-
1Algebraic structure refers to the fact that words can be decomposed into a overcomplete basis, such that each word can be expressed as a sparse sum of base vectors
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
pretable set of tasks [Nayak et al., 2016, Köhn, 2015a], analysing intrinsic evaluation [Chiu et al., 2016, Faruqui et al., 2016], as well as proposing improvements to intrinsic evaluation [Avraham and Goldberg, 2016, Tsvetkov et al., 2015b].
In this paper we employ a transfer learning view, in which the main goal of representation learning is to make subsequent learning fast, i.e. use resulting word embeddings to maximize performance at the lowest sample complexity possible [Bengio et al., 2013, Glorot et al., 2011]2. Surprisingly, researchers rarely report model (using given word representation) performance under varying (benchmark) dataset sizes and model classes3, which is crucial for correct evaluation of transfer, especially given increasing importance of small data regime applications. Motivated by this, we propose an evaluation focused on data efficiency. To quantify precisely accessible information, we additionally propose focusing only on simple (supervised) tasks, as complex downstream tasks are challenging to interpret. In addition, we propose principled improvements to WS and WA tasks, which try to address some of the critiques both benchmarks have received in the literature [Faruqui et al., 2016], in authors’ opinion mostly due to their purely unsupervised nature.
2 Proposal
Our main goal is to better align evaluation of word embeddings with their transfer application. Future performance is correlated with the amount of easily accessible and useful information. By easily accessible information, we mean information that model can quickly learn to use. Useful information is defined as one that correlates well with the final task.
First argument for data efficiency focused evaluation is the growing evidence that pretrained word embeddings provide little benefit under various settings, especially deep learning models [Zhang et al., 2015, Zhang and Wallace, 2015, Andreas and Klein]. We hypothesize that most of the improvements (in downstream tasks) reported in literature are caused by small size of the supervised dataset, which is reasonable from the trans-
2Alternative goals might include maximizing interpretability, or analysing unsupervised corpora.
3What is claimed here is that vast majority of papers doesn’t take into consideration those factors. Nevertheless, there are notable exceptions [Andreas and Klein, Qu et al., 2015, Amir et al., 2017].
fer learning point of view. Therefore, measuring performance after seeing just a subset of the supervised dataset is crucial for comparing word embeddings. Another argument is the empiricial difference between how easily accessible is the information in various embeddings. As our experiment show, commonly used dense representations achieve different learning speeds. This effect should be even stronger for sparse representations, for which feature dimensions can have very strict semantic meaning [Faruqui and Dyer, 2015a]. An argument can be also made from theoretical point of view; it is easy to show that any injective (and thus not losing information) embedding preserves all information about corpora (see Appendix for details), i.e. having enough training data makes embeddings dispensable.
Second part of the proposal is to focus on simple supervised tasks to directly evaluate useful information content. In certain applications, like tagging, choosing the right, specialized, word embeddings is crucial for obtaining state of the art results [Sharp et al., 2016a, Lample et al., 2016]. We also confirm empirically that word embeddings trade off capacity between different information. In this work we pose hypothesis, that specialization of word embeddings can be best evaluated by checking what simple information is most easily recoverable. While word level classification problems (like noun classification) were proposed previously [Köhn, 2015b], here we also suggest including tests for recovery of relations between words (exemplified in experiments by Similarity and Analogy tasks) .
Importance of simple supervised tasks can be also seen in the light of algebraic structure that is encoded in word representation space. It has been observed in practice that word embeddings have useful information only in a small subspace [Sattigeri and Thiagarajan, 2016, Rothe and Schütze, 2016, Astudillo et al., 2015]. Thus, simple supervised tasks are closely aligned with the actual use of word embeddings and allow to quantify how quickly model can extract the most salient subspace (which leads to faster learning in general).
Our final remark is about diversity of models. Commonly used WS and WA datasets are solved by a constant models, i.e. model which does not learn from the data. We argue that such evaluation is not generally informative. If we are interested in how well our vector space helps solving a given
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 1: Vertical axis is normalized difference between score of the best performing non-linear model and the best performing linear model (the higher, the better non-linear model is). Each box plot represents a single tested embedding.
problem, we should in theory fit all possible models and pick the one that has the best generalization capabilities. While this is impractical, it illustrates that fixing one specific model gives answer to a different question, thus drawing general conclusions from it can be highly biased. A good rule of thumb might be to include representatives of typical model classes, or at least match the model with class of models we are interested in (which rarely will be constant), which concludes our guidelines for a correct evaluation.
We leave out details from the proposal how to order embeddings, as this is determined by the specific research question given evaluation should answer. A sensible default is to report AUC of learning curve for each task, and pick set of tasks that are most interesting to the researcher.
To summarize:
• Evaluation should focus on data efficiency (if transfer is the main goal of representation learning).
• Tasks should be supervised and simple.
• Unless focus is on specific application, evaluation should focus on a diverse set of models (including nonlinear and linear ones) and datasets (testing for various information content).
If we follow those guidelines, we truly approximate (for a given trained embedding) generalization error under distribution of tasks, dataset size and classifiers,
E [ Lt(cl(VU(Xtm),Y t m)(VU(x),y) ] , (1)
VU denotes the embeddings trainedw on U, Lt denotes task t loss and expectation is taken with respect to:
• p(cl) – distribution of classifiers,
• p(Xtm,Ytm) – distribution of training datasets, where we first sample task t and then uniformly sample dataset Xtm of size m.
• x,y – i.i.d. examples following training data distribution.
Distribution over classifiers and tasks should be carefully tuned to researcher’s needs, as we will argue soon. Further theoretical analysis is included in Appendix, and in the rest of the paper we present practical arguments for the proposed evaluation scheme.
3 Experiments
In this section we define specific metrics and tasks and perform exemplary evaluation of several pretrained embeddings in the advocated setting. Specifically, we try to empirically answer several questions, all geared towards providing experimental validation for the three main points of proposal:
• Do supervised versions of WA and WS benchmarks provide additional insights?
• How stable is the ranking of embeddings under changing dataset size?
• Are there embeddings that benefit from nonlinear models?
The first question will aid understanding how useful are simple supervised tasks coupled with data efficiency. Second question shows that ranking of embeddings do change under transfer learning evaluation. Last question explores if any embeddings encode information in a “non-linear” fashion; while one of the main goals of representation learning is disentangling factors of variation, usually learned representations are entangled and dense, which poses interesting question how hard it is to extract useful patterns from them. In this paper we report only a subset of results with the most interesting conclusions, all results (along with code) are also made available online for further analysis.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 2: Accuracy reached by 3 different models optimizing word similarity on SimLex dataset. Rightmost plot deptics performance of the traditionally used constant model. While NMT has been reported to have strong performance on SimLex (as shown on the rightmost plot), its relative gains diminish under supervised version of the benchmark (leftmost and central plot). Tested embeddings (ordered by performance on the rightmost plot): NMT, GloVe300, HPCAautoenc.
3.1 Datasets and models
Datasets are divided into 4 categories: Similarity, Analogy, Sentence and Single word. Analogy datasets are composed of quadruples (two pairs of words in a specific relation, for instance (king, queen, man, woman)). Similarity datasets are composed of pairs of words and assigned mean rank by human annotators. Sentence and Single word datasets have binary targets. In total our experimentation include 15 datasets:
• Similarity: SimLex999 [Hill et al., 2015], MEN [Bruni et al., 2014], WordSimilarity353 [Finkelstein et al., 2001] and Rare Words [Luong et al., 2013].
• Analogy: 4 categories from WordRep [Gao et al., 2014]4.
• Sentence: Stanford Sentiment Treebank [Socher et al., 2013] and News20 (3 binary datasets) [Tsvetkov et al., 2015a].
• Single word: Datasets constructed from lexicons collected in [Faruqui and Dyer, 2015b]: POS tagging (3 datasets for verb, noun and adjective), word sentiment (1 dataset), word color association (1 dataset) and WordNet synset membership (2 datasets).
Models for each datasets include both nonlinear and linear variants. When model is non-
4We experimented with MSR and Google datasets and observed that models easily overfit if the train and test sets share the same words (not 3-tuples). WordRep dataset is a set of pairs which we split into disjoint sets.
linear, for robustness we include in search a fallback to a simpler linear or constant model. Additionally, in the case of Similarity and Analogy we include commonly used constant models. Similarity between 2 vectors is approximated by their cosine similarity (cos(~v1, ~v2)). In the case of Analogy tasks embedding is evaluated for its ability to infer 4th word out from the first three and we use the following well-known constant models: 3COSADD (argmax~v∈V cos(~v, ~v2− ~v1+ ~v3)) and 3COSMUL (argmax~v∈V
ccos(~v, ~v3)ccos(~v, ~v2) ccos(~v, ~v1)+ )5. For each task class we evaluate a different set of classifiers:
• Similarity: cosine similarity, Random Forest (RF), Support Vector Regression (SVR) with RBF kernel6.
• Analogy: 3COSADD, 3COSMUL [Levy et al., 2015] and regression neural network (performing regression on the 4th word given the rest of the quadruple, see Appendix for further information).
• Sentence: Logistic Regression, Support Vector Machine (SVM) with RBF kernel taking as input averaged embedding vector and Convolutional Neural Network (CNN) [Kim, 2014] taking as input concatenation of embedding vectors.
• Single word: RF, SVM (with RBF kernel), Naive Bayes, k-Nearest Neighbor Classifier and Logistic Regression.
3.2 Embeddings
Our objective was to cover representatives of embeddings emerging from both shallow and deeper architectures. Deep embeddings are harder to train, so for the scope of this paper we decided to include pretrained and publicly available vectors7. Setup includes following “shallow” pretrained embeddings: GloVe (100 and 300 dimensions) [Pennington et al., 2014], Hellinger PCA (HPCA) [Lebret and Collobert, 2014], PDC (100 and 300 dimensions) and HDC (300 dimensions) [Sun et al., 2015], Additionally following “deep” embeddings are evaluated: Neural Translation Machine (NMT,
5ccos(~v1, ~v2) = 1+cos( ~v1, ~v2) 2 6We also tried RankSVM [Lee and Lin, 2014], but it did not perform better than other models, while being very computationally intensive.
7Vocabularies were lowercased and intersected before perfoming experiments. Vectors were normalized to a unit length.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
rank start rank end auc rank
GloVe100 2.09± 1.76 2.48± 1.55 2.17± 1.77 GloVe300 1.83± 2.63 1.91± 2.54 1.87± 2.57 HDC300 0.93± 1.34 1.06± 1.22 0.91± 1.40 HPCAautoenc 3.52± 2.04 3.68± 1.67 3.65± 2.03 HPCA 4.77± 2.33 4.75± 2.23 4.83± 2.29 NMT 3.90± 2.24 4.32± 1.93 4.28± 2.34 PDC300 0.64± 0.89 0.74± 1.07 0.65± 1.10 morphoRNNLM 4.13± 2.34 4.55± 2.02 4.26± 2.29
Table 1: First two column present ranks at the 30% and 100% splits of evaluation dataset averaged over all categories. Third column is rank computed by recommended default AUC of curve. As expected, when averaging over many tasks (of different dataset sizes), data efficiency is not changing final ordering (third column). On average rank increases as embeddings are becoming significantly different (as determined by ANOVA during rank computation).
activations of the deep model are extracted as word embeddings) [Hill et al., 2014], morphological embeddings (morph, which can learn morphological differences between words directly) [Luong et al., 2013] and HPCA variant trained using autoencoder architecture [Lebret and Collobert, 2015]. In some experiments we additionally include publicly available pretrained skip-gram embeddings on Google News corpora and skipn-gram embeddings trained on Wikipedia corpora [Ling et al., 2015] (used commonly in syntax demanding tasks, like tagging).
3.3 Results
For each dataset we first randomly select test set and run evaluation for increasing sizes of training dataset, thus scores approximate generalization error after seeing increasing amounts of data. Splits are repeated 6 times in total to reduce noise. Thus, for each task results are 6 learning curves, with a score for each subset of data (see Fig. 2).
Ranks of embeddings at each point are calculated using a greedy sequential procedure, where we assign embeddings the same rank if their scores (each point on the curve is represented by 6 scores) are not significantly different, as tested using pairwise ANOVA test. All results are available online8.
3.4 Learnable Similarity and Analogy tasks
Our first question was validating that adding learnable Similarity and Analogy tasks introduce any
8Results will be posted online upon publication.
additional insights. Positive answer to this question motivates introduction of simple tasks with varying dataset size, ideally defined on single or pair of words.
For solving Analogy we implemented a shallow neural network. Interestingly, WordRep authors [Gao et al., 2014] reported low accuracies (often even below 5%) on most analogy questions and we were able to improve absolute score upon the tested subset on average by absolute 11% (see Tab. 2). Having learnable models for Similarity and Analogy datasets enables reusing many publicly available datasets in the new context. Also, we can robustly evaluate if given information about relation between two words is present in the embedding based on analogy questions. In the case of HASCONTEXT and INSTANCEOF datasets no embeddings can recover analogy answers using static models (achieved accuracy is below 3%), but actually some embeddings do have information about the relations. In both cases HDC consistently outperforms other embeddings reaching around 25% accuracy, see Fig. 4 and Tab. 2.
In the case of Similarity dataset the best performing model was Support Vector Regression, similarly as in Analogy datasets we also improve over the constant models. What is more, we can draw novel conclusions. Interesting example is NMT performance on SimLex. It was claimed in [Hill et al., 2014] that NMT embeddings are better at encoding true similarity between words, but SVR on Glove embeddings performs better after training on the whole dataset (i.e. at the end of learning curve), see Fig.2.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
rank start rank end auc rank
GloVe100 1.80± 2.04 2.31± 1.83 1.86± 2.07 GloVe300 3.06± 3.17 3.14± 2.99 3.03± 3.09 HDC300 1.37± 1.68 1.37± 1.40 1.31± 1.69 HPCAautoenc 3.63± 2.30 3.80± 1.98 3.83± 2.28 HPCA 3.77± 2.77 3.60± 2.58 3.80± 2.79 NMT 3.74± 2.05 4.29± 1.74 4.23± 2.30 PDC300 0.60± 0.77 0.60± 1.01 0.51± 0.95 morphoRNNLM 3.00± 2.38 3.60± 2.16 3.17± 2.38
Table 3: First two column present ranks at the 30% and 100% splits of evaluation dataset averaged over all tasks for Single word datasets. Third column is rank computed by recommended default AUC of curve. Interestingly, information stored in Single word datasets is easily accessible even with small amount of data.
3.5 Rank stability under changing dataset size
Second question was how stable are the orderings under growing dataset. To this end we have measured rank at the beginning (30% of data) and end of training. Mean absolute value of change of ranking is approximately (averaged over all categories) 0.6 with standard deviation of 0.2. This means that usually an embeddings has a changed rank after training, which establishes usefulness of measuring data efficiency for the tested embeddings.
Interestingly, when averaged over many experiments, final ordering of embeddings tends to be similar, see Tab. 1 and Fig. 3. This is mainly because (tested) embeddings have different data efficiency properties for different tasks, i.e. none of embeddings is consistently more data efficient than others. On top of that standard deviation of both rank at the end and beginning is around 2.5, which further reinforces findings from [Schnabel and Labutov, 2015] that embeddings orderings are very task dependent.
Measuring data efficiency is crucial for a realistic (i.e. as close to application as possible) evaluation of representations. Besides a more accurate and practical ordering of embeddings, it also allows one to draw new conclusions, which is exemplified by differences between GloVe100 and GloVe300 (elaborated on in the next section). Another interesting point is that rank change after training on full dataset is relatively low for Single word datasets, which suggests that simple information about words like noun or verb is always quickly accessible to models, but more complicated information like relationships or similarities
between pair of words are not, see Tab. 3.
3.6 Linear vs non–linear models
Our last question was how stable is the ordering under changing model type. More specifically, are there embeddings especially fitted for use with linear models? Clearly some embeddings in fact are, see Fig. 1. This is an important empirical fact for practictioners, which motivates including such evaluation in experiments. In particular, it clearly shows that typically used evaluation does not answer the question “is there information about task X in the embedding Y” but only “is information about task X stored in embedding Y easily separable by a static (or linear) classifier”.
An illustrative example is the difference in performance between two pretrained GloVe embeddings of different dimensionality (100 and 300). It has been shown previously that lower dimensional GloVe embeddings are better at syntactic tasks [Lai et al.], but our evaluation reveals more complicated picture, that GloVe objective might encourage some sort of nonlinear encoding. We can see that by significantly better rank of GloVe100 at the beginning of learning of Single word datasets (mean rank 1.8), but lower at the end (mean rank 2.3), see Tab. 3. This is also visible when averaged over all categories, see Tab. 1.
3.7 Discussion
Performed eperiments show usefulness of the additional and more granular level of analysis enabled. Researcher can ask more precise questions, like “is it worth fitting syntax specific embeddings even when supervised dataset size is large?” (to which answer is positive based on our experiments) or “is HASINSTANCE relation encoded in the space?” (to which answer is also positive for some embeddings). Unfortunately, there is already a large volatility of final embeddings ordering when using standard evaluation, and our proposed scheme at times makes it even more challenging to decide which embeddings are optimal. This hints, that purely unsupervised large scale pretraining might not be suitable for NLP applications. Most importantly, evaluation should be more targeted, either at some specific application area, or at specific properties of representation.
One of the presented arguments for including supervised models for testing information content is algebraic interpretation of word embeddings [Arora et al., 2016]. The algebraic structure
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 3: Rank at the beginning and end of learning for a subset of tested embedding (horizontal axis) and category (color). Task categories from left to right are: Sentence (blue), Single word (green), Similarity (red) and Analogy (purple).
Figure 4: Accuracy reached by Neural Network regression model (left plot) and 3COSMUL (right plot) optimizing analogy (HASCONTEXT relation from WordRep benchmark [Gao et al., 2014]).
present in the representation space enables one to decompose word embedding space into a set of concepts (so each word vector can be well approximated by a sum of few concepts)9. Theoretically, tasks defined on single words should test for existence of such concepts, but in our case including (supervised) Analogy tasks was very useful, as those tasks are still very challenging for current embeddings. For Analogy tasks (see Fig. 4) achieved accuracy scores are below 25%, whereas in the case of Single word average accuracy is around 80% (and fitting classifier adds on average only 2%). These higher order (or subspace) focused tasks are also well aligned with the application of word embeddings, because empirically models tend to focus on a small subspace in the vector space.
9This decomposition can be obtained using standard methods like k-SVD.
4 Conclusions
As exemplified by experiments, proposed evaluation reveals differences between embeddings along usually overlooked dimensions: data efficiency, non-linearity of downstream model and simple supervised tasks (including recovery of higher order relations between words). Interesting new conclusions can be reached, including differences between different size GloVe embeddings or performance of non-linear models on similarity benchmarks.
Additionally, obtained results reinforce conclusions from other published studies that there are no universally good embeddings and finding such might not be achievable, or a well posed problem. One should take great care when designing evaluation and specify what is the main focus. For instance, if the main goal of the word embeddings is to be useful in transfer, one should include advocated data efficiency metrics. New word embedding algorithms are moving away from typical pretraining scheme, with increasing focus on specialized word embeddings and applications under very limited dataset size, where fast learning is crucial. We hope that proposed evaluation methodology will help advance research in these scenarios.
A Theoretical analysis
Let us first try to answer the question what is the information about the task and how can it be measured given some data representation. For simplicity let us assume that task is a binary classification, but the same reasoning applies to multiclass, multilabel, regressions etc. A quite natural, machine learning perspective is to define information stored as a Bayes risk of optimal model trained to perform this task. Obviously, raw representation already has some non-negative Bayes risk, which cannot be reduced during any embedding. Actually, it is quite easy to show, that nearly every embedding preserves all the information contained in the source representation.
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
Observation 1. Every injective embedding preserves all the information about the task.
Proof. Let us assume that Bayesian optimal classifier (the one obtaining the Bayes risk RX ) on the input space X be called o. Furthermore, let our embedding (learned in arbitrary manner, supervised or not) be a function E : X → X ′. According to assumptions, E is injective, thus for every x ∈ X there exists unique x′ ∈ X ′ such that E(x) = x′. Let us call the corresponding inverse assignment E−1 (defined only on the image of E). Consequently we can define classifier on E(X) ⊂ X ′ through o′(x′) = o(E−1(x′)). It is now easy to show, that Bayes risks of these two models are exactly the same
RX = ∫ ∑
y
`(o(x), y)p(y|x)p(x)dx
= ∫ ∑ y `(o(E−1(E(x))), y)p(y|x)p(x)dx
= ∫ ∑ y `(o′(E(x), y)p(y|x)p(x)dx
= RX ′ .
This remains an open question how frequent in general are injective embeddings. If one considers continuous spaces as X then this is extremely small class of functions (especially if dim(X ′) ≤ dim(X )). However in case of natural language processing (and many other fields), the input space is actually discrete or even finite. In such case, non-injective embeddings are rare phenomenon. In particular, for any finite set, probability of selecting at random linear projection which gives non-injective embedding is zero.
The above reasoning is in some sense trivial, yet still worth underlying, as it gives an important notion of what should be measured when evaluating embeddings. Even though Bayes risk is the same for both spaces, the complexity of inferring o′ can be completely different from complexity of inferring o. We argue, that this is a crucial element - to measure how hard is to learn o′ (or any reasonable approximation). There are two basic dimensions of such analysis:
• check how complex the set of hypotheses H 3 o′ needs to be in order to be able to find it using given data,
• verify how well one can approximate o′ as a function of growing training size. In other words - how fast an estimator of o′ converges.
Thus, to really distinguish various embeddings, we should rather ask what is the best achievable performance under limited amount of data or under constrained class of models, which is theoretical argument for data efficiency oriented evaluation.
B Regression neural network for word analogy task
Let D be the dimension of a given word embedding. We assume that all embedded words have euclidean norm equal to one – this guarantees that the scalar product of two embedded words is also their cosine similarity. The word analogy task is defined in the following way: given (embedded) words v1, v2 and v3, predict word v4 that satisfies analogy “v1 is related to v2 as v3 is related to v4”. Our estimator of v4 is defined as:
v̂4 = −W1v1 +W2v2 +W3v3 + b ‖−W1v1 +W2v2 +W3v3 + b‖2
where the model trainable parameters are:
• W1, W2 and W3 – D×D matrices initialized with identities,
• b – D-dimensional vector initialized with zeros,
and the cost is defined as: − ∑ j 〈vj4, v̂ j 4〉.
The model was trained with gradient descent optimization on minibatches. Hyperparameters: learning rate, number of epochs, optimizer, batch size and (boolean) fallback to constant model were chosen using cross-validation. The actual prediction has two steps:
• calculate v̂4,
• choose (embedded) word v that minimizes 〈v, v̂4〉.
Observe that this model is initialized in such a way, that it is equivalent to 3COSADD – the idea is to check, if applying trainable affine transformations to input vectors would boost 3COSADD performance. It should also be noted that this approach turned out to be very computationally intensive.

This paper proposes a framework for evaluation of word embeddings based on data efficiency and simple supervised tasks. The main motivation is that word embeddings are generally used in a transfer learning setting, where evaluation is done based on how faster is to train a target model. The approach uses a set of simple tasks evaluated in a supervised fashion, including common benchmarks such as word similarity and word analogy. Experiments on a broad set of embeddings show that ranks tend to be task-specific and change according to the amount of training data used.  Strengths  - The transfer learning   data efficiency motivation is an interesting one, as it directly relates to the idea of using embeddings as a simple  semi-supervised  approach.  Weaknesses  - A good evaluation approach would be one that propagates to end tasks. Specifically, if the approach gives some rank R for a set of embeddings, I would like it to follow the same rank for an end task like text classification, parsing or machine translation. However, the approach is not assessed in this way so it is difficult to trust the technique is actually more useful than what is traditionally done. - The discussion about injective embeddings seems completely out-of-topic and does not seem to add to the paper's understanding. - The experimental section is very confusing. Section 3.7 points out that the analysis results in answers to questions as  is it worth fitting syntax specific embeddings even when supervised datset is large?  but I fail to understand where in the evaluation the conclusion was made. - Still in Section 3.7, the manuscript says  This hints, that purely unsupervised large scale pretraining might not be suitable for NLP applications . This is a very bold assumption and I again fail to understand how this can be concluded from the proposed evaluation approach. - All embeddings were obtained as off-the-shelf pretrained ones so there is no control over which corpora they were trained on. This limits the validity of the evaluation shown in the paper. - The manuscript needs proofreading, especially in terms of citing figures in the right places (why Figure 1, which is on page 3, is only cited in page 6?).  General Discussion  I think the paper starts with a very interesting motivation but it does not properly evaluate if their approach is good or not. As mentioned above, for any intrinsic evaluation approach I expect to see some study if the conclusions propagate to end tasks and this is not done in the paper. The lack of clarity and proofreading in the manuscript also hinders the understanding. In the future, I think the paper would vastly benefit from some extrinsic studies and a more controlled experimental setting (using the same corpora to train all embeddings, for instance). But in the current state I do not think it is a good addition to the conference.


Joint Modeling of Content and Discourse Relations in Dialogues
We present a joint modeling approach to identify salient discussion points in spoken meetings as well as to label the discourse relations between speaker turns. A variation of our model is also discussed when discourse relations are treated as latent variables. Experimental results on two popular meeting corpora show that our joint model can outperform SVM-based classifiers for both phrase-based content selection and discourse relation prediction tasks. We also evaluate our model on predicting the consistency among team members’ understanding of their group decisions. Classifiers trained with features constructed from our model achieve significant better predictive performance than the state-of-the-art.
1 Introduction
Goal-oriented dialogues, such as meetings, negotiations, or customer service transcripts, play an important role in our daily life. Automatically extracting the critical points and important outcomes from dialogues would facilitate generating summaries for complicated conversations, understanding the decision-making process of meetings, or analyzing the effectiveness of collaborations.
We are interested in a specific type of dialogues — spoken meetings, which is a common way for collaboration and idea sharing. Previous work (Kirschner et al., 2012) has shown that discourse structure can be used capture the main discussion points and arguments put forward during problem-solving and decision-making processes in meetings. Indeed, content of different speaker turns do not occur in isolation, and should be interpreted within the context of discourse. Meanwhile, content can also reflect the purpose of speaker turns, thus facilitate with discourse relation understanding. Take the meeting snippet from AMI corpus (Carletta et al., 2006) in Figure 1 as
D: Three different types of batteries. Um can either use a hand dynamo, or the kinetic type ones, you know that they use in watches, or else uh a solar powered one.
B: Um the bat uh the battery for a a watch wouldn't require a lot of power, would be my one query. Is a kinetic one going to be able to supply enough power?
D: Yeah, I don't think it would. C: Yeah.
D: We should probably just use conventional batteries. B: Which I suppose as well would allow us to go off the shelf again, you'd say ?
D: Yeah.
Uncertain
Option
an example. This discussion is annotated with discourse structure based on the Twente Argumentation Schema (TAS) by Rienks et al. (2005), which focuses on argumentative discourse information. As can be seen, meeting participants evaluate different options by showing doubt (UNCERTAIN), bringing up alternative solution (OPTION), or giving feedback. The discourse information helps with the identification of the key discussion point, i.e., “which type of battery to use”, by revealing the discussion flow.
To date, most efforts to leverage discourse information to detect salient content from dialogues have focused on encoding gold-standard discourse relations as features for use in classifier training (Murray et al., 2006; Galley, 2006; McKeown et al., 2007; Bui et al., 2009). However, automatic discourse parsing in dialogues is still a challenging problem (Perret et al., 2016). Moreover, acquiring human annotation on discourse relations is a timeconsuming and expensive process, and does not scale for large datasets.
In this paper, we propose a joint modeling approach to select salient phrases reflecting key dis-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
cussion points as well as label the discourse relations between speaker turns in spoken meetings. We hypothesize that leveraging the interaction between content and discourse has the potential to yield better prediction performance on both phrase-based content selection and discourse relation prediction. Specifically, we utilize argumentative discourse relations as defined in Twente Argument Schema (TAS) (Rienks et al., 2005), where discussions are organized into tree structures with discourse relations labeled between nodes (as shown in Figure 1). Algorithms for joint learning and joint inference are proposed for our model. We also present a variation of our model to treat discourse relations as latent variables when true labels are not available for learning. We envision that the extracted salient phrases by our model can be used as input to abstractive meeting summarization systems (Wang and Cardie, 2013; Mehdad et al., 2014). Combined with the predicted discourse structure, a visualization tool can be exploited to display conversation flow to support intelligent meeting assistant systems.
To the best of our knowledge, our work is the first to jointly model content and discourse relations in meetings. We test our model with two meeting corpora — the AMI corpus (Carletta et al., 2006) and the ICSI corpus (Janin et al., 2003). Experimental results show that our model yields accuracy of 63.2 and 59.2 on phrase selection and discourse prediction on AMI, which is significantly better than comparisons based on Support Vector Machines (SVM) classifiers (accuracy of 57.8 and 51.2). Our model trained with latent discourse also outperforms SVMs on both AMI and ICSI corpora. We further evaluate the usage of selected phrases as extractive meeting summaries. Results evaluated by ROUGE (Lin and Hovy, 2003) demonstrate that our system summaries obtain a ROUGE-SU4 F1 score of 21.3 on AMI corpus, which outperforms two utterancelevel extractive summarization baselines that select the longest and the most representative utterance from each discussion.
Moreover, since both content and discourse structure are critical for building shared understanding among participants (Mulder et al., 2002; Mercer, 2004), we further investigate whether our learned model can be utilized to predict the consistency among team members’ understanding of their group decisions. This task is first defined as
consistency of understanding (COU) by Kim and Shah (2016), who have labeled a portion of AMI discussions with consistency or inconsistency labels. We construct features from our model predictions to capture different discourse patterns and word entrainment scores for discussion with different COU level. Results on AMI discussions show that SVM classifiers trained with our features significantly outperform the state-of-the-art results (Kim and Shah, 2016) (F1: 63.1 vs. 50.5) and non-trivial baselines.
2 Related Work
Our model is inspired by research work that leverages discourse structure for identifying salient content in conversations, which is still largely reliant on features derived from gold-standard discourse labels (McKeown et al., 2007; Murray et al., 2010; Bokaei et al., 2016). There is much less work that jointly predicts the importance of content along with the discourse structure in dialogus. Oya and Carenini (2014) employs Dynamic Conditional Random Field to recognize sentences in email threads for use in summary as well as their dialogue acts. Only local discourse structures from adjacent utterances are considered. Our model is built on tree structures, which captures more global information.
Our work is also in line with keyphrase identification or phrase-based summarization for conversations. Due to the noisy nature of dialogues, recent work focuses on identifying summary-worthy phrases from meetings (Fernández et al., 2008; Riedhammer et al., 2010) or email threads (Loza et al., 2014). Our work also targets at detecting salient phrases from meetings, but focuses on the joint modeling of critical discussion points and discourse relations held between them.
For the area of discourse analysis in dialogues, a significant amount of work has been done in predicting local discourse structures, such as recognizing dialogue acts or social acts of adjacent utterances from phone conversations (Stolcke et al., 2000; Kalchbrenner and Blunsom, 2013; Ji et al., 2016), spoken meetings (Dielmann and Renals, 2008), or emails (Cohen et al., 2004). Although discourse information from non-adjacent turns has been studied in the context of online discussion forums (Ghosh et al., 2014) and meetings (HakkaniTur, 2009), none of them models the effect of discourse structure on content selection, which is a
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
gap that this work fills in.
3 The Joint Model of Content and Discourse Relations
3.1 Model Description
Our proposed model learns to jointly perform phrase-based content selection and discourse relation prediction by making use of the interaction between the two sources of information. Assume that a meeting discussion is denoted as x, where x consists of a sequence of discourse units x = {x1, x2, · · · , xn}. Each discourse unit can be a complete speaker turn or a part of it. As demonstrated in Figure 1, a tree-structured discourse diagram is constructed for each discussion with each discourse unit xi as a node of the tree. In this work, we consider the argumentative discourse structure by Twente Argument Schema (TAS) (Rienks et al., 2005). For each note xi, it is attached to another node xi′ (i′ < i) in the discussion, and a discourse relation di is hold on the link 〈xi, xi′〉 (di is empty if xi is the root). Let t denote the set of links 〈xi, xi′〉 in x. Following previous work on discourse analysis in meetings (Rienks et al., 2005; Hakkani-Tur, 2009), we assume that the attachment structure between discourse units are given during both training and testing.
A set of candidate phrases are extracted from each discourse unit xi, from which salient phrases that contain gist information will be identified. We obtain constituent and dependency parses for utterances using Stanford parser (Klein and Manning, 2003). We restrict eligible candidate to be a noun phrase (NP), verb phrase (VP), prepositional phrase (PP), or adjective phrase (ADJP) with at most 5 words, and its head word cannot be a stop word. If a candidate is a parent of another candidate in the constituent parse tree, we will only keep the parent. We further merge a verb and a candidate noun phrase into one candidate if the later is the direct object or subject of the verb. For example, from utterance “let’s use a rubber case as well as rubber buttons”, we can identify candidates “use a rubber case” and “rubber buttons”. For xi, the set of candidate phrases are denoted as ci = {ci,1, ci,2, · · · , ci,mi}, where mi is the number of candidates. ci,j takes a value of 1 if the corresponding candidate is selected as salient phrase; otherwise, ci,j is equal to 0. All candidate phrases in discussion x are represented as c.
We then define a log-linear model with feature
parameters w for the candidate phrases c and discourse relations d in x as: p(c,d|x,w) ∝ exp[w · Φ(c,d,x)]
∝ exp[w · n∑
i=1,<xi,xi′>∈t
φ(ci, di, di′ ,x)]
∝ exp[ n∑
i=1,<xi,xi′>∈t
(wc · mi∑ j=1 φc(ci,j ,x)
+ wd · φd(di, di′ ,x) + wcd · mi∑ j=1 φcd(ci,j , di,x))]
(1)
Here Φ(·) and φ(·) denote feature vectors. We utilize three types of feature functions: (1) content-only features φc(·), which capture the importance of phrases, (2) discourse-only features φd(·), which characterize the (potentially higherorder) discourse relations, and (3) joint features of content and discourse φcd(·), which model the interaction between the two. wc, wd, and wcd are corresponding feature parameters. Detailed feature descriptions can be found in Section 3.4. Discourse Relations as Latent Variables. As we mentioned in the introduction, acquiring labeled training data for discourse relations is a timeconsuming process since it would require human annotators to inspect the full discussions. Therefore, we further propose a variation of our model where it treats the discourse relations as latent variables, so that p(c|x,w) = ∑ d p(c,d|x,w). Its learning algorithm is slightly different as described in the next section.
3.2 Joint Learning for Parameter Estimation
For learning the model parameters w, we employ an algorithm based on SampleRank (Rohanimanesh et al., 2011), which is a stochastic structure learning method. In general, the learning algorithm constructs a sequence of configurations for sample labels as a Markov chain Monte Carlo (MCMC) chain based on a task-specific loss function, where stochastic gradients are distributed across the chain. This is suitable for our learning problem because we aim to optimize the prediction performance for both phrase selection and discourse relations with various types of features.
The full learning procedure is described in Algorithm 1. To start with, the feature weights w is initialized with each value randomly drawn from [−1, 1]. Multiple epochs are run through all samples. For each sample, we randomly initialize the assignment of candidate phrases labels c and discourse relations d. Then an MCMC chain is con-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
structed with a series of configurations σ = (c, d): at each step, it first samples a discourse structure d based on the proposal distribution q(d′|d,x), and then samples phrase labels conditional on the new discourse relations and previous phrase labels based on q(c′|c,d′,x). Local search is used for both proposal distributions. The new configuration is accepted if it improves on the score by ω(σ′). The parameters w are updated accordingly.
For the scorer ω, we use a weighted combination of F1 scores of phrase selection (F1c) and discourse relation prediction (F1d): ω(σ) = α · F1c + (1− α) · F1d. We fix α to 0.1.
When discourse relations are treated as latent, we initialize discourse relations for each sample with a label in {1, 2, . . . ,K} if there are K relations indicated, and we only use F1c as the scorer.
Input : X = {x}: discussions in the training set, η: learning rate, : number of epochs, δ: number of sampling rounds, ω(·): scoring function, Φ(·): feature functions
Output: feature weights 1|W| ∑ w∈W w Initialize w; W ← {w}; for e = 1 to do
for x in X do // Initialize configuration for
x Initialize c and d; σ = (c,d); for s = 1 to δ do
// New configuration via local search d′ ∼ qd(·|x,d); c′ ∼ qd(·|x, c,d′); σ′ = (c′,d′); σ+ = arg maxσ̃∈{σ,σ′} ω(σ̃); σ− = arg minσ̃∈{σ,σ′} ω(σ̃); ∇̂ = Φ(σ+)− Φ(σ−); ∆ω = ω(σ+)− ω(σ−); // Update parameters if w · ∇̂ < ∆ω & ∆ω 6= 0 then
w′ = w + η · ∇̂; Add w′ inW;
end // Accept or reject new
configuration if σ+ == σ′ then
σ = σ′ end
end end
end Algorithm 1: SampleRank-based joint learning.
3.3 Joint Inference for Prediction
Given a new sample x and learned parameters w, we predict phrase labels and discourse relations as arg maxc,d p(c,d|x,w).
Dynamic programming can be employed to carry out joint inference, however, it would
be time-consuming since our objective function has a large search space for both content and discourse labels. Hence we propose an alternating optimizing algorithm to search for c and d iteratively. Concretely, for each iteration, we first optimize on d by maximizing ∑n i=1,<xi,x′i>∈t
(wd · φd(di, di′ ,x) + wcd ·∑mi j=1 φcd(ci,j , di,x)). Message-passing (Smith and Eisner, 2008) is used to find the best d. In the second step, we search for c that max-
imizes ∑n
i=1,<xi,x′i>∈t (wc · ∑mi j=1 φc(ci,j ,x) +
wcd · ∑mi
j=1 φcd(ci,j , di,x)). We believe that candidate phrases based on the same concepts should have the same predicted label. Therefore, candidates of the same phrase type and sharing the same head word are grouped into one cluster. We then cast our task as an integer linear programming problem.1 We optimize our objective function under constraints: (1) ci,j = ci′,j′ if ci,j and ci′,j′ are in the same cluster, and (2) ci,j ∈ 0, 1, ∀i, j.
The inference process is the same for models trained with latent discourse relations.
3.4 Features
We use features that characterize content, discourse relations, and the combination of both. Content Features. For modeling the salience of content, we calculate the minimum, maximum, and average of TF-IDF scores of words and number of content words in each phrase based on the intuition that important phrases tend to have more content words with high TF-IDF scores (Fernández et al., 2008). We also consider whether the head word of the phrase has been mentioned in preceding turn, which implies the focus of a discussion. The size of the cluster this phrase belongs to is also included. Number of POS tags and phrase types are counted to characterize the syntactic structure. Previous work (Wang and Cardie, 2012) has found that a discussion usually ends with decision-relevant information. We thus identify the absolute and relative positions of the turn containing the candidate phrase in the discussion. Finally, we record whether the candidate phrase is uttered by the main speaker, who speakers the most words in the discussion. Discourse Features. For each discourse unit, we
1We use lpsolve: http://lpsolve. sourceforge.net/5.5/.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
collect the dialogue act types of the current unit and its parent node in discourse tree, whether there is any adjacency pair held between the two nodes (Hakkani-Tur, 2009), and the Jaccard similarity between them. We record whether two turns are uttered by the same speaker, for example, ELABORATION is commonly observed between the turns from the same participant. We also calculate the number of candidate phrases based on the observation that OPTION and SPECIALIZATION tend to contain more informative words than POSITIVE feedback. Length of the discourse unit is also relevant. Therefore, we compute the time span and number of words. To incorporate global structure features, we encode the depth of the node in the discourse tree and the number of its siblings. Finally, we include an order-2 discourse relation feature that encodes the relation between current discourse unit and its parent, and the relation between the parent and its grandparent if it exists. Joint Features. For modeling the interaction between content and discourse, the discourse relation is added to each content feature to compose a joint feature. For example, if candidate c in discussion x has a content feature φ[avg−TFIDF ](c,x) with a value of 0.5, and its discourse relation d is POSITIVE, then the joint feature takes the form of φ[avg−TFIDF,Positive](c, d,x) = 0.5.
4 Datasets and Experimental Setup
Meeting Corpora. We evaluate our joint model on two meeting corpora with rich annotations: the AMI meeting corpus (Carletta et al., 2006) and the ICSI meeting corpus (Janin et al., 2003). AMI corpus consists of 139 scenario-driven meetings, and ICSI corpus contains 75 naturally occurring meetings. Both of the corpora are annotated with dialogue acts, adjacency pairs, and topic segmentation. We treat each topic segment as one discussion, and remove discussions with less than 10 turns or labeled as “opening”. 694 discussions from AMI and 1139 discussions from ICSI are extracted, and these two datasets are henceforth referred as AMI-FULL and ICSI-FULL. Acquiring Gold-Standard Labels. Both corpora contain human constructed abstractive summaries and extractive summaries on meeting level. Short abstracts, usually in one sentence, are constructed by meeting participants — participant summaries,
and external annotators — abstractive summaries. Dialogue acts that contribute to important output of the meeting, e.g. decisions, are identified and used as extractive summaries, and some of them are also linked to the corresponding abstracts.
Since the corpora do not contain phrase-level importance annotation, we induce gold-standard labels for candidate phrases based on the following rule. A candidate phrase is considered as a positive sample if its head word is contained in any abstractive summary or participant summary.
Furthermore, a subset of discussions in AMIFULL are annotated with discourse structure and relations based on Twente Argumentation Schema (TAS) by Rienks et al. (2005)2. A tree-structured argument diagram (as shown in Figure 1) is created for each discussion or a part of the discussion. The nodes of the tree contain partial or complete speaker turns, and discourse relation types are labeled on the links between the nodes. In total, we have 129 discussions annotated with discourse labels. This dataset is called AMI-SUB hereafter. Experimental Setup. 5-fold cross validation is used for all experiments. All real-valued features are uniformly normalized to [0,1]. For the joint learning algorithm, we use 10 epochs and carry out 50 sampling for MCMC for each training sample. The learning rate is set to 0.01. We run the learning algorithm for 20 times, and use the average of the learned weights as the final parameter values. For models trained with latent discourse relations, we fix the number of relations to 9. Baselines and Comparisons. For both phrasebased content selection and discourse relation prediction tasks, we consider (1) a baseline that always predicts the majority label (Majority), and (2) a random baseline. Previous work has shown that Support Vector Machines (SVMs)-based classifiers achieve state-of-the-art performance for keyphrase selection in meetings (Fernández et al., 2008; Wang and Cardie, 2013) and discourse parsing for formal text (Hernault et al., 2010). Therefore, we compare with linear SVM-based classifiers, trained with the same feature set of content features or discourse features. We fix the trade-off parameter to 1.0 for all SVM-based experiments. For discourse relation prediction, we use one-vsrest strategy to build multiple binary classifiers.
2There are 9 types of relations in TAS: POSITIVE, NEGATIVE, UNCERTAIN, REQUEST, SPECIALIZATION, ELABORATION, OPTION, OPTION EXCLUSION, and SUBJECT-TO.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Acc F1 Comparisons Baseline (Majority) 60.1 37.5 Baseline (Random) 50.0 33.1 SVM (w content features in § 3.4) 57.8 54.6 Our Models Joint-Learn + Joint-Inference 63.2∗ 62.6∗ Joint-Learn + Separate-Inference 57.9 57.8 Separate-Learn 53.4 52.6 Our Models (Latent Discourse) w/ True Attachment Structure Joint-Learn + Joint-Inference 60.3∗ 60.3∗ Joint-Learn + Separate-Inference 56.4 56.2 w/o True Attachment Structure Joint-Learn + Joint-Inference 56.4 56.4 Joint-Learn + Separate-Inference 52.7 52.3
Table 1: Phrase-based content selection performance on AMI-SUB with accuracy (acc) and F1. We display results of our models trained with gold-standard discourse relation labels and with latent discourse relations. For the later, we also show results based on True Attachment Structure, where the gold-standard attachments are known, and without the True Attachment Structure. Our models that significantly outperform SVM-based model are highlighted with ∗ (p < 0.05, paired t-test). Best result for each column is in bold.
Acc F1 Comparisons Baseline (Majority) 51.2 7.5 Baseline (Random) 11.1 1.9 SVM (w discourse features in § 3.4) 51.2 22.8 Our Models Joint-Learn + Joint-Inference 58.0∗ 21.7 Joint-Learn + Separate-Inference 59.2∗ 23.4 Separate-Learn 58.2∗ 25.1
Table 2: Discourse relation prediction performance on AMI-SUB. Our models that significantly outperform SVM-based model are highlighted with ∗ (p < 0.05, paired t-test).
5 Experimental Results
5.1 Phrase Selection and Discourse Labeling
Here we present the experimental results on phrase-based content selection and discourse relation prediction. We experiment with two variations of our joint model: one is trained on goldstandard discourse relations, the other is trained by treating discourse relations as latent models as described in Section 3.1. Remember that we have gold-standard argument diagrams on the AMISUB dataset, we can thus conduct experiments by assuming the True Attachment Structure is given for latent versions. When argument diagrams are not available, we build a tree among the turns in each discussion as follows. Two turns are attached if there is any adjacency pair between them. If one turn is attached to more than one previous turns, the closest one is considered. For the rest of the turns, they are attached to the preceding turn.
We also investigate whether joint learning and joint inference can produce better prediction performance. We consider joint learning with separate inference, where only content features or discourse features are used for prediction. We further study learning separate classifiers for content selection and discourse relations without joint features (Separate-Learn).
We first show the phrase selection and discourse relation prediction results on AMI-SUB in Tables 1 and 2. As shown in Table 1, our models, trained with gold-standard discourse relations or latent ones with true attachment structure, yield significant better accuracy and F1 scores than SVM-based classifiers trained with the same feature sets for phrase selection (paired t-test, p < 0.05). Moreover, both Tables 1 and 2 demonstrate that joint learning usually produces superior performance for both tasks than separate learning. Combined with joint inference, our model obtains the best accuracy and F1 on phrase selection. This indicates that leveraging the interplay between content and discourse boost the prediction performance. Similar results are achieved on AMI-FULL and ICSI-FULL in Table 3, where latent discourse relations without true attachment structure are employed for training.
5.2 Phrase-Based Extractive Summarization
We further evaluate whether the prediction of the content selection component can be used for summarizing the key points on discussion level. For each discussion, salient phrases identified by our model are concatenated in sequence for use as the summary. We consider two types of gold-standard summaries. One is utterance-level extractive summary, which consists of human labeled summaryworthy utterances. The other is abstractive summary, where we collect human abstract with at least one link from summary-worthy utterances.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Extractive Summaries as Gold-Standard ROUGE-1 ROUGE-SU4 Len Prec Rec F1 Prec Rec F1 Longest DA 30.9 64.4 15.0 23.1 58.6 9.3 15.3 Centroid DA 17.5 73.9 13.4 20.8 62.5 6.9 11.3 SVM 49.8 47.1 24.1 27.5 22.7 10.7 11.8 Our Model 66.6 45.4 44.7 41.1∗ 24.1∗ 23.4∗ 20.9∗ Our Model-latent 85.9 42.9 49.3 42.4∗ 21.6 25.7∗ 21.3∗ Abstractive Summaries as Gold-Standard
ROUGE1 ROUGE-SU4 Len Prec Rec F1 Prec Rec F1
Longest DA 30.9 14.8 5.5 7.4 4.8 1.4 1.9 Centroid DA 17.5 24.9 5.6 8.5 11.6 1.4 2.2 SVM 49.8 13.3 9.7 9.5 4.4 2.4 2.4 Our Model 66.6 12.6 18.9 13.1∗ 3.8 5.5∗ 3.7∗ Our Model-latent 85.9 11.4 20.0 12.4∗ 3.3 6.1∗ 3.5∗
Table 4: ROUGE scores for phrase-based extractive summarization evaluated against human-constructed utterance-level extractive summaries and abstractive summaries. Our models that statistically significantly outperform all comparisons are highlighted with ∗ (p < 0.05, paired t-test).
We calculate scores based on ROUGE (Lin and Hovy, 2003), which is a popular tool for evaluating text summarization (Gillick et al., 2009; Liu and Liu, 2010). ROUGE-1 (unigrams) and ROUGE-SU4 (skip-bigrams with at most 4 words in between) are used. Following previous work on meeting summarization (Riedhammer et al., 2010; Wang and Cardie, 2013), we consider two dialogue act-level summarization baselines: (1) LONGEST DA in each discussion is selected as the summary, and (2) CENTROID DA, the one with the highest TF-IDF similarity with all DAs in the discussion. We also compare with summaries consisting of salient phrases predicted by an SVM classifier trained with on our content features.
From the results in Table 4, we can see that phrase-based extractive summarization methods by our system and SVM-based classifiers can yield better ROUGE scores for recall and F1 than baselines that extract the whole sentences. Meanwhile, our system significantly outperforms the SVMbased classifiers when evaluated on ROUGE recall and F1, while achieving comparable precision.
Sample summaries by our model along with two baselines are displayed in Figure 2. Utterancelevel extract-based baselines unavoidably contain disfluency and unnecessary details. Our phrasebased extractive summary is able to capture the key points from both the argumentation process and important outcomes of the conversation.
5.3 Further Analysis and Discussions
Features Analysis. We first discuss salient features with top weights learned by our joint model.
For content features, main speaker tends to utter more salient content. Higher TF-IDF scores also indicate important phrases. For discourse features, structure features matter the most. For instance, jointly modeling the discourse relation of the parent node along with the current node can lead to better inference. An example is that giving more details on the proposal (ELABORATION) tends to lead to POSITIVE feedback. For joint features, features that composite “phrase mentioned in previous turn” and relation POSITIVE feedback or REQUEST yield higher weight, which are indicators for both key phrases and discourse relations. We also find that main speaker information composite with ELABORATION and UNCERTAIN are associated with high weights. Error Analysis and Potential Directions. Taking a closer look at our prediction results, one major source of incorrect prediction for phrase selection is based on the fact that similar concepts might be expressed in different ways, and our model predicts inconsistently for different variations. For example, participants use both “thick” and “two centimeters” to talk about the desired shape of a remote control. However, our model does not group them into the same cluster and later makes different predictions. Furthermore, identifying discourse relations in dialogues is still a challenging task. For instance, “I wouldn’t choose a plastic case” should be labeled as OPTION EX-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
CLUSION, if the previous turns talk about different options. Otherwise, it can be labeled as NEGATIVE. Therefore, models that better handle semantics and context need to be considered.
6 Predicting Consistency of Understanding
In this section, we test whether our joint model can be utilized to predict the consistency among team members’ understanding of their group decisions, which is defined as consistency of understanding (COU) in Kim and Shah (2016).
Kim and Shah (2016) establish gold-standard COU labels on a portion of AMI discussions, by comparing participant summaries to determine whether participants report the same decisions. If all decision points are consistent, the associated topic discussion is labeled as consistent; otherwise, the discussion is identified as inconsistent. Their annotation covers the AMI-SUB dataset. Therefore, we run the prediction experiments on AMI-SUB by using the same annotation.3 Out of total 129 discussions in AMI-SUB, 86 discussions are labeled as consistent and 43 are inconsistent.
We construct three types of features by using our model’s predicted labels. Firstly, we learn two versions of our model based on the “consistent” discussions and the “inconsistent” ones in the training set, with learned parameters wcon and wincon. For a discussion in the test set, these two models output two probabilities pcon = maxc,d P (c,d|x,wcon) and pincon = maxc,d P (c,d|x,wincon). We use pcon − pincon as a feature.
Furthermore, we consider discourse relations of length one and two from the discourse structure tree. Intuitively, some discourse relations, e.g., ELABORATION followed by several POSITIVE implied consistent understanding.
The third feature is based on word entrainment, which has been shown to correlate with task success for groups (Nenkova et al., 2008). Using the formula in Nenkova et al. (2008), we compute the average word entrainment between the main speaker who utters the most words and all the other participants. The content words in the salient phrases predicted by our model is considered for entrainment computation. Results. Leave-one-out is used for experiments.
3We thank Joseph Kim for providing the dataset and rerunning their system based on our setup.
For training, our features are constructed from gold-standard phrase and discourse labels. Predicted labels by our model is used for constructing features during testing. SVM-based classifier is used. A majority class baseline and a random baseline are constructed as well. We also consider an SVM classifier trained with ngram features (unigrams and bigrams). Finally, we compare with the state-of-the-art method in Kim and Shah (2016), where discourse-relevant features and head gesture features are utilized in Hidden Markov Models to predict the consistency label.
The results are displayed in Table 5. All SVMs trained with our features surpass the ngrams-based baseline. Especially, the discourse features, word entrainment feature, and the combination of the three, all significantly outperform the state-of-theart system by Kim and Shah (2016).
7 Conclusion
We presented a joint model for performing phraselevel content selection and discourse relation prediction in spoken meetings. Experimental results on AMI and ICSI meeting corpora showed that our model can outperform SVM-based classifiers trained with the same feature sets. Further evaluation on the task of predicting consistencyof-understanding in meetings demonstrated that classifiers trained with features constructed from our model’s predictions produced superior performance compared to state-of-the-art model.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper proposes a joint model of salient phrase selection and discourse relation prediction in spoken meeting. Experiments using meeting corpora show that the proposed model has higher performance than the SVM-based classifier.  - Strengths: The paper is written to be easy to read. Technical details are described fully, and high performance is also shown in experimental evaluation. It also shows useful comparisons with related research in the field of discourse structure analysis and key phrase identification. It is interesting to note that not only the performance evaluation of phrase selection from discourse, discourse relation labeling, and summary generation as their applications, but also application to the prediction of the consistency of  understanding by team members is also verified .  - Weaknesses: Jointly Modeling salient phrase extraction and discourse relationship labeling between speaker turns has been proposed. If intuitive explanation about their interactivity and the usefulness of considering it is fully presented.  - General Discussion: SVM-based classifier is set as a comparative method in the experiment. It would be useful to mention the validity of the setting.