Topical Coherence in LDA-based Models through Induced Segmentation
This paper presents an LDA-based model that generates topically coherent segments within documents by jointly segmenting documents and assigning topics to their words. The coherence between topics is ensured through a copula, binding the topics associated to the words of a segment. In addition, this model relies on both document and segment specific topic distributions so as to capture fine grained differences in topic assignments. We show that the proposed model naturally encompasses other state-of-the-art LDA-based models designed for similar tasks. Furthermore, our experiments, conducted on six different publicly available datasets, show the effectiveness of our model in terms of perplexity, Normalized Pointwise Mutual Information, which captures the coherence between the generated topics, and the Micro F1 measure for text classification.
1 Introduction
Since the seminal works of Hofmann (1999) and Blei et al. (2003), there have been several developments in probabilistic topic models. Many extensions have indeed been proposed for different applications, including ad-hoc information retrieval (Wei and Croft, 2006), clustering search results (Zeng et al., 2004) and driving faceted browsing (Mimno and McCallum, 2007). In most of these studies, the initial exchangeability assumptions of PLSA and LDA, stipulating that words within a document are interdependent, has led to incoherent topic assignments within semantically meaningful text units, even though the importance of having topically coherent phrases is generally admitted (Griffiths et al., 2005). More recently, (Balikas
et al., 2016b) has shown that binding topics, so as to obtain more coherent topic assignments, within such text segments as noun phrases improves the performance (e.g. in terms of perplexity) of LDAbased models. The question nevertheless remains as to which segmentation one should rely on.
Furthermore, text segments can refer to topics that are barely present in other parts of the document. For example, the segment “the Kurdish regional capital” in the sentence1 “A thousand protesters took to the main street in Erbil, the Kurdish regional capital, to condemn a new law requiring all public demonstrations to have government permits.” refers to geography in a document that is mainly devoted to politics. Relying on a single topic distribution, as done in most previous studies including (Balikas et al., 2016b), may prevent one from capturing those segment specific topics.
In this paper, we propose a novel LDA-based model that automatically segments documents into topically coherent sequences of words. The coherence between topics is ensured through copulas (Elidan, 2013) that bind the topics associated to the words of a segment. In addition, this model relies on both document and segment specific topic distributions so as to capture fine grained differences in topic assignments. A simple switching mechanism is used to select the appropriate distribution (document or segment specific) for assigning a topic to a word. We show that this model naturally encompasses other state-of-the-art LDA-based models proposed to accomplish the same task, and that it outperforms these models over six publicly available collections in terms of perplexity, Normalized Pointwise Mutual Information (NPMI), a measure used to assess the coherence of topics with documents, and the Micro F1-measure in a text classification context.
1This sentence is taken from New York Times news (NYT) collection described in Section 4.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Related work
Probabilistic Latent Semantic Analysis (PLSA) proposed by (Hofmann, 1999) is the first probabilistic model that explains the generation of cooccurrence data using latent radom topics and, the EM algorithm for parameter estimation. The model was found more flexible and scalable than the Latent Semantic Analysis (Deerwester et al., 1990), which is based on the singular value decomposition of the document-term matrix, however PLSA is not a generative model as parameter estimation should be performed at each addition of new documents. To overcome this drawback, Blei et al. (2003) proposed the Latent Dirichlet Allocation (LDA) by assuming that the latent topics are random variables sampled from a Dirichlet distribution and that the generated words, occurring in a document, are exchangeable. The interdependence assumption allows the parameter estimation and the inference of the LDA model to be carried out efficiently, but it is not realistic in the sense that topics assigned to similar words of a text span are generally incoherent.
Different studies, presented in the following sections, attempted to remedy this problem and they can be grouped in two broad families depending on whether they make use of external knowledgebased tools or not in order to exhibit text structure for word-topic assignment.
2.1 Knowledge-based topic assignments
The main assumption behind these models are that text-spans such as sentences, phrases or segments are related in their content. Therefore, the integration of these dependent structures can help to discover coherent latent topics for words. Different attempts to combine LDA-based models with statistical tools to discover document structures have been successfully proposed, such as the study of Griffiths et al. (2005) who investigated the effect of combining a Hidden Markov Model with LDA to capture long and short distance dependencies. Similarly, (Boyd-Graber and Blei, 2008; Balikas et al., 2016a,b) integrated text structure exhibited by a parser or a chunker in their topic models. In this line, Du et al. (2013) following (Du et al., 2010) presented a hierarchical Bayesian model for unsupervised topic segmentation. This model integrates a boundary sampling method used in a Bayesian segmentation model introduced by Purver et al.(2006) to the topic model. For inference, a
non-parametric Markov Chain inference is used that splits and merges the segments while a PitmanYor process (Teh, 2006) binds the topics. Recently, Tamura and Sumita (2016) extended this idea to the bilingual setting. They assume that documents consist of segments and the topic distribution of each segment is generated using a Pitman-Yor process (Teh, 2006).
Though, the topic assignments follow the structure of the text; these models suffer from the bias of statistical or linguistic tools they rely on. To overpass this limitation, other systems integrated automatically the extraction of text structure, in the form of phrases, in their process.
2.2 Knowledge-free topic assignments
This type of models extract text-spans using ngram counts and word collections and use bigrams to integrate the order of words as well as to capture the topical content of a phrase (Lau et al., 2013). In (Wang et al., 2007), depending on the topic a particular bigram can be either considered as a single token or as two unigrams. Further, Wang et al. (2009) merged topic models with a unigram model over sentences that assigns topics to the sentences instead of the words.
Our proposed approach also does not make use of external statistical tools to find text segments. The main difference with the previous knowledgefree topic model approaches is that the proposed approach assigns topics to words based on two, segment-specific and document-specific distributions selected from a Bernoulli law. Topics within segments are then constrained using copulas that bind their distributions. In this way, segmentation is embedded in the model and it naturally comes along with the topic assignment.
3 Joint latent model for topics and segments
We define here a segment as a topically coherent sequence of contiguous words. By topically coherent, we mean that, even though words in a segment can be associated to different topics, these topics are usually related. This view is in line with the one expressed in (Balikas et al., 2016b), in which a latent topic model, referred to as copLDA in the remainder, includes a binding mechanism between topics within coherent text spans, defined in their study as noun phrases (NPs). The relation between topics is captured through a copula that provides
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
α
θd
z1 zn
w1 wn
φ βλ
|S| D
K
. . .
. . .
(a) copLDA
α
θd
z1 zn
w1 wnS
|Sd|
φ βλ
|S| D
K
. . .
. . .
(b) segLDAcopp=0
α
θd
fn
θd,s,n
p
θs
zn
wnS
|Sd|
φ β
|S| D
K
(c) segLDAcopλ=0
α
θd
f1 fn
θd,s,1 θd,s,n
p
θs
z1 zn
w1 wnS
|Sd|
φ βλ
|S| D
K
. . .
. . .
. . .
. . .
(d) segLDAcop
Figure 1: Graphical model for Copula LDA (copLDA), extension of Copula LDA with segmentation (segLDAcopp=0), LDA with segmentation and topic shift (segLDAcopλ=0) and complete model (segLDAcop).
a joint probability for all the topics used in a segment. That is, to generate words in a segment, one first jointly generates all the word specific topics z via a copula, and then generates each word in the segment from its word specific topic and the word-topic distribution φ. Figure 1(a) illustrates this.
Copulas are particularly useful when modeling dependencies between random variables, as the joint cumulative distribution function (CDF) FX1,··· ,Xn of any random vector X = (X1, · · · , Xn) can be written as a function of its marginals, according to Sklar’s Theorem (Nelsen, 2006):
FX1,··· ,Xn(x1, · · · , xp) = C(FX1(x1), · · · , FXn(xn))
where C is a copula. For latent topic models, as discussed in (Amoualian et al., 2016), Frank’s copula is particularly interesting as (a) it is invariant by permutations and associative, as are the words and topics z in each segment due to the exchangeability assumption, and (b) it relies on a single parameter (denoted λ here) that controls the strength of dependence between the variables and is thus easy to implement. In Frank’s copula, when the parameter λ approaches 0, the variables are independent of each other, whereas when λ approaches +∞, the variables take the same value. For further details on copulas, we refer the reader to (Nelsen, 2006).
One important problem, however, with copLDA is its reliance on a predefined segmentation. Although the information brought by the segmentation based on NPs helps to improve topic assignment, it may not be flexible enough to capture all the possible segments of a text. It is easy to correct this problem by considering all possible segmentations of a document and by choosing the most appropriate one at the same time that topics are assigned to words. This is illustrated in Figure 1(b), where a segmentation S is chosen from the set Sd of possible segmentations for a document d, and where each segment in S are generated in turn. We refer to the associated model as segLDAcopp=0 for reasons that will become clear later.
Another point to be noted about copLDA (and segLDAcopp=0) is that the topics used in each segment come from the same document specific topic distribution θd. This entails that, in these models, one cannot differentiate the main topics of a document from potential segment specific topics that can explain some parts of it. Indeed, some text segments can refer to topics that are barely present in other parts of the document; relying on a single topic distribution may prevent one from capturing those segment specific topics.
It is possible to overcome this difficulty by generating a segment specific topic distribution as illustrated in Figure 1(c) (this model is referred to
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
as segLDAcopλ=0, again for reasons that will become clear later). However, as some words in a segment can be associated to the general topics of a document, we introduce a mechanism to choose, for each word in a segment, a topic either from the segment specific topic distribution θs or from the document specific topic distribution θd (this mechanism is similar to the one used for routes and levels in (Paul and Girju, 2010)). The choice between them is based on the Bernoulli variable f , as explained in the generative story given below.
The above developments can be combined in a single, complete model, illustrated in Figure 1(d) and detailed below. We will simply refer to this model as segLDAcop.
3.1 Complete generative model
As in standard LDA based models, with V denoting the size of the vocabulary of the collection and K the number of latent topics, β and φk, 1 ≤ k ≤ K, are V dimensional vectors, α and θ (i.e., θd, θs, θd,s,n) are K dimensional vectors, whereas zn takes value in {1, · · · ,K}. Lower indices are used to denote coordinates of the above vectors. Lastly, Dir denotes the Dirichlet distribution, Cat the categorical distribution (which is a multinomial distribution with one draw) and we omit, as is usual, the generation of the length of the document. The complete model segLDAcop is then based on the following generative process:
1. Generate, for each topic k, 1 ≤ k ≤ K, a distribution over the words: φk ∼ Dir(β);
2. For each document d, 1 ≤ d ≤ D:
(a) Choose a document specific topic distribution: θd ∼ Dir(α);
(b) Choose a segmentation S of the document uniformly from the set of all possible segmentations Sd: P (S) = 1|Sd| ; (c) For each segment s in S: (i) Choose a segment specific topic distri-
bution: θs ∼ Dir(α); (ii) For each position n in s, choose fn ∼
Ber(p) and set:
θd,s,n = { θs if fn = 1 θd otherwise
(iii) Choose topics Zs = {z1, . . . , zn} from Frank’s copula with parameter λ and marginals Cat(θd,s,n);
(iv) For each position n in s, choose word wn: wn ∼ Cat(φzn).
As on can note, the generative process relies on a segmentation uniformly chosen from the set of possible segmentations (step 2.b) to generate related topics within each segment (Frank’s copula in step 2.c.(iii)), the distribution underlying each word specific topic zn being either specific to the segment or general to the document (steps 2.c.(i) and 2.c.(ii)). The other steps are similar to the standard LDA steps.
As in almost all previous studies on LDA, α and β are considered fixed and symmetric, each coordinate of the vector being equal: α1 = · · · = αK . The hyperparameters p (∈ [0, 1]) of the Bernoulli distribution and λ (∈ [0,+∞]) of Frank’s copula respectively regulate the choice between the segment specific and the document specific topic distributions and the strength of the dependence between topics in a segment. As for the other hyperparameters, we consider them fixed here (the values for all hyperparameters are given in Section 4).
As mentioned before, all the models presented in Figure 1 are special cases of the complete model segLDAcop: hence segLDAcopλ=0 is obtained by dropping the topic dependencies, which amounts to setting λ to (a value close to) 0, segLDAcopp=0 is obtained by relying only on the topic distribution obtained for the document, which amounts to setting p to 0, and the previously introduced copLDA model is obtained by setting p to 0, and fixing the segmentation.
3.2 Inference with Gibbs sampling
The parameters of the complete model can be directly estimated through Gibbs sampling. The Gibbs updates for the parameters φ and θ are the same as the ones for standard LDA (Blei et al., 2003). The parameters fn are directly estimated through: fn ∼ Ber(p). Lastly, for the variables z, we follow the same strategy as the one described in (Balikas et al., 2016b) and based on (Amoualian et al., 2016), leading to:
P (Zs|Z−s,W,Θ,Φ, λ) = p(Zs|Θ, λ) ∏ n φznwn
where W denotes the document collection, and Θ and Φ the sets of all θ and φk, 1 ≤ k ≤ K, vectors. p(Zs|Θ, λ) is obtained by Frank’s copula with parameter λ and marginals Cat(θd,s,n). As is standard in topic models, the notation −s means excluding the information from s.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
From the above equation, one can formulate an acceptance/rejection algorithm based on the following steps: (a) sample Zs from p(Zs|Θ, λ) using Frank’s copula, and (b) accept the sample with probability ∏ n φ zn wn , where n runs over all the positions in segment s.
3.3 Efficient segmentation
As topics may change from one sentence to another, we assume here that segments cannot overlap sentence boundaries. The different segmentations of a document are thus based on its sentence segmentations. In the remainder, we use L to denote the maximum length of a segment and g(M ;L) to denote the number of segmentations in a sentence of length M , each segment comprising at most L words.
Generating all possible segmentations of a sentence and then selecting one at random is not an efficient process as the number of segments rapidly grows with the length of the sentence. In practice, however, one can define an efficient segmentation on the basis of the following proposition, the proof of which is given in Appendix A:
Proposition 3.1. Let lsi be the random variable associated to the length of the segment starting at position i in a sentence of length M (positions go from 1 to M and lsi takes value in {1, · · · , L}). Then P (lsi = l) := g(M+1−i−l);L) g(M+1−i;L) defines a probability distribution over lsi .
Furthermore, the following process is equivalent to choosing sentence segmentations uniformly from the set of possible segmentations.
From pos. 1, repeat till end of sentence: (a) Generate segment length acc. to P; (b) Add segment to current segmentation; (c) Move to position after the segment.
In practice, we thus replace steps 2.b and 2.c of the generative story by a loop over all sentences, and in each sentence use the process described in Prop, 3.1. Furthermore, as described in Appendix A, the values of g needed to compute P (lsi = l) can be efficiently computed by recurrence.
4 Experiments
We conducted a number of experiments aimed at studying the impact of simultaneously segmenting and assigning topics to words within segments using the proposed segLDAcop model.
Datasets: We considered six publicly available datasets derived from Pubmed2 (Tsatsaronis et al., 2015), Wikipedia (Partalas et al., 2015), Reuters3 and New York Times (NYT)4 (Yao et al., 2016). The first two collections were considered in (Balikas et al., 2016a), we followed their setup by considering 3 subsets of Wikipedia with different number of classes (namely, Wiki0, Wiki1 and Wiki2). The Reuters dataset comes from Reuters-21578, Distribution 1.0 as investigated in (Bird et al., 2009) and the NYT dataset is collected from full text of New York Times global news, from January 1st to December 31st, 2011.
These collections were processed following (Blei et al., 2003) by removing a standard list of 50 stop words, lemmatizing, lowercasing and keeping only words made of letters. To deal with relatively homogeneous collections, we also removed documents that are too long. The statistics of these datasets, as well as the admissible maximal length for documents, in terms of the number of words they contain, can be found in Table 1.
Settings: We compared our models (segLDAcopp=0, segLDAcopλ=0, segLDAcop) with three models, namely the standard LDA model, and two previously introduced models aiming at binding topics within segments:
1. LDA: Standard Latent Dirichlet Allocation implemented using collapsed Gibbs sampling inference (Griffiths and Steyvers, 2004)5. Note that there are neither segmentation nor topic
2https://github.com/balikasg/ topicModelling/tree/master/data
3https://archive.ics.uci.edu/ ml/datasets/Reuters-21578+Text+ Categorization+Collection
4https://github.com/yao8839836/COT/ tree/master/data
5http://gibbslda.sourceforge.net
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Models Wiki0 Wiki1 Wiki2 Pubmed Reuters NYT 20 100 20 100 20 100 20 100 20 100 20 100 LDA 853.7 370.9 1144.6 541.1 1225.2 570.6 1267.8 628.7 210.6 118.8 1600.1 1172.1 senLDA 958.4 420.5 1236.7 675.3 1253.1 625.2 1346.3 674.3 254.3 173.6 1735.9 1215.3 copLDA 753.1 264.3 954.1 411.5 1028.6 420.6 1031.5 483.2 206.3 101.3 1551.5 1063.2 segLDAcopp=0 670.2 235.4 904.2 382.4 975.7 409.2 985.5 459.3 194.2 96.7 1504.2 1033.2 segLDAcopλ=0 655.1 222.1 890.3 370.2 949.2 404.3 971.3 451.2 190.1 91.3 1474.6 1014.3 segLDAcop 621.2 213.5 861.2 358.6 934.7 394.4 960.4 442.1 182.1 87.5 1424.2 992.3
Table 2: Perplexity with respect to different number of topics (20 and 100).
binding mechanisms in this model;
2. senLDA: Sentence LDA, introduced in (Balikas et al., 2016a), which forces all words within a sentence to be assigned to the same topic. The segments considered thus correspond to sentences, and the binding between topics within segments is maximal as all word specific topics are equal;
3. copLDA: Copula LDA, introduced in (Balikas et al., 2016b) already discussed before, which relies on two types of segments, namely NPs (extracted with the nltk.chunk package (Bird et al., 2009)) and single words. In addition, a copula is also used to bind topics within NPs, from the document specific topic distribution.
Both senLDA and copLDA implementations, can be found in https://github.com/ balikasg/topicModelling.
In all models α and β play a symmetric role and are respectively fixed to 1/K, following (Asuncion et al., 2009). For copula based models, λ is set to 5, following (Balikas et al., 2016b). As already discussed, p is set to 0 for segLDAcopp=0; it is set to 0.5 for segLDAcop so as not to privilege a priori one topic distribution (document or segment specific) over the other. For sampling from Frank’s copula, we relied on the R copula package (Hofert and Maechler, 2011) 6. We chose L (the maximum length of a segment) using line search forL ∈ [2, 5] and used L = 3 in all our experiments. Finally, to illustrate the behaviors of the different models with different number of topics, we present here the results obtained with K = 20 and K = 100.
We now compare the different models along three main dimensions: perplexity, use of topic representations for classification and topic coherence.
4.1 Perplexity
We first randomly split here all the collections, using 75% of them for training, and 25% for testing.
In order to see how well the models fit the data and following (Blei et al., 2003), we first evaluated the methods in terms of perplexity defined as:
Perplexity = exp
( − ∑ d∈D ∑ w∈d log ∑K k=1 θ d kφ
k w∑
d∈D |d|
) ,
where d is a test document from the test set D, and |d| is the total number of words in d, and K is the total number of topics. The lower the perplexity is, the better the model fits the test data. Table 2 shows perplexities of different methods for K = 20 and K = 100 topics.
50 100 150 200 250 300 350
1,400
1,600
1,800
2,000
2,200
Iterations
Pe rp
le xi
ty
NYT
LDA senLDA copLDA segLDAcopp=0 segLDAcopλ=0 segLDAcop
Figure 2: Perplexity with respect to training iteration on NYT collection (20 topics).
From Table 2, it comes out that the best performing model in terms of perplexity over all datasets and for different number of topics is segLDAcop. Further, segLDAcopλ=0, that uses both document and segment specific topic distributions, performs better than segLDAcopp=0, which in turn outperforms copLDA, bringing evidence that using all possible segmentations rather than only NPs unit extracted using a chunker yields a more flexible and natural topic assignment.
6Our complete code will be available for research purposes.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Models Wiki0 Wiki1 Wiki2 Pubmed Reuters 20 100 20 100 20 100 20 100 20 100 LDA 55.3 63.5 42.4 51.4 41.2 48.7 54.1 63.5 75.5 82.7 senLDA 41.4 53.2 33.5 44.5 36.4 40.9 50.2 62.5 69.4 74.2 copLDA 51.2 62.7 43.4 52.1 40.8 46.5 53.5 63.1 75.2 81.5 segLDAcopp=0 59.1 64.2 44.8 51.2 42.3 50.1 55.4 63.1 76.8 82.5 segLDAcopλ=0 61.1 67.4 46.5 53.8 44.1 52.2 57.1 65.2 79.6 84.4 segLDAcop 62.3 68.4 48.4 55.2 44.8 53.5 59.3 66.5 80.2 85.1
Table 3: MiF score (percent) with respect to different number of topics (20 and 100).
segLDAcop also converges faster than the other methods to its minimum as it is shown in Figure 2, depicting the evolution of perplexity of different models over the number of iterations on the NYT collection (a similar behavior is observed on the other collections).
4.2 Topical induced representation for classification
Some studies compare topic models using extrinsic tasks such as document classification. In this case, it is possible to reduce the dimensionality of the representation space by using the induced topics (Blei et al., 2003). In this study, we first randomly splitted the datasets, except NYT that does not contain class information, into training (75%) and test (25%) sets. We then applied SVMs with a linear kernel; the value of the hyperparameter C was found by cross-validation over the training set {0.01, 0.1, 1, 10, 100}. For datasets where certain documents have more than one label (Pubmed, Reuters), we used the one-versus-all approach for performing multi-label classification.
In Table 3, we report the Micro F1 (MiF) score of different models on the test sets. Again, the best results are obtained with segLDAcop, followed by segLDAcopλ=0. This shows the importance of relying on both document and segment specific topic distributions. As conjectured before, our model is able to captures fine grained topic assignments within documents. In addition, all models relying on an inferred segmentation (segLDAcopp=0, segLDAcopλ=0, segLDAcop) outperform the models relying on fixed segmentations (sentences or NPs). This shows the importance of being able to discover flexible segmentations for assigning topics within documents.
4.3 Topic coherence
Another common way to evaluate topic models is by examining how coherent the produced topics are. Doing this manually is a time consuming pro-
cess and cannot scale. To overcome this limitation the task of automatically evaluating the coherence of topics produced by topic models received a lot of attention (Mimno et al., 2011). It has been found that scoring the topics using co-occurrence measures, such as the pointwise mutual information (PMI) between the top-words of a topic, correlates well with human judgments (Newman et al., 2010). For this purpose an external, large corpus is used as a meta-document where the PMI scores of pairs of words are estimated using a sliding window. As discussed above, calculating the co-occurrence measures requires selecting the top-N words of a topic and performing the manual or automatic evaluation. Hence, N is a hyper-parameter to be chosen and its value can impact the results. Very recently, Lau and Baldwin (2016) showed that N actually impacts the quality of the obtained results and, in particular, the correlation with human judgments. In their work, they found that aggregating the topic coherence scores over several topic cardinalities leads to a substantially more stable and robust evaluation.
Following the findings of Lau and Baldwin (2016) and using (Newman et al., 2010)’s equation, we present in Figure 3 the topic coherence scores as measured by the Normalized Pointwise Mutual Information (NPMI) . Their values are in [- 1,1], where in the limit of -1 two words w1 and w2 never occur together, while in the limit of +1 they always occur together (complete co-occurrence). For the reported scores, we aggregate the topic coherence scores over three different topic cardinalities: N ∈ {5, 10, 15}. segLDAcop model which uses copulas and segmentation together, shows the best score for the given reference meta-data (Wikipedia) in all of the datasets. It should be noted that segLDAcopλ=0 which has not copula binder inside the model has less improvement against the segLDAcopp=0 which has the copula. This means using copula has more effect on the topic coherence than only the segment-specific topic distribution.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Wiki0 Wiki1 Wiki2 PubmedReuters NYT
4
6
8
10
12
5.1
6
8
9
4.4
7.1
6.5 6.7
9.2
10.4
6.1
8.2
6.2 6.3
9.4
10.1
5.8
8.5
6.3
7.2
10.1
10.9
6.4
8.9
6
6.9
9.9 10.3
6
8.6
6.9
7.6
10.5
11.5
6.8
9.2
N PM
I( %
) LDA senLDA copLDA segLDAcopp=0 segLDAcopλ=0 segLDAcop
Figure 3: Topic coherence (NPMI) score with respect to 100 of topics.
4.4 Visualization
In order to illustrate the results obtained by segLDAcop, we display in Figure 4 the top 10 most probable words over 5 topics (K = 20) for the Reuters dataset, for both segLDAcop and LDA. In segLDAcop, topic 1, the top-ranked words are mostly relevant to the topic “date” (e.g., march, january, year, fall, february, week). However, a similar topic learned by LDA appears to involve less such words (year, january, february), indicating a less coherent topic.
Figure 5 illustrates another aspect of our model, namely the possibility to detect topically coherent segments. In particular, as one can note, the NP Rear Admiral Benjamin Sands is segmented in two parts, the first one, Rear Admiral, corresponding to the title, and the second one, Benjamin Sands, to the first and family names of the person. The
Topic1 march, fell, rose, january, rise, year, fall, february, pct, week fell, mln, year, january, dlrs, rise, rose, pct, billion, february Topic2 currency, bank, pct, cut, rate, day, prime, exchange, interest, national billion, prime, day, rate, dlrs, pct, reserve, federal, fed, bank Topic3 term, agreement, acquire, buy, sell, unit, acquisition, corp, company, sale term, dlrs, buy, company, sell, unit, corp, acquisition, sale, mln Topic4 approved, american, common, split, merger, company, board, stock, share, shareholder acquire, mln, company, common, stock, shareholder, share, corp, merger, dlrs Topic5 tokyo, life, intent, letter, buy, insurance, yen, japan, dealer, dollar central, european, japan, yen, ec, dollar, bank, rate, dealer, market
Figure 4: Top-10 words of segLDAcop (left) vs LDA (right) for the Reuters (5 out of 20 topics).
Rear Admiral Benjamin Sands was an officer in the United States
Figure 5: Topic assignments with segmentation boundaries using segLDAcop. Colors are topics (examples from Wiki0 including stopwords with 20 topics).
data-driven approach we have adopted here can discover such fine grained differences, something the approaches based on fixed segmentations (either based on sentences or NPs), are less likely to achieve.
5 Conclusion
In this paper, we have introduced an LDA-based model that generates topically coherent segments within documents by jointly segmenting documents and assigning topics to their words. The coherence between topics is ensured through Frank’s copula, that binds the topics associated to the words of a segment. In addition, this model relies on both document and segment specific topic distributions so as to capture fine grained differences in topic assignments. We have shown that this model naturally encompasses other state-of-the-art LDA-based models proposed to accomplish the same task, and that it outperforms these models over six publicly available collections in terms of perplexity, Normalized Pointwise Mutual Information (NPMI), a measure used to assess the coherence of topics with documents, and the Micro F1-measure in a text classification context. Our results confirm the importance of a flexible segmentation as well as a binding mechanism to produce topically coherent segments.
In the future, we plan on relying on other inference approaches, based for example on variational Bayes known to yield better estimates for perplexity (Asuncion et al., 2009); it is however not certain that the gain in perplexity one can expect from the use of variational Bayes approaches will necessarily result in a gain in, say, topic coherence. Indeed, the impact of the inference approach on the different usages of latent topic models for text collections remains to be better understood.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Efficient segmentation
Let us recall the property presented before:
Proposition A.1. Let lsi be the random variable associated to the length of the segment starting at position i in a sentence of length M (positions go from 1 to M and lsi takes value in {1, · · · , L}). Then P (lsi = l) := g(M+1−i−l);L) g(M+1−i;L) defines a probability distribution over lsi .
Furthermore, the following process is equivalent to choosing sentence segmentations uniformly from the set of possible segmentations. From pos. 1, repeat till end of sentence: (a) Generate segment length acc. to P; (b) Add segment to current segmentation; (c) Move to position after the segment.
Proof Any segmentation of the sentence of length M starts with either a segment of length 1, a segment of length 2, · · · , or a segment of length L. Thus, g(M ;L) can be defined through the following recurrence relation:
g(M ;L) = L∑ l=1 g(M − l;L) (1)
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
together with the initial values g(1;L), g(2;L), · · · , g(L;L), which can be computed offline (for example, for L = 3, one has: g(1; 3) = 1, g(2; 3) = 2, g(3; 3) = 4). Note that g(1;L) = 1 for all L.
Thus:
L∑ l=1 P (lsi = l) = L∑ l=1 g(M + 1− i− l);L) g(M + 1− i;L) = 1
due to the recurrence relation on g. This proves the first part of the proposition.
Using the process described above where segments are generated one after another according to P , for a segmentation S, comprising |S| segments, let us denote by l1, l2, · · · , l|S| the lengths of each segment and by i1, i2, · · · , i|S| the starting positions of each segment (with i1 = 1). One has, as segments are independent of each other:
P (S) = |S|∏ j=1 P (lsij = lj) = |S|∏ j=1 g(M + 1− (ij + lj);L) g(M + 1− ij ;L)
= g(M − l1;L) g(M ;L) g(M − l1 − l2;L) g(M − l1;L) · · · = 1 g(M ;L)
as g(1;L) = 1. This concludes the proof of the proposition. 2
Furthermore, as one can note from Eq. 1, the various elements needed to compute P (lsi = l) can be efficiently computed, the time complexity being equal to O(M). In addition, as the number of different sentence lengths is limited, one can store the values of g to reuse them during the segmentation phase.

### Strengths: - Well-written, well-organized - Incorporate topical segmentation to copula LDA to enable the joint learning of segmentation and latent models - Experimental setting is well-designed and show the superiority of the proposed method from several different indicators and datasets  ### Weaknesses: - No comparison with  novel  segmentation methods  ### General Discussion: This paper presents segLDAcop, a joint latent model for topics and segments. This model is based on the copula LDA and incorporates the topical segmentation to the copula LDA. The authors conduct comprehensive experiments by using several different datasets and evaluation metrics to show the superiority of their model.  This paper is well-written and well-organized. The proposed model is a reasonable extension of the copula LDA to enable the joint inference of segmentations and topics. Experimental setting is carefully designed and the superiority of the proposed model is fairly validated. One concern is that the authors only use the simple NP segmentation and single word segmentation as segments of the previous method. As noted in the paper, there are many work to smartly generate segments before running LDA though it is largely affected by the bias of statistical or linguistic tools used. The comparison with more novel (state-of-the-art) segments would be preferable to precisely show the validity of the proposed method.  ### Minor comment - In line 105,  latent radom topics  to  latent random topics


Enriching Complex Networks with Word Embeddings for Detecting Mild Cognitive Impairment from Speech Transcripts
Mild Cognitive Impairment (MCI) is a mental disorder difficult to diagnose. Although language impairment is an important marker it is frequently undervalued in cognitive assessments. Linguistic features, mainly from parsers, have been used to detect MCI. However, MCI disfluencies produce agrammatical speech impacting in parsing results; manually correcting transcripts of patient’s speech is not a solution to large scale assessments. In this paper, we use complex network features to automatically identifying MCI in transcripts, using several classification algorithms, as it a lightweight and language independent representation. We modeled transcripts into complex networks and enriched them with word embeddings to better represent short texts produced in assessments. We evaluate our model in three datasets: one from the DementiaBank; Cinderella and Arizona-Battery in Portuguese were produced by assessments applied at University of São Paulo Medical School. The results show that complex networks are suitable to detect MCI and outperform linguistic features in all datasets. We also combined the classifiers through ensemble and multi-view learning. The ensemble of algorithms and multi-view method achieved 10% of improvement in Cinderella dataset compared to the best individual classifier. For the Arizona dataset, the multi-view achieved the highest accuracy (80%), a 4.71% of improvement.
Mild Cognitive Impairment (MCI) is a mental disorder difficult to diagnose. Although language impairment is an important marker it is frequently undervalued in cognitive assessments. Linguistic features, mainly from parsers, have been used to detect MCI. However, MCI disfluencies produce agrammatical speech impacting in parsing results; manually correcting transcripts of patient’s speech is not a solution to large scale assessments. In this paper, we use complex network features to automatically identifying MCI in transcripts, using several classification algorithms, as it a lightweight and language independent representation. We modeled transcripts into complex networks and enriched them with word embeddings to better represent short texts produced in assessments. We evaluate our model in three datasets: one from the DementiaBank; Cinderella and Arizona-Battery in Portuguese were produced by assessments applied at University of São Paulo Medical School. The results show that complex networks are suitable to detect MCI and outperform linguistic features in all datasets. We also combined the classifiers through ensemble and multi-view learning. The ensemble of algorithms and multi-view method achieved 10% of improvement in Cinderella dataset compared to the best individual classifier. For the Arizona dataset, the multi-view achieved the highest accuracy (80%), a 4.71% of improvement.
1 Introduction
Mild Cognitive Impairment (MCI) can affect one or multiple cognitive domains (e.g. memory, language, visuospatial skills and the executive function), and may represent a pre-clinical stage of Alzheimer’s disease (AD). The impairment that affects memory, referred to as amnestic MCI, is the most frequent, with the highest conversion rate for AD, at 15% a year versus 1 to 2% for the general population. Since dementias are chronic and progressive diseases, their early diagnosis ensures a greater chance of success to engage patients in non-pharmacological treatment strategies such as cognitive training, physical activity and socialization (Teixeira et al., 2012).
Language is one of the most efficient information sources for assessing cognitive functions. Changes in language usage are frequent in patients with dementia and are normally first recognized by the patients themselves or their family members. Therefore, the automatic analysis of discourse production is promising for diagnosing MCI at early stages, which may address potentially reversible factors (Muangpaisan et al., 2012). Proposals to detect language-related impairment in dementias include machine learning (Jarrold et al., 2010; Roark et al., 2011; Fraser et al., 2014, 2015), magnetic resonance imaging (Dyrba et al., 2015), and data screening tests added to demographic information (Weakley et al., 2015). The discourse production (mainly narratives) is attractive because it allows for the analysis of linguistic microstructures, including phonetic-phonological, morphosyntactic and semantic-lexical components, as well as semanticpragmatic macrostructures.
Automated discourse analysis based on Natural Language Processing (NLP) resources and tools to diagnose dementias via machine learning meth-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
ods have been used for the English language (Lehr et al., 2012; Jarrold et al., 2014; Orimaye et al., 2014; Fraser et al., 2015; Davy et al., 2016) and for Brazilian Portuguese (Aluı́sio et al., 2016). A variety of features are required for this analysis, including Part-of-Speech (PoS), syntactic complexity, lexical diversity and acoustics features. Producing robust tools to extract these features is extremely difficult because speech transcripts used in neuropsychological evaluations contain disfluencies (repetitions, revisions, paraphasias) and patient’s comments about the task being evaluated. Another problem in using linguistic knowledge is the high dependence on manually created resources, such as hand-crafted linguistic rules and/or annotated corpora. Even when traditional statistical techniques (Bag of Words (BoW) or ngrams) are applied they also have problems to deal with disfluencies.
An approach applied successfully to several areas of NLP (Mihalcea and Radev, 2011), which may suffer less from the problems mentioned above, relies on the use of complex networks and graph theory. The word adjacency network model (i Cancho and Solé, 2001; Roxas and Tapang, 2010; Amancio et al., 2012a; Amancio, 2015b) has provided good results in text classification (de Arruda et al., 2016) and related tasks, namely author detection (Amancio, 2015a), identification of literary movements (Amancio et al., 2012c), authenticity verification (Amancio et al., 2013) and sense discrimination (Amancio et al., 2012b).
In this paper, we show that speech transcripts (narratives or descriptions) can be modeled into complex networks that are enriched with word embeddings. We modeled narratives and short descriptions into complex networks and enriched them with word embeddings to better represent short texts produced in these assessments. When applied to a machine learning classifier, the complex network features were able to distinguish between control participants and mild cognitive impairment participants. Discrimination of the two classes could be improved by combining complex networks with linguistic and traditional statistical features; we also identified the best scenario for application of network features extracted from transcripts of neuropsychological assessments.
2 Related Work
Detection of memory impairment has been based on linguistic, acoustic, and demographic features, in addition to scores of neuropsychological tests. Linguistic and acoustic features were used to automatically detect aphasia (Fraser et al., 2014); and AD (Fraser et al., 2015) or dementia (Orimaye et al., 2014) in public corpora of DementiaBank. Other studies distinguished differents types of dementia (Garrard et al., 2014; Jarrold et al., 2014), in which speech samples were elicited using the Picnic picture of the Western Aphasia Battery. Davy et al. (2016) also used the Picnic scene for detecting MCI, where the subjects were asked to write (by hand) a detailed description of the scene.
As for automatic detection of MCI in narrative speech, Roark et al. (2011) extracted speech features and linguistic complexity measures of speech samples obtained with the Wechsler Logical Memory (WLM) subtest (Wechsler et al., 1997), and Lehr et al. (2012) fully automatized the WLM subtest. Some studies used short animated films to evaluate immediate and delayed recall in MCI patients which were asked to talk about the first film shown, then about their previous day, and finally about another film shown last. Tóth et al. (2015) adopted automatic speech recognition (ASR) to extract a phonetic level segmentation, which they used to calculate acoustic features. Vincze et al. (2016) used speech, morphological, semantic, and demographic features collected from their speech transcripts to automatically identify patients suffering from MCI.
For the Portuguese language, machine learning algorithms were used to identify subjects with AD and MCI. (Aluı́sio et al., 2016) used a variety of linguistic metrics, such as syntactic complexity, idea density (da Cunha et al., 2015), and text cohesion through latent semantics. PLN tools with high precision are needed to compute these metrics, which is a problem for Portuguese since there is no robust dependency or constituency parser. Therefore, the transcriptions had to be manually revised; they were segmented in sentences, following a semantic-structural criterion and capitalization was added afterwards. The authors also removed disfluencies and inserted omitted subjects when they were hidden, in order to reduce parsing errors. This process is obviously expensive, which has motivated us to use complex networks in the
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
present study to model transcriptions and avoid a manual preprocessing step.
3 Modeling and Characterizing Texts as Complex Networks
The theory and concepts of complex networks have been used in several NLP tasks (Mihalcea and Radev, 2011; Cong and Liu, 2014), such as text classification (de Arruda et al., 2016), summarization (Antiqueira et al., 2009; Amancio et al., 2012a) and word sense disambiguation (Silva and Amancio, 2012). In this study, we used the word co-occurrence model (also called the word adjacency model) with a small modification because most of the syntactical relations occur among neighbouring words (i Cancho et al., 2004). Each distinct word becomes a node and words that are adjacent in the text are connected by an edge. Mathematically, a network is defined as an undirected graph G = {V,E}, formed by a set V = {v1, v2, ..., vn} of nodes (words) and a set E = {e1, e2, ..., em} of edges (co-occurrence) that are represented by an adjacency matrix A, whose elementsAij are equal to 1 whenever there is an edge connecting nodes (words) i and j, and equal to 0 otherwise.
Before modeling texts into complex networks, it is often necessary to do some preprocessing in the raw text. Preprocessing starts with tokenization where each document/text is divided into tokens (meaningful elements, e.g.: words and punctuation marks) and then stopwords and punctuation marks are removed, since they have little semantic meaning. One last step we decided to eliminate from the preprocessing pipeline is lemmatization, which transforms each word into its canonical form. This decision was made based on two factors: first, recent work has shown that lemmatization has little or no influence when network modeling is adopted in related tasks (Machicao et al., 2016). Second, the lemmatization process requires part-of-speech (POS) tagging that may introduce undesirable noises/errors in the text, since the texts in our work are transcriptions containing disfluencies.
Another problem with transcriptions in our work is their size. As demonstrated by Amancio (2015c), the classification of small texts using networks can be impaired, since short texts have almost linear networks, and the topological measures of these networks have little or no informa-
tion relevant to classification. To solve this problem, we adapted the approach of modeling networks with word embeddings proposed by Perozzi et al. (2014) to enrich the networks with semantic information. In this process of word embedding, language networks are generated from continuous word representations, where each word is represented by a dense, real-valued vector obtained by trainning neural networks in the language model task (or variations, such as context prediction) (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013a,b). This structure is known to capture syntatic and semantic information (Mikolov et al., 2013a,b). Perozzi et al. (2014), in particular, takes advantage of word embeddings to build networks where each word is a vertice and edges are defined by similarity between words established by the proximity of the word vectors.
Following this methodology, in our model we added new edges to the co-occurrence networks considering similarities between words, that is, for all pairs of words in the text that were unconnected, an edge was created if their vectors (from word embedding) had a cosine similarity higher than a given threshold. Figure 1 shows an example of a co-occurrence network enriched by similarity links (the dotted edges). The gain in information by enriching a co-occurrence network with seman-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
(a) (b)
Figure 2: Example of (a) co-occurrence network created for a transcript of the Cookie Theft dataset (see Supplementary Information, Section A1) and (b) the same co-occurrence network enriched with semantic information. Note that (b) is a more informative network than (a), since (a) is practically a linear network.
tic information is readily apparent in Figure 2.
4 Datasets, Features and Methods
4.1 Datasets
The datasets consisted of: (i) transcribed samples, manually segmented of the DementiaBank1 and Cinderella story and (ii) transcribed samples of ABCD battery which were automatically segmented, since we are working towards a fully automated system to detect MCI in transcripts. The first dataset is composed of short English descriptions, while the second contains longer Brazilian Portuguese narratives. A third dataset had very short narratives, also in Portuguese. Below, we describe in more detail the datasets, participants, and the task in which they were used.
4.1.1 The Cookie Theft Picture Description Dataset
The clinical dataset used for English was created during a longitudinal study conducted by the University of Pittsburgh School of Medicine on Alzheimer’s and related dementia, funded by the National Institute of Aging. To be eligible for inclusion in the study, all participants were required to be above 44 years of age, have at least 7 years of education, no history of nervous system disorders or be taking neuroleptic medication, have an initial Mini-Mental State Exam (MMSE) score of 10 or greater, and be able to give informed consent. The dataset contains transcripts of verbal interviews with AD and related Dementia patients,
1https://talkbank.org/DementiaBank/
including those with MCI (for more detail, see (Becker et al., 1994)).
We used 43 transcripts of patients with MCI, sampled from 326 people diagnosed with probable AD; 21 with possible AD. Table 1 shows the demographic information for the two diagnostic groups.
For this dataset, interviews were conducted in English and narrative speech was elicited using the Cookie Theft picture (Goodglass et al., 2001) (Figure 3 from Goodglass et al. (2001) in Section A.1). During the interview, patients were given the picture and were told to discuss everything they could see happening in the picture. The patients’ verbal utterances were recorded and then transcribed into the CHAT (Codes for the Human Analysis of Transcripts) transcription format (MacWhinney, 2000).
We extracted the word-level transcript patient sentences from the CHAT files and discarded the annotation, as our goal was to create a fully automated system that does not require the input of a human annotator. We automatically removed filled pauses such as uh, um , er , and ah (e.g. uh it seems to be summer out), short false starts (e.g. just t the ones ), and repetition (e.g. mother’s finished certain of the the dishes ), like (Fraser et al., 2015). The Control group has an average of 9.58 sentences per narrative, with each sentence having an average of 9.18 words; as for the MCI group, it has an average of 10.97 sentences per narrative, with 10.33 words per sentence in average.
4.1.2 The Cinderella Narrative Dataset
The dataset examined in this study included 20 subjects with MCI and 20 subjects without MCI for control, as diagnosed at the Medical School of the University of São Paulo (FMUSP). Table 2 shows the demographic information of the two diagnostic groups, which were also used in (Aluı́sio et al., 2016).
The criteria used to diagnose MCI came from (Petersen, 2004). Diagnostics were done by a multidisciplinary team with psychiatrists, geria-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Demographic Control MCI Avg. Age (SD) 74.8 (11.3) 73.3 (5.9) Avg. Years of 11.4 (2.6) 10.8 (4.5)Education (SD)
No. of Male/Female 27/16 29/14
Table 2: Demographic information of participants in the Cinderella dataset. The Avg. Education is given in years.
tricians, neurologists, neuropsychologists, speech pathologists, and occupational therapists, by a criterion of consensus. Inclusion criteria for the control group were elderly people with no cognitive deficits and preservation of functional capacity in everyday life. The exclusion criteria for the normal group were: poorly controlled clinical diseases, sensitive deficits that are not being compensated for and interfere with the performance in tests, other neurological or psychiatric diagnoses that are associated with dementia or cognitive deficits and use of medications in doses that affect cognition.
Speech narrative samples were elicited by having participants tell the Cinderella story; participants were given as much time as they needed to examine a picture book illustrating the story (Figure 4 in Section A). When each participant had finished looking at the pictures, the examiner asked the subject to tell the story in their own words, as in (Saffran et al., 1989). The time was recorded, but there was no limit imposed to the narrative length. If the participant had difficulty initiating, continuing speech, or presenting a long pause, an evaluator used a stimulus question “What happens next ?”, seeking to encourage the participant to continue his/her narrative. When the subject was unable to proceed with the narrative, the examiner asked if he/she had finalized the story and had something to add. Each speech sample was recorded and then manually transcribed at the word level following the NURC/SP N. 338 EF and 331 D2 transcription norms.
Other tests were applied after the narrative in the following sequence: phonemic verbal fluency test, action verbal fluency, Camel and Cactus test (Bozeat et al., 2000), and Boston Naming test (Kaplan et al., 2001), in order to diagnose the groups.
Since our goal was to create a fully automated system that did not require the input of a human annotator, we segmented sentences to simulate a high-quality ASR transcript with sentence seg-
mentation, and we automatically removed the disfluencies following the same guidelines of TalkBank project2. However, other disfluencies (revisions, elaboration, paraphasias and comments about the task) were kept. The Control group has an average of 30.80 sentences per narrative, and each sentence averages 12.17 words; as for the MCI group, it has an average of 29.90 sentences per narrative, and each sentence averages 13.03 words.
We also evaluated a different version of this dataset used in Aluı́sio et al. (2016), where narratives were manually revised to improve parsing results: agramatical sentences and all the patients’ comments not related to the story were removed, and omitted subjects were inserted. In this dataset, the Control group has an average of 45.10 sentences per narrative, and each sentence averages 8.17 words. The MCI group has an average of 31.40 sentences per narrative, with each sentence averaging 10.91 words.
4.1.3 The Arizona Battery for Communication Disorders of Dementia (ABCD) Dataset
The ABCD dataset examined included 23 subjects with MCI and 20 subjects without MCI for control, as diagnosed at the Medical School of the University of São Paulo (FMUSP). MCI subjects produced 46 narratives and the controls 39 ones. We used the automatic sentence segmentation method referred to as DeepBond (Treviso et al., 2017). Table 3 shows the demographic information. The Control group has an average of 5.23 sentences per narrative, with 11 words per sentence in average, and the MCI group has an average of 4.95 sentences per narrative, with an average of 12.04 words per sentence. Interviews were conducted in Portuguese and the subject listened to the examiner read a brief narrative. The subject then retold the narrative to the examiner twice: once immediately upon hearing it and again after a 30-minute delay (Bayles and Tomoeda, 1991). Each speech sample was recorded and then manually transcribed at the word level following the NURC/SP N. 338 EF and 331 D2 transcription norms.
4.2 Features
Features of three distinct natures were used for classifying the transcribed texts: topological met-
2https://talkbank.org/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Demographic Control MCI Avg. Age (SD) 61 (7.5) 72,0 (7.4) Avg. Years of 16 (7.6) 13.3 (4.2)Education (SD)
No. of Male/Female 6/14 16/7
Table 3: Demographic information of participants in the ABCD dataset. The Avg. Education is given in years.
rics of co-occurrence networks, linguistic features and statistics of bag of words representation.
4.2.1 Topological Characterization of Networks
Each transcription was mapped into a cooccurrence network, and then enriched via word embedding using the cosine similarity of words. Since the occurrence of out-of-vocabulary words is common in texts, we used the method proposed by (Bojanowski et al., 2016) to generate word embeddings. This method extends the skipgram model to use character-level information, with each word being represented as a bag of character n-grams. It provides some improvement in comparison with the traditional skip-gram model in syntactic evaluation (Mikolov et al., 2013b) but not for the semantic evaluation.
Once the network has been enriched with semantic information, we characterize its topology using the following ten measurements:
1. PageRank: is a centrality measurement that reflects the relevance of a node based on its connections to other relevant nodes (Brin and Page, 1998);
2. Betweenness: is a centrality measurement that considers a node as relevant if it is highly accessed via shortest paths. The betweenness of a node v is defined as the fraction of shortest paths going through node v;
3. Eccentricity: of a node is calculated by measuring the shortest distance from the node to all other vertices in the graph and taking the maximum;
4. Eigenvector centrality: is a measurement that defines the importance of a node based on its connectivity to high-rank nodes;
5. Average Degree of the Neighbors of a Node: is the average of the degrees of all its direct neighbors;
6. Average Shortest Path Length of a Node: is the average distance between this node and all other nodes of the network;
7. Degree: is the number of edges connected to the node;
8. Assortativity Degree: or degree correlation measures the tendency of nodes to connect to other nodes that have similar degree;
9. Diameter: is defined as the maximum shortest path;
10. Clustering Coefficient: measures the probability that two neighbors of a node are connected.
Most of the measurements described above are local measurements, i.e. each node i possesses a value Xi, so we calculate the average µ(X), standard deviation σ(X) and skewness γ(X) for each measurement (Amancio, 2015b).
4.2.2 Linguistic Features
Linguistic features for classification of neuropsychological assessments have been utilized in several works (Roark et al., 2011; Jarrold et al., 2014; Fraser et al., 2014; Orimaye et al., 2014; Fraser et al., 2015; Vincze et al., 2016; Davy et al., 2016). We used the Coh-Metrix3(Graesser et al., 2004) tool to extract features of English transcripts, resulting in 106 features. The metrics are divided into eleven categories: Descriptive, Text Easability Principal Component, Referential Cohesion, Latent Semantic Analysis (LSA), Lexical Diversity, Connectives, Situation Model, Syntactic Complexity, Syntactic Pattern Density, Word Information, Readability (Flesch Reading Ease, Flesch-Kincaid Grade Level, Coh-Metrix L2 Readability).
For Portuguese, Coh-Metrix-Dementia (Aluı́sio et al., 2016) was used. The metrics affected by constituency and dependency parsing were not used because they are not robust to deal with disfluencies. Metrics based on manual annotation (such as proportion short pauses, mean pause duration, mean number of empty words, and others) were also discarded. The metrics of Coh-MetrixDementia are divided into twelve categories: Ambiguity, Anaphoras, Basic Counts, Connectives,
3cohmetrix.com
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Co-reference Measures, Content Word Frequencies, Hypernyms, Logic Operators, Latent Semantic Analysis, Semantic Density, Syntactical Complexity, and Tokens. The metrics used are shown in detail in Section A.2. In total, 58 metrics were used, from the 73 available in the website4.
4.2.3 Bag of Words
The representation of text collections under the Bag-of-Words (BoW) assumption (i.e., with no information relating to word order) has been a robust solution for text classification by capturing word content- and frequency-specific differences that are relevant to the categories under investigation (Joachims, 1998; Drucker et al., 1999). In this methodology, transcripts are represented by a table in which the columns represent the terms (or existing words) in the transcripts and the values represent frequency of a term in a document, such as binary weights, term frequency (tf) or term frequency – inverse document frequency (tf–idf) (Salton, 1989). In this work term frequency was used.
4.3 Classification Algorithms
In order to quantify the ability of the topological characterization of networks, linguistic metrics and BoW features to distinguish subjects with MCI from those without, we employed four machine learning algorithms to induce classifiers from a training set. The techniques are Gaussian Naive Bayes (GaussianNB), k-Nearest Neighbor (K-NN), Support Vector Machine (SVM), linear and radial bases functions (RBF), and Random Forest (RF). We also combine these classifiers through ensemble and multi-view learning. In ensemble learning, multiple models/classifiers are generated and combined using, e.g. majority vote or average of class probabilities, to produce a single result (Zhou, 2012). In multi-view learning, multiple classifiers are trained in different feature spaces and thus combined to produce a single result. This approach is an elegant solution rather than combining all features in the same vector or space, primarily because the combination is not a straightforward step, which can lead to noise insertion, since the data have different natures. Secondly, using different classifiers for each feature space allows different weights to be given for each type of feature, weights that can be learned by a
4http://143.107.183.175:22380
regression method to improve the model. This approach has been successful for NLP (Collobert et al., 2011), sentiment classification (Xia et al., 2011) and pedestrian detection in images (Oliveira et al., 2010; Kim et al., 2016).
5 Experiments and Results
All experiments were conducted using the Scikitlearn5 (Pedregosa et al., 2011), with classifiers evaluated on the basis of classification accuracy or the total proportion of narratives which were correctly classified. The evaluation was performed using 5-fold cross validation, and the threshold parameter was optimized with best values being 0.7 in Cookie Theft dataset and 0.4 in both Cinderella and ABCD dataset.
We trained the model proposed by Bojanowski et al. (2016) with default parametrs (100 dimensional embeddings, 5 size of the context window, 5 number of epochs). The accuracy in classification is given in Tables 4, 5, 6, and 7. CN, CNE, LM, and BoW denote, respectively, complex networks, complex network enriched with embeddings, linguistic metrics and Bag-of-Words, and CNE-LM, CNE-BoW, LM-BoW and CNE-LM-BoW refer to combinations of the feature spaces (multiview learning), using the majority vote. Cells with the – sign mean that it is not possible to apply majority voting. The last line represents the use of an ensemble of machine learning algorithms, in which the combination used was the majority voting in both ensemble and multiview learning.
The results for the three datasets show that characterizing transcriptions into complexity networks is competitive with other traditional methods, such as with the use of linguistic metrics. In fact, among the three types of features, using enriched networks (CNE) provided the highest accuracies, and in general CNE is better than using only complex networks. SVM gives better accuracy in most cases compared to other machine learning algorithms. As for ensemble and multi-view learnings there are some good results. For the Cookie Theft dataset, multi-view learning achieved the highest accuracy (65% of accuracy for narrative texts, a 2.3% of improvement compared to the best individual classifier). The ensemble of algorithms and multi-view CNE-LM achieved the highest accuracy in Cinderella dataset (75% of accuracy for narrative texts, a 10% of improvement compared
5http://scikit-learn.org
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Classifier CN CNE LM BoW CNE-LM CNE-BoW LM-BoW CNE-LM-BoW
SVM-Linear 0.5250 0.5542 0.5694 0.5917 – – – 0.6056 SVM-RBF 0.5694 0.6264 0.5806 0.6014 – – – 0.6500 KNN 0.5903 0.6167 0.4611 0.5792 – – – 0.5903 RF 0.5222 0.4778 0.4569 0.4889 – – – 0.5000 G-NB 0.5125 0.4861 0.5694 0.5583 – – – 0.5000 Ensemble 0.5625 0.6042 0.5458 0.5806 0.5000 0.5625 0.5000 0.6500
Table 4: Classification accuracy achieved on Cookie Theft dataset.
Classifier CN CNE LM BoW CNE-LM CNE-BoW LM-BoW CNE-LM-BoW
SVM-Linear 0.5250 0.6000 0.5250 0.5000 – – – 0.5250 SVM-RBF 0.5750 0.6500 0.4750 0.3750 – – – 0.5000 KNN 0.4750 0.5000 0.4750 0.3750 – – – 0.3750 RF 0.5500 0.5750 0.4750 0.4500 – – – 0.5250 G-NB 0.4750 0.5250 0.4750 0.5500 – – – 0.5250 Ensemble 0.5250 0.6000 0.5000 0.3750 0.7500 0.5000 0.3750 0.4750
Table 5: Classification accuracy achieved on Cinderella dataset.
Classifier CNE LM BoW CNE-BoW
SVM-Linear 0.6500 0.6250 0.5250 – SVM-RBF 0.6750 0.7250 0.5500 – KNN 0.4750 0.5500 0.5000 – RF 0.4750 0.5750 0.4500 – G-NB 0.6500 0.6000 0.4500 – Ensemble 0.6000 0.6750 0.4500 0.7500
Table 6: Classification accuracy achieved on Cinderella dataset manually processed to revise agrammatical sentences.
Classifier CNE LM BoW CNE-LM-BoW
SVM-Linear 0.7173 0.5904 0.7547 0.8018 SVM-RBF 0.6436 0.6819 0.5413 0.6583 KNN 0.6208 0.7056 0.5881 0.7171 RF 0.4921 0.6813 0.6265 0.6252 G-NB 0.5722 0.5744 0.6257 0.6450 Ensemble 0.6783 0.7056 0.6964 0.7636
Table 7: Classification accuracy achieved on ABCD dataset
to the best individual classifier). For the ABCD dataset, multi-view CNE-LM-BoW achieved the highest accuracy (80% of accuracy for narrative texts, a 4.71% of improvement compared to the best individual classifier). Somewhat surprising were the results of SVM with linear kernel in BoW feature space (75% of accuracy).
6 Conclusions and Future Work
In this study, we employed metrics of topological properties of CN in a machine learning classification approach to distinguish between healthy
controls and patients with MCI. To the best of our knowledge, these metrics have never been used before to detect MCI in speech transcripts; CN were enriched with word embeddings to better represent short texts produced in neuropsychological assessments. We have shown that the topological properties of CN outperfom traditionally linguistic metrics, in individual classifiers results. Linguistic features depend on grammatical texts to present good results, as can be seen in the results of Cinderella dataset manually processed (Table 6). Furthermore, we found that combining machine and multi-view learning can improve accuracy. The accuracies found here are comparable to the values reported by other authors, ranging from 60% to 85% (Prud’hommeaux and Roark, 2011; Lehr et al., 2012; Tóth et al., 2015; Vincze et al., 2016), which means that it is not easy to distinguish between healthy subjects and those with cognitive impairments. The comparison with our results is not straightforward, though, because the databases used in the studies are different. There is a clear need of publicly available datasets to compare different methods, which would allow for optimizing detection of MCI in elderly people.
As future work, we intend to explore other methods to enrich CN, such as Recurrent Language Model language, and other metrics to characterize an adjacency network. The pursue of these strategies is relevant because language is one of the most efficient information sources for assessing cognitive functions, commonly used in neuropsychological assessments.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplementary Material
A.1 Example of transcription
Below follows an example of a transcript of the Cookie Theft dataset.
You just want me to start talking ? Well the little girl is asking her brother we ’ll say for a cookie . Now he ’s getting the cookie one for him and one for her . He unbalances the step the little stool and he ’s about to fall . And the lid ’s off the cookie jar . And the mother is drying the dishes abstractly so she ’s left the water running in the sink and it
is spilling onto the floor . And there are two there ’s look like two cups and a plate on the sink and board . And that boy ’s wearing shorts and the little girl is in a short skirt . And the mother has an apron on . And she ’s standing at the window . The window ’s opened . It must be summer or spring . And the curtains are pulled back . And they have a nice walk around their house . And there ’s this nice shrubbery it appears and grass . And there ’s a big picture window in the background that has the drapes pulled off . There ’s a not pulled off but pulled aside . And there ’s a tree in the background . And the house with the kitchen has a lot of cupboard space under the sink board and under the cabinet from which the cookie you know cookies are being removed .
A.2 Cohmetrix-Dementia metrics
1. Ambiguity: verb ambiguity, noun ambiguity, adjective ambiguity, adverb ambiguity;
2. Anaphoras: adjacent anaphoric references, anaphoric references;
3. Basic Counts: Flesch index, number of word, number of sentences, number of paragraphs, words per sentence, sentences per paragraph, syllables per content word, verb incidence, noun incidence, adjective incidence, adverb incidence, pronoun incidence, content word incidence, function word incidence;
4. Connectives: connectives incidence, additive positive connectives incidence, additive negative connectives incidence, temporal positive connectives incidence, temporal negative connectives incidence, casual positive connectives incidence, casual negative connectives incidence, logical positive connectives incidence, logical negative connectives incidence;
5. Co-reference Measures: adjacent argument overlap, argument overlap, adjacent stem overlap, stem overlap, adjacent content word overlap;
6. Content Word Frequencies: Content words frequency, minimum among content words frequency;
7. Hypernyms: Mean hypernyms per verb;
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
8. Logic Operators: Logic operators incidence, and incidence, or incidence, if incidence, negation incidence;
9. Latent Semantic Analysis (LSA): Average and standard deviation similarity between pairs of adjacent sentences in the text, Average and standard deviation similarity between all sentence pairs in the text, Average and standard deviation similarity between pairs of adjacent paragraphs in the text, Givenness average and standard deviation of each sentence in the text;
10. Semantic Density: content density;
11. Syntactical Complexity: only cross entropy;
12. Tokens: personal pronouns incidence, typetoken ratio, Brunet index, Honoré Statistics.

The paper describes a novel application of mostly existing representations, features sets, and methods: namely, detecting Mild Cognitive Impairment (MCI)  in speech narratives. The nature of the problem, datasets, and domain are thoroughly described. While missing some detail, the proposed solution and experiments sound reasonable. Overall, I found the study interesting and informative.  In terms of drawbacks, the paper needs some considerable editing to improve readability. Details on some key concepts appear to be missing. For example,  details on the multi-view learning used are omitted; the set of  linguistic features d needs to be clarified; it is not entirely clear what datasets were used to generate the word embeddings (presumably the 3 datasets described in the paper, which appear to be too small for that purpose u2026). It is also not clear why disfluencies (filled pauses, false starts, repetitions, etc.) were removed from the dataset. One might suggest that they are important features in the context of MCI. It is also not clear why the most popular tf-idf weighting scheme was not used for the BoW classifications. In addition, tests for significance are not provided to substantiate the conclusions from the experiments. Lastly, the related work is described a bit superficially.   Detailed comments are provided below:  Abstract: The abstract needs to be shortened. See detailed notes below.  Lines 22,23 need rephrasing.             However, MCI disfluencies produce agrammatical speech impacting in parsing results d  u2192 impacting the parsing results?  Lines 24,25: You mean correct grammatical errors in transcripts manually? It is not clear why this should be performed, doesn t the fact that grammatical errors are present indicate MCI?  u2026 Only after reading the Introduction and Related Work sections it becomes clear what you mean. Perhaps include some examples of disfluencies.  Lines 29,30 need rephrasing:  as it a lightweight and language  independent representation d  Lines 34-38 need rephrasing: it is not immediately clear which exactly are the 3 datasets. Maybe:  the other two: Cinderella and  u2026                Line 70:  15% a year d  u2192 Not sure what exactly  per year d means u2026  Line 73 needs rephrasing.  Lines 115 - 117: It is not obvious why BoW will also have problems with disfluencies, some explanation will be helpful.  Lines 147 - 149: What do you mean by  the best scenario d?  Line 157:  in public corpora of Dementia Bank d  u2192 a link or citation to Dementia Bank will be helpful.   Line 162: A link or citation describing the  Picnic picture of the Western Aphasia Battery d will be helpful.  Line 170: An explanation as to what the WML subtest is will be helpful.  Line 172 is missing citations.  Lines 166 - 182: This appears to be the core of the related work and it is described a bit superficially. For example, it will be helpful to know precisely what methods were used to achieve these tasks and how they compare to this study.  Line 185: Please refer to the conference citation guidelines. I believe they are something along these lines:  Aluisio et al. (2016)  used u2026 d  Line 188: The definition of  PLN d appears to be missing.  Lines 233 - 235 could you some rephrasing. Lemmatization is not necessarily a last step in text pre-processing and normalization, in fact there are also additional common normalization preprocessing steps omitted.   Lines 290-299: Did you create the word embeddings using the MCI datasets or external datasets?  Line 322: consisted of  u2192 consist of  Lines 323: 332 need to be rewritten. ...  manually segmented of the DementiaBank and Cinderella d  u2192  What do you mean by segmented, segmented into sentences? Why weren t all datasets automatically segmented?;  ABCD d is not defined; You itemized the datasets in i) and ii), but subsequently  you refer to 3 dataset, which is a bit confusing. Maybe one could explicitly name the datasets, as opposed to referring to them as  first d,  second d,  third d.  Table 1 Caption: The demographic information is present, but there are no any additional statistics of the dataset, as described.  Lines 375 - 388:  It is not clear why filled pauses, false starts, repetitions, etc. were removed. One might suggest that they are important features in the context of MCI  u2026.  Line 399:  u2026 multidisciplinary team with psychiatrists ...  u2192 consisting of psychiatrists u2026  Lines 340-440: A link or citation describing the transcription norms will be helpful.  Section 4.2.1. It is not clear what dataset was used to generate the word embeddings.   Line 560. The shortest path as defined in feature 6?  Section  4.2.2 Linguistic Features d needs to be significantly expanded for clarity. Also, please check the conference guidelines regarding additional pages ( Supplementary Material d).  Line 620:  In this work term frequency was  u2026 d  u2192  In this work, term frequency was  u2026 d Also, why not tf-idf, as it seems to be the most common weighting scheme?   The sentence on lines 641-645 needs to be rewritten.  Line 662: What do you mean by  the threshold parameter d? The threshold for the word embedding cosine distance?  Line 735 is missing a period.  Section 4.3 Classification Algorithms: Details on exactly what scheme of multi-view learning was used are entirely omitted. Statistical significance of result differences is not provided.