Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective. We show that the resulting approach captures uniquely expressive semantic information, and outperforms alternatives, such as word2vec skip-grams, and Gaussian embeddings, on benchmark datasets such as word similarity and entailment.
1 Introduction
To model language, we must represent words. We can imagine representing every word with a binary one-hot vector corresponding to a dictionary position. But such a representation contains no valuable semantic information: distances between word vectors represent only differences in alphabetic ordering. Modern approaches, by contrast, learn to map words with similar meanings to nearby points in a vector space (Mikolov et al., 2013a), from large datasets such as Wikipedia. These learned word embeddings have become ubiquitous in predictive tasks.
Vilnis and McCallum (2014) recently proposed an alternative view, where words are represented by a whole probability distribution instead of a deterministic point vector. Specifically, they model each word by a Gaussian distribution, and learn its mean and covariance matrix from data. This approach generalizes any deterministic point embedding, which can be fully captured by the mean vector of the Gaussian distribution. Moreover, the full distribution provides much richer information
than point estimates for characterizing words, representing probability mass and uncertainty across a set of semantics. However, since a Gaussian distribution can have only one mode, the learned uncertainty in this representation can be overly diffuse for words with multiple distinct meanings (polysemys), in order for the model to assign some density to any plausible semantics (Vilnis and McCallum, 2014). Moreover, the mean of the Gaussian can be pulled in many opposing directions, leading to a biased distribution that centers its mass mostly around certain meaning while leaving the others not well represented. In this paper, we propose to represent each word with an expressive multimodal distribution, for multiple distinct meanings, entailment, heavy tailed uncertainty, and enhanced interpretability. For example, one mode of the word ‘bank’ could overlap with distributions for words such as ‘finance’ and ‘money’, and another mode could overlap with the distributions for ‘river’ and ‘creek’. It is our contention that such flexibility is critical for both qualitatively learning about the meanings of words, and for optimal performance on many predictive tasks. In particular, we model each word with a mixture of Gaussians (Section 3.1). We learn all the parameters of this mixture model using a maximum margin energy-based ranking objective (Joachims, 2002; Vilnis and McCallum, 2014) (Section 3.3), where the energy function describes the affinity between a pair of words. For analytic tractability with Gaussian mixtures, we use the inner product between probability distributions in a Hilbert space, known as the expected likelihood kernel (Jebara et al., 2004), as our energy function (Section 3.4). Additionally, we propose transformations for numerical stability and initialization, resulting in a robust, straightforward, and scal-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
able learning procedure, capable of training on a corpus with billions of words in days. We show that the model is able to automatically discover multiple meanings for words (Section 4.2), and significantly outperform other alternative methods across several tasks such as word similarity and entailment (Section 4.3, 4.4, 4.6).
2 Related Work
In the past decade, there has been an explosion of interest in word vector representations. word2vec, arguably the most popular word embedding, uses continuous bag of words and skipgrammodels, as well as negative sampling for efficient conditional probability estimation (Mikolov et al., 2013a,b). Other popular approaches use feedforward (Bengio et al., 2003) and recurrent neural network language models (Mikolov et al., 2010, 2011; Collobert and Weston, 2008) to predict missing words in sentences, producing hidden layers that can act as word embeddings that encode semantic information. A different approach to learning word embeddings is through factorization of word co-occurrence matrices such as GloVe embeddings (Pennington et al., 2014). The matrix factorization approach has been shown to have an implicit connection with skip-gram and negative sampling Levy and Goldberg (2014). Bayesian matrix factorization where row and columns are modeled as Gaussians has been explored in Salakhutdinov and Mnih (2008) and provides a different probabilistic perspective of word embeddings.
In exciting recent work, Vilnis and McCallum (2014) propose a Gaussian distribution to model each word. Their approach is significantly more expressive than typical point embeddings, with the ability to represent concepts such as entailment, by having the distribution for one word (e.g. ‘music’) encompass the distributions for sets of related words (‘jazz’ and ‘pop’). However, with a unimodal distribution, their approach cannot capture multiple distinct meanings, much like most deterministic approaches. A small body of work develops multiple deterministic embeddings that can capture polysemys, for example through a cluster centroid of context vectors (Huang et al., 2012), or an adapted skip-gram model with an EM algorithm to learn multiple latent representations per word Tian et al. (2014). Another related work by Nalisnick and Ravi (2015) models embeddings in
infinite-dimensional space where each embedding can gradually represent incremental word sense if complex meanings are observed. Probabilistic word embeddings have only recently begun to be explored, and have so far shown great promise. In this paper, we propose, to the best of our knowledge, the first probabilistic word embedding that can capture multiple meanings. We use a Gaussian mixture model which allows for a highly expressive distributions over words. At the same time, we retain scalability and analytic tractability with an expected likelihood kernel energy function for training. The model and training procedure harmonize to learn descriptive representations of words, with superior performance on several benchmarks.
3 Methodology
In this section, we introduce our Gaussian mixture (GM) model for word representations, and present a training method to learn the parameters of the Gaussian mixture. This method uses an energy-based maximum margin objective, where we wish to maximize the similarity of distributions of nearby words in sentences. We propose an energy function that compliments the GM model by retaining analytic tractability. We also provide critical practical details for numerical stability and initialization.
3.1 Word Representation
We represent each word w in a dictionary as a Gaussian mixture with K components. Specifically, the distribution of w, fw, is given by the density
fw(x⃗) = K∑ i=1 pw,i N [x⃗; µ⃗w,i,Σw,i] (1)
= K∑ i=1 pw,i√ 2π|Σw,i| e− 1 2 (x⃗−µ⃗w,i)⊤Σ−1w,i(x⃗−µ⃗w,i) ,
where ∑K
i=1 pw,i = 1. The mean vectors µ⃗w,i represent the location of the ith component of word w, and are akin to the point embeddings provided by popular approaches like word2vec. pw,i represents the component probability (mixture weight), and Σw,i is the component covariance matrix, containing uncertainty information. Our goal is to learn all of the model parameters µ⃗w,i, pw,i,Σw,i from a corpus of natural sentences to extract semantic information of words. Each
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Gaussian component’s mean vector of word w should be able to represent one of the word’s distinct meanings. For instance, one component of a polysemous word such as rock should represent the meaning related to stone or pebbles, whereas another component should represent the meaning related to music such as jazz, pop. Figure 1 illustrates our word embedding model, and the difference between multimodal and unimodal representations, for words with multiple meanings.
music
rock
jazz
basalt
pop
stone
rock
stone
jazz
pop
music
basalt
music
jazz
rock
basalt
pop
stone
rock
music
rock
jazz
basalt stone
pop
rock
basalt
stone
music
jazz
pop
Figure 1: Top: Each word is represented by a Gaussian mixture, where each mixture component corresponds to a distinct meaning. Each Gaussian component is represented by an ellipsoid, whose center is specified by the mean vector and contour surface specified by the covariance matrix, reflecting subtleties in meaning and uncertainty. On the left, we show examples of Gaussian mixture distributions of words where Gaussian components are randomly initialized. After training, we see on the right that one component of the word rock is closer to stone and basalt, whereas the other component is closer to jazz and pop. We also demonstrate the entailment concept where the distribution of more general word music encapsulates words such as jazz, rock, pop. Bottom: An embedding model where each word is represented by a Gaussian distribution (Vilnis and McCallum, 2014). For words with multiple meanings, such as ‘rock’, the variance of the learned representation becomes unnecessarily large in order to assign some probability to both meanings. Moreover, the mean vector for such words can be pulled between two clusters, centering the mass of the distribution on a region which is far from certain meanings.
3.2 Skip-Gram
The training objective for learning θ = {µ⃗w,i, pw,i,Σw,i} draws inspiration from the continuous skip-gram model (Mikolov et al., 2013a), where word embeddings are trained to maximize the probability of observing a word given another nearby word. This procedure follows the distributional hypothesis that words occurring in natural contexts tend to be semantically related. For instance, the words jazz and music tend to occur near one another more often than jazz and cat; hence, jazz and music are more likely to be related. The learned word representation contains useful semantic information and can be used to perform a variety of NLP tasks such as analogy, word similarity analysis, sentiment classification, or as a preprocessed input for complex system such as statistical machine translation.
3.3 Energy-based Max-Margin Objective
Each sample in the objective consists of two pairs of words, (w, c) and (w, c′). w is sampled from a sentence in a corpus and c is a nearby word within a context window of length ℓ. For instance, a word w = jazz which occurs in the sentence I listen to jazz music has context words I, listen, to , music. c′ is a negative context word (e.g. airplane) obtained from random sampling across the whole document from distribution Pn(c′) (details in supplementary material, Section A.1). The objective is to maximize the energy between words that occur near each other, w and c, and minimize the energy between w and its negative context c′. This approach is similar to negative sampling (Mikolov et al., 2013a,b), which contrasts the dot product between positive context pairs with negative context pairs. The energy function is a measure of similarity between distributions and will be presented and discussed in Section 3.4. We use a max-margin ranking objective (Joachims, 2002), used for Gaussian embeddings in Vilnis and McCallum (2014), which pushes the similarity of a word and its positive context higher than that of its negative context by a margin m:
Lθ(w, c, c ′) = max(0,
m− logEθ(w, c) + logEθ(w, c′))
This objective can be minimized by mini-batch stochastic gradient descent with respect to the pa-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
rameters θ = {µ⃗w,i, pw,i,Σw,i} – the mean vectors, covariance matrices, and mixture weights – of our multimodal embedding in Eq. (1).
3.4 Energy Function
For vector representation of words, a usual choice for similarity measure (energy function) is a dot product between two vectors. Our word representations are distributions instead of point vectors and therefore need a measure that reflects not only the point similarity, but also the uncertainty. In contrast to Vilnis and McCallum (2014), who use a KL energy function, we propose to use the expected likelihood kernel, which is a generalization of an inner product between vectors to an inner product between distributions (Jebara et al., 2004). That is,
E(f, g) = ∫ f(x)g(x) dx = ⟨f, g⟩L2
where ⟨·, ·⟩L2 denotes the inner product in Hilbert space L2. We choose this form of energy since it can be evaluated in a closed form given our choice of probabilistic embedding in Eq. (1).
For Gaussian mixtures f, g representing the words wf , wg, f(x) = ∑K i=1 piN (x; µ⃗f,i,Σf,i)
and g(x) = ∑K i=1 qiN (x; µ⃗g,i,Σg,i), ∑K i=1 pi =
1, and ∑K
i=1 qi = 1, we find (See Section A.1) the log energy is
logEθ(f, g) = log K∑ j=1 K∑ i=1 piqje ξi,j (2)
where
ξi,j ≡ logN (0; µ⃗f,i − µ⃗g,j ,Σf,i +Σg,j)
= −1 2 log det(Σf,i +Σg,j)− D 2 log(2π)
−1 2 (µ⃗f,i − µ⃗g,j)⊤(Σf,i +Σg,j)−1(µ⃗f,i − µ⃗g,j)
(3)
We call the term ξi,j partial (log) energy. Observe that this term captures the similarity between the ith meaning of word wf and the jth meaning of word wg. The total energy in Equation 2 is the sum of possible pairs of partial energies, weighted accordingly by the mixture probabilities pi and qj .
The term−(µ⃗f,i−µ⃗g,j)⊤(Σf,i+Σg,j)−1(µ⃗f,i− µ⃗g,j) in ξi,j explains the difference in mean vectors of semantic pair (wf , i) and (wg, j). If the semantic uncertainty (covariance) for both pairs are low,
this term has more importance relative to other terms due to the inverse covariance scaling. We observe that the loss function Lθ in Section 3.3 attains a low value when Eθ(w, c) is relatively high. This could be achieved by µ⃗f,i being close to µ⃗g,j , which brings the term close to zero. It could also be achieved by high values ofΣf,i andΣg,j , which washes out the importance of the mean vector difference. The term − log det(Σf,i + Σg,j) serves as a regularizer that prevents the covariances from being pushed too high at the expense of learning a good mean embedding. At the beginning of training, ξi,j roughly are on the same scale among all pairs (i, j)’s. During this time, all components learn the signals from the word occurrences equally. As training progresses and the semantic representation of each mixture becomes more clear, there can be one term of ξi,j’s that is predominantly higher than other terms, giving rise to a semantic pair that is most related. The negative KL divergence is another sensible choice of energy function, providing an asymmetric metric between word distributions. However, unlike the expected likelihood kernel, KL divergence does not have a closed form if the two distributions are Gaussian mixtures.
4 Experiments
We have introduced a model for multi-prototype embeddings, which expressively captures word meanings with whole probability distributions. We show that our combination of energy and objective functions, proposed in section 3, enables one to learn interpretable multimodal distributions through unsupervised training, for describing words with multiple distinct meanings. Our model also reduces the unnecessarily large variance of a Gaussian embedding model, and has improved results on word entailment tasks. To learn the parameters of the proposed mixture model, we train on a concatenation of two datasets: UKWAC (2.5 billion tokens) and Wackypedia (1 billion tokens) (Baroni et al., 2009). We discard words that occur fewer than 100 times in the corpus, which results in a vocabulary size of 314, 129 words. Unless stated otherwise, our Gaussian mixture model has two Gaussian components (see Section 4.2 for discussion on higher number of components). Our word sampling scheme is similar to that of word2vec with one negative context word for each positive
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Word Co. Nearest Neighbors
rock 0 basalt:1, boulder:1, boulders:0, stalagmites:0, stalactites:0, rocks:1, sand:0, quartzite:1, bedrock:0 rock 1 rock/:1, ska:0, funk:1, pop-rock:1, punk:1, indie-rock:0, band:0, indie:0, pop:1 bank 0 banks:1, mouth:1, river:1, River:0, confluence:0, waterway:1, downstream:1, upstream:0, dammed:0 bank 1 banks:0, banking:1, banker:0, Banks:1, bankas:1, Citibank:1, Interbank:1, Bankers:0, transactions:1 Apple 0 Strawberry:0, Tomato:1, Raspberry:1, Blackberry:1, Apples:0, Pineapple:1, Grape:1, Lemon:0 Apple 1 Macintosh:1, Mac:1, OS:1, Amiga:0, Compaq:0, Atari:1, PC:1, Windows:0, iMac:0 star 0 stars:0, Quaid:0, starlet:0, Dafoe:0, Stallone:0, Geena:0, Niro:0, Zeta-Jones:1, superstar:0 star 1 stars:1, brightest:0, Milky:0, constellation:1, stellar:0, nebula:1, galactic:1, supernova:1, Ophiuchus:1 cell 0 cellular:0, Nextel:0, 2-line:0, Sprint:0, phones.:1, pda:1, handset:0, handsets:1, pushbuttons:0 cell 1 cytoplasm:0, vesicle:0, cytoplasmic:1, macrophages:0, secreted:1, membrane:0, mitotic:0, endocytosis:1 left 0 After:1, back:0, finally:1, eventually:0, broke:0, joined:1, returned:1, after:1, soon:0 left 1 right-hand:0, hand:0, right:0, left-hand:0, lefthand:0, arrow:0, turn:0, righthand:0, Left:0
Word Nearest Neighbors
rock band, bands, Rock, indie, Stones, breakbeat, punk, electronica, funk bank banks, banking, trader, trading, Bank, capital, Banco, bankers, cash Apple Macintosh, Microsoft, Windows, Macs, Lite, Intel, Desktop, WordPerfect, Mac star stars, stellar, brightest, Stars, Galaxy, Stardust, eclipsing, stars., Star cell cells, DNA, cellular, cytoplasm, membrane, peptide, macrophages, suppressor, vesicles left leaving, turned, back, then, After, after, immediately, broke, end
Table 1: Nearest neighbors based on cosine similarity between the mean vectors of Gaussian components for Gaussian mixture embedding (top) and Gaussian embedding (bottom). The notation w:i denotes the ith mixture component of the word w.
context word. For reproducibility, we describe hyperparameter settings and implementation details in the supplementary material. We performed experiments using 12 cores of Intel i7 3.5GHz CPU. Code will be made publicly available upon publication.
After training, we obtain learned parameters {µ⃗w,i,Σw,i, pi}Ki=1 for each word w. We treat the mean vector µ⃗w,i as the embedding of the ith mixture component with the covariance matrix Σw,i representing its subtlety and uncertainty. We perform qualitative evaluation to show that our embeddings learn meaningful multi-prototype representations and compare to existing models using a quantitative evaluation on word similarity datasets and word entailment.
We name our model as Word to Gaussian Mixture (w2gm) in constrast to Word to Gaussian (w2g) (Vilnis and McCallum, 2014). Unless stated otherwise, w2g refers to our implementation of w2gm model with one mixture component.
4.1 Similarity Measures
Since our word embeddings contain multiple vectors and uncertainty parameters per word, we use the following measures that generalizes similarity scores. These measures pick out the component pair with maximum similarity and therefore determine the meanings that are most relevant.
4.1.1 Expected Likelihood Kernel
The most natural choice for a similarity score is the expected likelihood kernel, an inner product between distributions, which we discussed in section 3.4. This metric incorporates the uncertainty from the covariance matrices in addition to the similarity between the mean vectors.
4.1.2 Maximum Cosine Similarity
This metric measures the maximum similarity of mean vectors among all pairs of mixture components between distributions f and g. That is,
d(f, g) = max i,j=1,...,K ⟨µf,i,µg,j⟩ ||µf,i|| · ||µg,j || , which corresponds to matching the meanings of f and g that are the most similar. For a Gaussian embedding, maximum similarity reduces to the usual cosine similarity.
4.1.3 Minimum Euclidean Distance
Cosine similarity is popular for evaluating embeddings. However, our training objective directly involves the Euclidean distance in Eq. (3), as opposed to dot product of vectors such as in word2vec. Therefore, we also consider the Euclidean metric: d(f, g) = min
i,j=1,...,K [||µf,i−µg,j ||].
4.2 Qualitative Evaluation
In Table 1, we show examples of polysemous words and their nearest neighbors in the embed-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
ding space to demonstrate that our trained embeddings capture multiple word senses. For instance, a word such as rock that could mean either stone or rock music should have each of its meanings represented by a distinct Gaussian component. Our results for a mixture of two Gaussians model confirm this hypothesis, where we observe that the 0th component of rock being related to basalt, boulders and the 1st component being related to indie, funk, hip-hop. Similarly, the word bank has the 0th component representing the river bank and the 1st component representing the financial bank. By contrast, in Table 1 (bottom), see that for Gaussian embeddings with one mixture component, nearest neighbors of polysemous words are predominantly related to a single meaning. For instance, rockmostly has neighbors related to rock music and bank mostly related to the financial bank. The alternative meanings of these polysemous words are not well represented in the embeddings. As a numerical example, the cosine similarity between rock and stone for the Gaussian representation of Vilnis and McCallum (2014) is only 0.029, much lower than the cosine similarity 0.586 between the 0th component of rock and stone in our multimodal representation.
Embedding Visualization We provide an anonymous interactive visualization website (http://35.161.153.223: 6002) for our embeddings (w2gm) that allows real-time queries of words’ nearest neighbors (in the embeddings tab). We use a notation similar to that of Table 1, where a token w: i represents the component i of a word w. For instance, if we search for bank:0, we obtain the nearest neighbors such as river:1, confluence:0, waterway:1, which indicates that the 0th component of ‘bank’ has the meaning ‘river bank’. On the other hand, searching for bank:1 yields nearby words such as banking:1, banker:0, ATM:0, indicating that this component is close to the ‘financial bank’. We also have a visualization of a unimodal Gaussian embedding (w2g) for comparison.
In addition, we include results for our Gaussian mixture model where K = 3, which can learn three distinct meanings. For instance, each of the three components of ‘cell’ is close to (‘keypad’, ‘digits’), (‘incarcerated’, ‘inmate’) or (‘tissue’, ‘antibody’), indicating that the distribution
captures the concept of ‘cellphone’, ‘jail cell’, or ‘biological cell’, respectively. We note that due to the limited number of words with more than 2 meanings, our model with K = 3 does not generally offer performance differences to our model with K = 2; hence, we do not further display K = 3 results for compactness.
4.3 Word Similarity
We evaluate our embeddings on several standard word similarity datasets, namely, SimLex (Hill et al., 2014), WS or WordSim-353, WS-S (similarity), WS-R (relatedness) (Finkelstein et al., 2002), MEN (Bruni et al., 2014), MC (Miller and Charles, 1991), RG (Rubenstein and Goodenough, 1965), YP (Yang and Powers, 2006), MTurk(287,-771) (Radinsky et al., 2011; Halawi et al., 2012), and RW (Luong et al.). Each dataset contains a list of word pairs with a human score of how related or similar the two words are. We calculate the Spearman correlation (Spearman, 1904) between the labels and our scores generated by the embeddings. The Spearman correlation is a rank-based correlation measure that assesses how well the scores describe the true labels. The correlation results are shown in Table 2 using the scores generated from the expected likelihood kernel, maximum cosine similarity, and maximum Euclidean distance. We show the results of our Gaussian mixture model and compare the performance with that of word2vec and the original Gaussian embedding by Vilnis and McCallum (2014). We note that our model of a unimodal Gaussian embedding w2g also outperforms the original model, which differs in model hyperparameters and initialization, for most datasets. Our multi-prototype model w2gm also performs better than skip-gram or Gaussian embedding methods on many datasets, namely, WS, WS-R, MEN, MC, RG, YP, MT-287, RW. The maximum cosine similarity yields the best performance on most datasets; however, the minimum Euclidean distance is a better metric for the datasets MC and RW. These results are consistent for both the single-prototype and the multi-prototype models. We also compare out results on WordSim-353 with the multi-prototype embedding method by Huang et al. (2012), shown in Table 3. We observe that our single-prototype model w2g is competitive compared to models by Huang et al.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Dataset sg* w2g* w2g/mc w2g/el w2g/me w2gm/mc w2gm/el w2gm/me
SL 29.39 32.23 29.35 25.44 25.43 29.31 26.02 27.59 WS 59.89 65.49 71.53 61.51 64.04 73.47 62.85 66.39 WS-S 69.86 76.15 76.70 70.57 72.3 76.73 70.08 73.3 WS-R 53.03 58.96 68.34 54.4 55.43 71.75 57.98 60.13 MEN 70.27 71.31 72.58 67.81 65.53 73.55 68.5 67.7 MC 63.96 70.41 76.48 72.70 80.66 79.08 76.75 80.33 RG 70.01 71 73.30 72.29 72.12 74.51 71.55 73.52 YP 39.34 41.5 41.96 38.38 36.41 45.07 39.18 38.58 MT-287 - - 64.79 57.5 58.31 66.60 57.24 60.61 MT-771 - - 60.86 55.89 54.12 60.82 57.26 56.43 RW - - 28.78 32.34 33.16 28.62 31.64 35.27
Table 2: Spearman correlation for word similarity datasets. The models sg, w2g, w2gm denote word2vec skip-gram, Gaussian embedding, and Gaussian mixture embedding (K=2). The measures mc, el, me denote maximum cosine similarity, expected likelihood kernel, and minimum Euclidean distance. For each of w2g and w2gm, we underline the similarity metric with the best score. For each dataset, we boldface the score with the best performance across all models. We note that the correlation scores for sg*, w2g* are taken from Vilnis and McCallum (2014) and the scores for w2g and w2mg are trained with window size 10.
MODEL ρ× 100 HUANG 64.2 HUANG* 71.3 W2G 70.9 W2GM 73.5
Table 3: Spearman’s correlation (ρ) on WordSim353 datasets for our Word to Mixture of Gaussians (W2MG) embeddings, as well as the multiprototype embedding by Huang et al. (2012). Huang* is trained using data with all stop words removed. W2MG-(S,M) refers to our model with 1 or 2 mixture components.
(2012), even without using a corpus with stop words removed. This could be due to the autocalibration of importance via the covariance learning which decrease the importance of very frequent words such as the, to, a, etc. Moreover, our multi-prototype model substantially outperforms the model of Huang et al. (2012) on the WordSim-353 dataset.
4.4 Word Similarity for Polysemous and Homonymous Words
We use the dataset SCWS introduced by Huang et al. (2012), where word pairs are chosen to have variations in meanings of polysemous and homonymous words.
We compare our method with multiprototype models by Huang (Huang et al., 2012), Tian (Tian et al., 2014), and Chen (Chen et al., 2014).
We note that Chen model uses an external lexical source WordNet that gives it an extra advantage.
We use many metrics to calculate the scores for the Spearman correlation. MaxSim refers to the maximum cosine similarity. AveSim is the average of cosine similarities with respect to the component probabilities.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
MODEL SCORE BEST AP BEST F1 W2G (5) COS 73.1 76.4 W2G (5) KL 73.7 76.0 W2GM (5) COS 73.6 76.3 W2GM (5) KL 75.7 77.9
W2G (10) COS 73.0 76.1 W2G (10) KL 74.2 76.1 W2GM (10) COS 72.9 75.6 W2GM (10) KL 74.7 76.3
Table 5: Entailment results for models w2g and w2gm with window size 5 and 10. The metrics used are the maximum cosine similarity, or the maximum negative KL divergence. We calculate the best average precision as well as the best F1 score. w2gm consistently outperforms w2g for describing entailment.
4.5 Reduction in Variance of Polysemous Words
One motivation for our Gaussian mixture embedding is to model word uncertainty more accurately than Gaussian embeddings, which can have overly large variances for polysemous words. We see that our Gaussian mixture model does indeed reduce the variances of each component for such words. For instance, we observe that the word rock in w2g also has much higher variance per dimension (e−1.8 ≈ 1.65) compared to that of Gaussian components of rock in w2gm (which has variance of roughly e−2.5 ≈ 0.82). We also see, in the next section, that the Gaussian mixture model has desirable qualitative behavior for word entailment.
4.6 Word Entailment
We evaluate our embeddings on the word entailment dataset from Baroni et al. (2012). The lexical entailment between words is denoted by w1 |= w2 which means that all instances of w1 are w2. The entailment dataset contains positive pairs such as aircraft |= vehicle and negative pairs such as aircraft ̸|= insect.
We generate entailment scores of word pairs and find the best threshold, measured by Average Precision (AP) or F1 score, which identifies negative versus positive entailment. We use the maximum cosine similarity and the minimum KL divergence, d(f, g) = min
i,j=1,...,K KL(f ||g), for en-
tailment scores. The minimum KL divergence is similar to the maximum cosine similarity, but also incorporates the embedding uncertainty. And KL divergence is an asymmetric measure, which is
more suitable for certain tasks such as word entailment. For instance, w1 |= w2 does not imply w2 |= w1 such as aircraft |= vehicle and car |= vehicle. The difference betweenKL(w1||w2) versus KL(w2||w1) distinguishes which word distribution encompasses another distribution, a concept demonstrated in Figure 1. Table 5 shows the results of our w2gm model versus the Gaussian embedding model w2g. We observe a trend for both models with window size 5 and 10 that the KL metric yields improvement (both AP and F1) over cosine similarity. In addition, w2gm has a better performance compared to w2g. The multi-prototype model estimates the meaning uncertainty better since it is no longer constrained to be unimodal, leading to better characterizations of entailment. On the other hand, the Gaussian embedding model suffers from large variance problem for polysemous words, which results in less informative word distribution and inferior entailment scores.
5 Discussion
We introduced a model that represents words with expressive multimodal distributions formed from Gaussian mixtures. To learn the properties of each mixture, we proposed an analytic energy function for combination with a maximum margin objective. The resulting embeddings capture different semantics of polysemous words, uncertainty, and entailment, and also perform favorably on word similarity benchmarks. Elsewhere, latent probabilistic representations are proving to be exceptionally valuable, able to capture nuances such as face angles with variational autoencoders (Kingma and Welling, 2013) or subtleties in painting strokes with the InfoGAN (Chen et al., 2016). Similarly, probabilistic word embeddings can capture a range of subtle meanings, and advance the state of the art in predictive tasks. In the future, such representations could also open the doors to a new suite of applications in language modelling, where word distributions are used as inputs to probabilistic LSTMs, or in decision functions where uncertainty matters.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplementary Material
A.1 Derivation of Expected Likelihood Kernel
We derive the form of expected likelihood kernel for Gaussian mixtures. Let f, g be Gaussian mixture distributions representing the words wf , wg . That is, f(x) =
∑K i=1 piN (x;µf,i,Σf,i) and g(x) =∑K
i=1 qiN (x;µg,i,Σg,i), ∑K i=1 pi = 1, and ∑K
i=1 qi = 1. The expected likelihood kernel is given by
Eθ(f, g) = ∫ ( K∑ i=1 piN (x;µf,i,Σf,i) ) ·(
K∑ j=1 qjN (x;µg,j ,Σg,j)
) dx
= K∑ i=1 K∑ j=1 piqj ∫ N (x;µf,i,Σf,i) · N (x;µg,j ,Σg,j) dx
= K∑ i=1 K∑ j=1 piqjN (0;µf,i − µg,j ,Σf,i +Σg,j)
= K∑ i=1 K∑ j=1 piqje ξi,j
where we note that ∫ N (x;µi,Σi)N (x;µj ,Σj) dx = N (0, µi − µj ,Σi + Σj) (Vilnis and McCallum, 2014) and ξi,j is the log partial energy, given by equation 3.
A.2 Implementation In this section we discuss practical details for training the proposed model. Our code is implemented with Tensorflow (Abadi et al., 2015) and will be made publicly available upon publication.
Word Sampling We use a word sampling scheme similar to the implementation in word2vec (Mikolov et al., 2013a,b). We use window size 10, unless stated otherwise, to generate the the context words for a given center word.
Frequent words such as the, a, to are not as meaningful as relatively more infrequent words such as dog, love, rock and we are often more interested in learning the semantics of the less frequently observed words.
We use the subsampling method to balance the importance of frequent words and rare words, which has been shown to improve the performance of learning word vectors (Mikolov et al., 2013b). This technique discards word wi with probability P (wi) = 1 − √ t/f(wi), where f(wi) is the frequency of word wi in the training corpus and t is a frequency threshold. We use t = 10−5 in our experiments, which is the recommended value for word2vec skip-gram on large datasets.
To generate negative context words, each word type wi is sampled according to a distribution Pn(wi) ∝ U(wi)3/4 which is a distorted unigram distribution U(wi)3/4 that also serves to diminish the relative importance of frequent words. Both subsampling and the negative distribution choice are proven effective in word2vec training (Mikolov et al., 2013b).
Reduction to Diagonal Covariance We use a diagonal Σ, in which case inverting the covariance matrix is trivial and computations are particularly efficient.
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
Let df ,dg denote the diagonal vectors of Σf ,Σg The expression for ξi,j reduces to
ξi,j = − 1
2 D∑ r=1 log(dpr + d q r)
−1 2
∑[ (µp,i − µq,j) ◦
1
dp + dq ◦ (µp,i − µq,j) ] where ◦ denotes element-wise multiplication. The spherical case which we use in all our experiments is similar since we simply replace a vector d with a single value.
Optimization Constraint and Stability We optimize logd since each component of diagonal vector d is constrained to be positive. Similarly, we constrain the probability pi to be in [0, 1] and sum to 1 by optimizing over unconstrained scores si ∈ (−∞,∞) and using a softmax function to convert the scores to probability pi = e
si∑K j=1 e sj .
The loss computation can be numerically unstable if elements of the diagonal covariances are very small, due to the term log(dfr + dgr) and 1dq+dp . Therefore, we add a small constant ϵ = 10−4 so that dfr + dgr and dq + dp becomes dfr + d g r + ϵ and dq + dp + ϵ.
In addition, we observe that ξi,j can be very small which would result in eξi,j ≈ 0 up to machine precision. In order to stabilize the computation in eq. 2, we compute its equivalent form
logE(f, g) = ξi′,j′ + log K∑ j=1 K∑ i=1 piqje ξi,j−ξi′,j′
where ξi′,j′ = maxi,j ξi,j .
Model Parameters and Training Details In the loss function Lθ , we use a margin m = 1. We initialize the word embeddings with D dimension with a uni-
form distribution over [− √ 3 D , √ 3 D ] so that the expectation of variance is 1 and the mean is zero. (LeCun et al., 1998) In our experiments, we use the embedding size D = 50, unless stated otherwise. We initialize each dimension of the diagonal matrix (or a single value for spherical case) with a constant value v = 0.05, which yields a faster convergence given the scale of the norm used above. We also initialize the mixture scores si to be 0 so that the initial probabilities are equal among all K components. We use a batch size of 128.
We also use a separate output embeddings in addition to input embeddings, similar to word2vec implementation (Mikolov et al., 2013a,b). That is, each word has two sets of distributions qI and qO , each of which is a Gaussian mixture. For a given pair of word and context (w, c), we use the input distribution qI for w (input word) and the output distribution qO for context c (output word). We optimize the parameters of both qI and qO and use the trained input distributions qI as our final word representations.
We use mini-batch asynchronous gradient descent with Adagrad (Duchi et al., 2011) which performs adaptive learning rate for each parameter. We also experiment with Adam (Kingma and Ba, 2014) which corrects the bias in adaptive gradient update of Adagrad and is proven very popular for most recent neural network models. However, we found that it is much slower than Adagrad (≈ 10 times). This is because the gradient computation of the model is relatively fast, so a complex gradient update algorithm such as Adam becomes the bottleneck in the optimization. Therefore, we choose to use Adagrad which allows us to better scale to large datasets. We use a linearly decreasing learning rate from 0.05 to 0.00001.

This work uses Gaussian mixtures to represent words and demonstrates its potential in capturing multiple word meanings for polysemy. The training process is done based on a max-margin objective. The expected likelihood kernel is used as the similarity between two words' distributions. Experiment results on word similarity and entailment tasks show the effectiveness of the proposed work.  - Strengths:  The problem is clearly motivated and defined. Gaussian mixtures are much more expressive than deterministic vector representations. It can potentially capture different word meanings by its modes, along with probability mass and uncertainty around those modes. This work represents an important contribution to word embedding.   This work propose a max-margin learning objective with closed-form similarity measurement for efficient training.  This paper is mostly well written.   - Weaknesses:  See below for some questions.   - General Discussion:  In the Gaussian mixture models, the number of gaussian components (k) is usually an important parameter. In the experiments of this paper, k is set to 2. What is your criteria to select k? Does the increase of k hurt the performance of this model? What does the learned distribution look like for a word that only has one popular meaning?  I notice that you use the spherical case in all the experiments (the covariance matrix reduces to a single number). Is this purely for computation efficiency? I wonder what's the performance of using a general diagonal covariance matrix. Since in this more general case, the gaussian mixture defines different degrees of uncertainty along different directions in the semantic space, which seems more interesting.  Minor comments: Table 4 is not referred to in the text. In reference, Luong et al. lacks the publication year.  I have read the response.


A Neural Local Coherence Model
We propose a local coherence model based on a convolutional neural network that operates over the entity grid representation of a text. The model captures long range entity transitions along with entity-specific features without loosing generalization, thanks to the power of distributed representation. We present a pairwise ranking method to train the model in an end-to-end fashion on a task and learn task-specific high level features. Our evaluation on three different coherence assessment tasks demonstrates that our model achieves state of the art results outperforming existing models by a good margin.
1 Introduction and Motivation
What distinguishes a coherent text from a random sequence of sentences is that it binds the sentences together to express a meaning as a whole — the interpretation of a sentence usually depends on the meaning of its neighbors. Coherence models that can distinguish a coherent from incoherent texts have a wide range of applications in text generation, summarization, and coherence scoring.
Several formal theories of coherence have been proposed (Mann and Thompson, 1988a; Grosz et al., 1995; Asher and Lascarides, 2003), and their principles have inspired development of existing coherence models (Barzilay and Lapata, 2008; Lin et al., 2011; Li and Hovy, 2014). Among these models, the entity grid (Barzilay and Lapata, 2008), which is based on Centering Theory (Grosz et al., 1995), is arguably the most popular, and has seen a number of improvements over the years. As shown in Figure 2, the entity grid model represents a text by a grid that captures how grammatical roles of different entities change from
sentence to sentence. The grid is then converted into a feature vector containing probabilities of local entity transitions, which enables machine learning models to learn the degree of text coherence. Extensions of this basic grid model incorporate entity-specific features (Elsner and Charniak, 2011), multiple ranks (Feng and Hirst, 2012), and coherence relations (Feng et al., 2014).
While the entity grid and its extensions have been successful in many applications, they are limited in several ways. Firstly, they use discrete representation for grammatical roles and features, which limits the model to consider sufficiently long transitions (Bengio et al., 2003). Secondly, feature vector computation in existing models is decoupled from the target task, which limits the models to learn task-specific features.
In this paper, we propose a neural architecture for coherence assessment that can capture long range entity transitions along with arbitrary entityspecific features. Our model obtains generalization through distributed representations of entity transitions and entity features. We also present an end-to-end training method to learn task-specific high level features automatically in our model.
We evaluate our approach on three different evaluation tasks: discrimination, insertion, and summary coherence rating, proposed previously for evaluating coherence models (Barzilay and Lapata, 2008; Elsner and Charniak, 2011). Discrimination and insertion involve identifying the right order of the sentences in a text with different levels of difficulty. In summary coherence rating task, we compare the rankings, given by the model, against human pairwise judgments of coherence.
The experimental results show that our neural models consistently improve over the nonneural counterparts (i.e., existing entity grid models) yielding absolute gains of about 4% on discrimination, up to 2.5% on insertion, and more
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
than 4% on summary coherence rating. Our model achieves state of the art results in all these tasks. We have released our code with the submission.
The remainder of this paper is organized as follows. We describe entity grid, its extensions, and its limitations in Section 2. In Section 3, we present our neural model. We describe evaluation tasks and results in Sections 4 and 5. We give a brief account of related work in Section 6. Finally, we conclude with future directions in Section 7.
2 Entity Grid and Its Extensions
Motivated by Centering Theory (Grosz et al., 1995), Barzilay and Lapata (2008) proposed an entity-based model for representing and assessing text coherence. Their model represents a text by a two-dimensional array called entity grid that captures transitions of discourse entities across sentences. As shown in Figure 2, the rows of the grid correspond to sentences, and the columns correspond to discourse entities appearing in the text. They consider noun phrases (NP) as entities, and employ a coreference resolver to detect mentions of the same entity (e.g., Obama, the president). Each entry Gi,j in the entity grid represents the syntactic role that entity ej plays in sentence si, which can be one of: subject (S), object (O), or other (X). In addition, entities not appearing in a sentence are marked by a special symbol (-). If an entity appears more than once with different grammatical roles in the same sentence, the role with the highest rank (S O X) is considered.
To represent an entity grid with a feature vector, Barzilay and Lapata (2008) compute probability for each local entity transition of length k (i.e., {S,O,X,−}k), and represent each grid by a vector of 4k transitions probabilities. To distinguish between transitions of important entities from unimportant ones, they consider the salience of the entities, which they quantify by their occurrence frequency in the document. Assessment of text coherence is then formulated as a ranking problem in an SVM preference ranking framework (Joachims, 2002).
Subsequent studies proposed to extend the basic entity grid model. Filippova and Strube (2007) attempted to improve the model by grouping entities based on semantic relatedness, but did not get significant improvement. Elsner and Charniak (2011) proposed a number of improvements. They initially show significant improvement by includ-
ing non-head nouns (i.e., nouns that do not head NPs) as entities in the grid.1 Then, they extend the grid to distinguish between entities of different types by incorporating entity-specific features like named entity, noun class, modifiers, etc. These extensions led to the best results reported so far.
Entity grid and its extensions have been successfully applied to many downstream tasks including coherence rating (Barzilay and Lapata, 2008), essay scoring (Burstein et al., 2010), story generation (McIntyre and Lapata, 2010), and readability assessment (Pitler et al., 2010; Barzilay and Lapata, 2008). They have also been critical components in state-of-the-art sentence ordering models (Soricut and Marcu, 2006; Elsner and Charniak, 2011; Lin et al., 2011).
2.1 Limitations of Entity Grid Models
Despite its success, existing entity grid models are limited in several ways.
• Existing models use discrete representation for grammatical roles and features, which leads to the so-called curse of dimensionality problem (Bengio et al., 2003). In particular, to model transitions of length k withR different grammatical roles, the basic entity grid model needs to computeRk transition probabilities from a grid. One can imagine that the estimated distribution becomes sparse as k increases. This limits the model to consider longer transitions – existing models use k ≤ 3.
1They match the nouns to detect coreferent entities.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
This problem is exacerbated when we want to include entity-specific features, as the number of parameters grows exponentially with the number of features (Elsner and Charniak, 2011).
• Existing models compute feature representations from entity grids in a task-agnostic way. In other words, feature extraction is decoupled from the target downstream tasks. This can limit the models to learn task-specific features. Therefore, models that can be trained in an end-to-end fashion on different target tasks are desirable.
In the following section, we present a neural architecture that allows us to capture long range entity transitions along with arbitrary entity-specific features without loosing generalization. We also present an end-to-end training method to learn task-specific features automatically.
3 The Neural Coherence Model
Figure 2 summarizes our neural architecture for modeling local coherence, and how it can be trained in a pairwise fashion. The architecture takes a document as input, and first extracts its entity grid.2 The first layer of the neural network transforms each grammatical role in the grid into a distributed representation, a real-valued vector. The second layer computes high-level features by going over each column (transitions) of the grid. The following layer selects the most important high-level features, which are in turn used for coherence scoring. The features computed at different layers of the network are automatically trained by backpropagation to be relevant to the task. In the following, we elaborate on the layers of the neural network model.
(I) Transforming grammatical roles into feature vectors: Grammatical roles are fed to our model as indices taken from a finite vocabulary V . In the simplest scenario, V contains {S,O,X,−}. However, we will see in Section 3.1 that as we include more entity-specific features, V can contain more symbols. The first layer of our network maps each of these indices into a distributed representation Rd by looking up a shared embedding matrix E ∈ R|V |×d. We consider E a model parameter to be learned by backpropagation on a given task. We can initialize E randomly or using pretrained vectors trained on a general coherence task.
2For clarification, pairwise input as shown in the figure is required only to train the model.
Given an entity grid G with columns representing entity transitions over sentences in a document, the lookup layer extracts a d-dimensional vector for each entry Gi,j from E. More formally,
L(G) = 〈 E(G1,1) · · · E(Gi,j) · · · E(Gm,n) 〉 (1) where E(Gi,j) refers to the row in E that corresponds to the grammatical role Gi,j ∈ V ; m is the total number of sentences and n is the total number of entities in the document. The output L(G) is a tensor in Rm×n×d, which is fed to the next layer of the network as we describe below.
(II) Modeling entity transitions: The vectors produced by the lookup layer are combined by subsequent layers of the network to generate a coherence score for the document. To compose higher-level features from the embedding vectors, we make the following modeling assumptions:
• Similar to existing entity grid models, we assume there is no spatio-temporal relation between the entities in a document. In other words, columns in a grid are treated independently.
• We are interested in modeling entity transitions of arbitrary lengths in a location-invariant way. This means, we aim to compose local patches of entity transitions into higher-level representations, while treating the patches independently of their position in the entity grid.
Under these assumptions, the natural choice to tackle this problem is to use a convolutional approach, used previously to solve other NLP tasks (Collobert et al., 2011; Kim, 2014).
Convolution layer: A convolution operation involves applying a filter w ∈ Rk.d (i.e., a vector of weight parameters) to each entity transition of length k to produce a new abstract feature
ht = f(w TLt:t+k−1,j + bt) (2)
where Lt:t+k−1,j denotes the concatenation of k vectors in the lookup layer representing a transition of length k for entity ej in the grid, bt is a bias term, and f is a nonlinear activation function, e.g., ReLU (Nair and Hinton, 2010) in our model.
We apply this filter to each possible k-length transitions of different entities in the grid to generate a feature map, hi = [h1, · · · , hm.n+k−1]. We
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 2: Neural architecture for modeling local coherence and the pairwise training method.
repeat this processN times withN different filters to get N different feature maps (Figure 2). Notice that we use a wide convolution (Kalchbrenner et al., 2014), as opposed to narrow, to ensure that the filters reach entire columns of a grid, including the boundary entities. This is done by performing zero-padding, where out-of-range (i.e., for t < 0 or t > {m,n}) vectors are assumed to be zero.
Convolutional filters learn to compose local transition features of a grid into higher-level representations automatically. Since it operates over the distributed representation of grid entries, compared to traditional grid models, the transition length k can be sufficiently large (e.g., 5 − 8 in our experiments) to capture long-range transitional dependencies without overfitting on the training data. Moreover, unlike existing grid models that compute transition probabilities from a single document, embedding vectors and convolutional filters are learned from all training documents, which helps the neural framework to obtain better generalization and robustness.
Pooling layer: After the convolution, we apply a max-pooling operation to each feature map.
m = [µp(h 1), · · · , µp(hN )] (3)
where µp(hi) refers to the max operation applied to each non-overlapping3 window of p features in the feature map hi. Max-pooling reduces the output dimensionality by a factor of p, and it drives the model to capture the most salient local features
3We set the stride size to be the same as the pooling length p to get non-overlapping regions.
from each feature map in the convolutional layer.
Coherence scoring: Finally, the max-pooled features are used in the output layer of the network to produce a coherence score y ∈ R.
y = vTm+ b (4)
where v is the weight vector and b is a bias term.
Why it works: Intuitively, each filter detects a specific transition pattern (e.g., ‘SS-O-X’ for a coherent text), and if this pattern occurs somewhere in the grid, the resulting feature map will have a large value for that particular region and small values for other regions. By applying max pooling on this feature map, the network then discovers that the transition appeared in the grid.
3.1 Incorporating Entity-Specific Features
Our model as described above neuralizes the basic entity grid model that considers only entity transitions without distinguishing between types of the entities. However, as Elsner and Charniak (2011) pointed out entity-specific features could be crucial for modeling local coherence. One simple way to incorporate entity-specific features into our model is to attach the feature value (e.g., named entity type) with the grammatical role in the grid. For example, if an entity ej of type PERSON appears as a subject (S) in sentence si, the grid entry Gi,j can be encoded as PERSON-S.
3.2 Training
Our neural model assigns a coherence score to an input document d based on the degree of lo-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
cal coherence observed in its entity grid G. Let y = φ(G|θ) define our model that transforms an input grid G to a coherence score y through a sequence of lookup, convolutional, pooling, and linear projection layers with parameter set θ. The parameter set θ includes the embedding matrix E, the filter matrix W , the weight vector v, and the biases. We use a pairwise ranking approach (Collobert et al., 2011) to learn θ.
The training set comprises ordered pairs (di, dj), where document di exhibits a higher degree of coherence than document dj . As we will see in Section 4 such orderings can be obtained automatically or through manual annotation. In training, we seek to find θ that assigns a higher coherence score to di than to dj . We minimize the following ranking objective with respect to θ:
J (θ) = max{0, 1− φ(Gi|θ) + φ(Gj |θ)} (5)
where Gi and Gj are the entity grids corresponding to documents di and dj , respectively. Notice that (also shown in Figure 2) the network shares its layers (and hence θ) to obtain φ(Gi|θ) and φ(Gj |θ) from a pair of input grids (Gi, Gj).
Barzilay and Lapata (2008) adopted a similar ranking criterion using an SVM preference kernel learner as they argue coherence assessment is best seen as a ranking problem as opposed to classification (coherent vs. incoherent). Also, the ranker gives a scoring function φ that a text generation system can use to compare alternative hypotheses.
4 Evaluation Tasks
We evaluate the effectiveness of our coherence models on two different evaluation tasks: sentence ordering and summary coherence rating.
4.1 Sentence Ordering
Following (Elsner and Charniak, 2011), we evaluate our models on two sentence ordering tasks: discrimination and insertion.
In the discrimination task (Barzilay and Lapata, 2008), a document is compared to a random permutation of its sentences, and the model is considered correct if it scores the original document higher than the permuted one. We use 20 permutations of each document in the test set in accordance with previous work.
In the insertion task (Elsner and Charniak, 2011), we evaluate models based on their ability
Sections # Doc. # Pairs Avg. # Sen.
TRAIN 00-13 1,378 26,422 21.5 TEST 14-24 1,053 20,411 22.3
Table 1: Statistics on WSJ dataset.
to locate the original position of a sentence previously removed from a document. To measure this, each sentence in the document is removed in turn, and an insertion place is located for which the model gives the highest coherence score to the document. The insertion score is then computed as the average fraction of sentences per document reinserted in their actual position.
Discrimination can be easier for longer documents, since a random permutation is likely to be different than the original one. Insertion is a much more difficult task since the candidate documents differ only by the position of one sentence.
Dataset: For sentence ordering tasks, we use the Wall Street Journal (WSJ) portion of Penn Treebank, as used by (Elsner and Charniak, 2008, 2011; Lin et al., 2011; Feng et al., 2014). Table 1 gives basic statistics about the dataset. Following previous works, we use 20 random permutations of each article, and we exclude permutations that match the original document.4 The fourth column (# Pairs) in Table 1 shows the resulting number of (original, permuted) pairs used for training our model and for testing in the discrimination task.
Some previous studies (Barzilay and Lapata, 2008; Li and Hovy, 2014) used the AIRPLANES and the EARTHQUAKES corpora, which contain reports on airplane crashes and earthquakes, respectively. Each of these corpora contains 100 articles for training and 100 articles for testing. The average number of sentences per article in these two corpora is 10.4 and 11.5, respectively.
We preferred WSJ corpus for several reasons. First and most importantly, WSJ corpus is larger than other corpora (see Table 1). Large training set is crucial for learning effective deep learning models (Collobert et al., 2011), and a large enough test set is necessary to make a general comment about model performance. Secondly, as Elsner and Charniak (2011) pointed out, texts in AIRPLANES and EARTHQUAKES are constrained in style, whereas WSJ documents are more like normal informative articles. Thirdly, we could re-
4Short articles may produce many matches.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
produce results on this dataset for the competing systems (e.g., entity grid and its extensions) using publicly available Brown coherence toolkit.5
4.2 Summary Coherence Rating
We further evaluate our models on the summary coherence rating task proposed by Barzilay and Lapata (2008), where we compare rankings given by a model to a pair of summaries against rankings elicited from human judges.
Dataset: The summary dataset was extracted from the Document Understanding Conference (DUC’03), which contains 6 clusters of multidocument summaries produced by human experts and 5 automatic summarization systems. Each cluster has 16 summaries of a document with pairwise coherence rankings given by humans judges; see (Barzilay and Lapata, 2008) for details on the annotation method. There are 144 pairs of summaries for training and 80 pairs for testing.
5 Experiments
In this section, we present our experiments — the models we compare, their settings, and the results.
5.1 Models Compared
We compare our coherence model against a random baseline and several existing models.
Random: The Random baseline makes a random decision for the evaluation tasks.
Graph-based Model: This is the graph-based unsupervised model proposed by Guinaudeau and Strube (2013). We use the implementation from the cohere6 toolkit (Smith et al., 2016), and run it on the test set with syntactic projection (command line option ‘projection=3’) for graph construction. This setting yielded best scores for this model.
Grid-all nouns (E&C): This is the simple extension of the original entity grid model, where all nouns are considered as entities. Elsner and Charniak (2011) report significant gains by considering all nouns as opposed to only head-nouns. Results for this model were obtained by training the baseline entity grid model (command line option ‘-n’) in Brown coherence toolkit on our dataset.
5https://bitbucket.org/melsner/browncoherence 6https://github.com/karins/CoherenceFramework
Extended grid (E&C): This represents the extended entity grid model of Elsner and Charniak (2011) that uses 9 entity-specific features; 4 of them were computed from external corpora. This model considers all nouns as entities. For this system, we train the extended grid model (command line option ‘-f’) in Brown coherence toolkit.
Grid-CNN: This is our proposed neural extension of the basic entity grid (all nouns), where we only consider entity transitions as input.
Extended Grid-CNN: This corresponds to our neural model that incorporates entity-specific features following the method described in Section 3.1. To keep the model simple, we include only three entity-specific features from (Elsner and Charniak, 2011) that are easy to compute and do not require any external corpus. The features are: (i) named entity type, (ii) salience as determined by occurrence frequency of the entity, and (iii) whether the entity has a proper mention.
A remark: We also experimented with the distributed sentence model proposed recently by Li and Hovy (2014). We trained it on our WSJ corpus using their code7 with the same setting that produced the best results on the discrimination task. However, the results on our dataset were very disappointing – accuracy of only 19.34% in discrimination. We were not sure what could go wrong, therefore, we excluded it from our table of results.
5.2 Settings for Neural Models
We held out 10% of the training documents to form a development set (DEV) on which we tune the hyper-parameters of our neural models. For discrimination and insertion tasks, the resulting DEV set contains 138 articles and 2,678 pairs after removing the permutations that match the original documents. For the summary rating task, DEV contains 14 pairs of summaries.
We implement our models in Theano (Theano Development Team, 2016). We use rectified linear units (ReLU) as activations (f ). The embedding matrix is initialized with samples from uniform distribution U(−0.01, 0.01), and the weight matrices are initialized with samples from glorotuniform distribution (Glorot and Bengio, 2010).
We train the models by optimizing the pairwise ranking loss in Equation 5 using the gradientbased online learning algorithm RMSprop with
7http://cs.stanford.edu/ bdlijiwei/code/
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Batch Emb. Dropout Filter Win. Pool
Grid-CNN 128 100 0.5 150 6 6 Ext. Grid-CNN 32 100 0.5 150 5 6
Table 2: Optimal hyper-parameter setting for our neural models based on development set accuracy.
Discr. Ins. Acc F1
Random 50.0 50.0 12.60
Graph-based (G&S) 64.23 65.01 11.93
Grid-all nouns (E&C) 81.58 81.60 22.13 Extended Grid (E&C) 84.95 84.95 23.28
Grid-CNN 85.57† 85.57† 23.12 Extended Grid-CNN 88.69† 88.69† 25.95†
Table 3: Results on Discrimination and Insertion tasks. † indicates a neural model is significantly superior to its nonneural counterpart with p-value < 0.01.
parameters (ρ and ) set to the values suggested by Tieleman and Hinton (2012).8 We use up to 25 epochs. To avoid overfitting, we use dropout (Srivastava et al., 2014) of hidden units, and do early stopping by observing accuracy on the DEV set – if the accuracy does not increase for 10 consecutive epochs, we exit with the best model recorded so far. We search for optimal minibatch size in {16, 32, 64, 128}, embedding size in {80, 100, 200}, dropout rate in {0.2, 0.3, 0.5}, filter number in {100, 150, 200, 300}, window size in {2, 3, 4, 5, 6, 7, 8}, and pooling length in {3, 4, 5, 6, 7}. Table 2 shows the optimal hyperparameter setting for our models. The best model on DEV is then used for the final evaluation on the TEST set. We run each experiment five times, each time with a different random seed, and we report the average of the runs to avoid any randomness in results. Statistical significance tests are done using an approximate randomization test based on the accuracy. We used SIGF V.2 (Padó, 2006) with 10,000 iterations.
5.3 Results on Sentence Ordering
Table 3 shows the results on discrimination and insertion tasks. Among the existing models, the graph-based model gets the lowest scores, where the extended grid gets the highest scores on both tasks. By neuralizing the basic grid model (Grid-
8Other adaptive algorithms, e.g., ADAM (Kingma and Ba, 2014), ADADELTA (Zeiler, 2012) gave similar results.
all nouns), our Grid-CNN model delivers absolute improvements of about 4% in discrimination and 1% in insertion. When we compare our Extended Grid-CNN with its non-neural counterpart Extended Grid, we observe similar gains in discrimination and more gains (2.5%) in insertion. Note that the Extended Grid-CNN yields these improvements considering only a subset of the Extended Grid features. This demonstrates the effectiveness of distributed representation and convolutional feature learning method.
Compared to discrimination, gain in insertion is less verbose. There could be two reasons. First, as mentioned before, insertion is a harder task than discrimination. Second, our models were not trained specifically on the insertion task. The model that is trained to distinguish an original document from its random permutation may learn features that are not specific enough to distinguish documents when only one sentence differs. It will be interesting to see how the model performs when we train it on the insertion task directly.
5.4 Results on Summary Coherence Rating
Table 4 presents the results on the summary coherence rating task, where we compare our models with the graph-based method and the reported results of Barzilay and Lapata (2008) on the same experimental setting.9 Since there are not many training instances, our neural models may not learn well for this task. Therefore, we also present versions of our model, where we use pre-trained models from discrimination task on WSJ corpus (last two rows in the table ). The pre-trained models are then fine-tuned on the summary rating task.
We can observe that even without pre-training
9The extended grid model does not use pairwise training, therefore could not be trained on the summarization dataset.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
our models outperform existing models, and pretraining gives further improvements. Specifically, Pre-trained Grid-CNN gives an improvement of 2.5% over the Grid model, and including entity features pushes the improvement further to 3.7%.
6 Related Work
Barzilay and Lapata (2005, 2008) introduced the entity grid representation of discourse to model local coherence that captures the distribution of discourse entities across sentences in a text. They also introduced three tasks to evaluate the performance of coherence models: discrimination, summary coherence rating, and readability.
A number of extensions of the basic entity grid model has been proposed. Elsner and Charniak (2011) included entity-specific features to distinguish between entities. Feng and Hirst (2012) used the basic grid representation, but improved its learning to rank scheme. Their model learns not only from original document and its permutations but also from ranking preferences among the permutations themselves. Guinaudeau and Strube (2013) convert a standard entity grid into a bipartite graph representing entity occurrences in sentences. To model local entity transition, the method constructs a directed projection graph representing the connection between adjacent sentences. Two sentences have a connected edge if they share at least one entity in common. The coherence score of the document is then computed as the average out-degree of sentence nodes.
In addition, there are some approaches that model text coherence based on coreferences and discourse relations. Elsner and Charniak (2008) proposed the discourse-new model by taking into account mentions of all referring expression (i.e., NPs) whether they are first mention (discoursenew) or subsequent (discourse-old) mentions. Given a document, they run a maximum-entropy classifier to detect each NP as a label Lnp ∈ {new, old}. The coherence score of the document is then estimated by ∏ np:NPs P (Lnp|np). In this work, they also estimate text coherence through pronoun coreference modeling. Lin et al. (2011) assume that a coherent text has certain discourse relation patterns. Instead of modeling entity transitions, they model discourse role transitions between sentences. In a follow up work, Feng et al. (2014) trained the same model but using features derived from deep discourse struc-
tures annotated with Rhetorical Structure Theory or RST (Mann and Thompson, 1988b) relations. Louis and Nenkova (2012) introduced a coherence model based on syntactic patterns in text by assuming that sentences in a coherent discourse should share the same structural syntactic patterns.
In recent years, there has been a growing interest in neuralizing traditional NLP approaches – language modeling (Bengio et al., 2003), sequence tagging (Collobert et al., 2011), syntactic parsing (Socher et al., 2013), and discourse parsing (Li et al., 2014), etc. Following this tradition, in this paper we propose to neuralize the popular entity grid models. Li and Hovy (2014) also proposed a neural framework to compute coherence score of a document by estimating coherence probability for every window of L sentences (in their experiments, L = 3). First, they use a recurrent or a recursive neural network to compute the representation for each sentence in L from its words and their pre-trained embeddings. Then the concatenated vector is passed through a non-linear hidden layer, and finally the output layer decides if the window of sentences is a coherent text or not. Our approach is fundamentally different from their approach; our model operates over entity grids, and we use convolutional architecture to model sufficiently long entity transitions.
7 Conclusion and Future Work
We presented a local coherence model based on a convolutional neural network that operates over the distributed representation of entity transitions in the grid representation of a text. Our architecture can model sufficiently long entity transitions, and can incorporate entity-specific features without loosing generalization power. We described a pairwise ranking approach to train the model on a target task and learn task-specific features. Our evaluation on discrimination, insertion and summary coherence rating tasks demonstrates the effectiveness of our approach yielding the best results reported so far on these tasks.
In future, we would like to include other sources of information in our model. Our initial plan is to include rhetorical relations, which has been shown to benefit existing grid models (Feng et al., 2014). We would also like to extend our model to other forms of discourse, especially, asynchronous conversations, where participants communicate with each other at different times (e.g., forum, email).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper proposes a convolutional neural network approach to model the coherence of texts. The model is based on the well-known entity grid representation for coherence, but puts a CNN on top of it.   The approach is well motivated and described, I especially appreciate the clear discussion of the intuitions behind certain design decisions (e.g. why CNN and the section titled 'Why it works').  There is an extensive evaluation on several tasks, which shows that the proposed approach beats previous methods. It is however strange that one previous result could not be reproduced: the results on Li Hovy (2014) suggest an implementation or modelling error that should be addressed.  Still, the model is a relatively simple 'neuralization' of the entity grid model. I didn't understand why 100-dimensional vectors are necessary to represent a four-dimensional grid entry (or a few more in the case of the extended grid). How does this help? I can see that optimizing directly for coherence ranking would help learn a better model, but the difference of transition chains for up to k=3 sentences vs. k=6 might not make such a big difference, especially since many WSJ articles may be very short.  The writing seemed a bit lengthy, the paper repeats certain parts in several places, for example the introduction to entity grids. In particular, section 2 also presents related work, thus the first 2 3 of section 6 are a repetition and should be deleted (or worked into section 2 where necessary). The rest of section 6 should probably be added in section 2 under a subsection (then rename section 2 as related work).  Overall this seems like a solid implementation of applying a neural network model to entity-grid-based coherence. But considering the proposed consolidation of the previous work, I would expect a bit more from a full paper, such as innovations in the representations (other features?) or tasks.  minor points:  - this paper may benefit from proof-reading by a native speaker: there are articles missing in many places, e.g. '_the_ WSJ corpus' (2x), '_the_ Brown ... toolkit' (2x), etc.  - p.1 bottom left column: 'Figure 2' to 'Figure 1'  - p.1 Firstly Secondly to First, Second  - p.1 'limits the model to' to 'prevents the model from considering ...' ?  - Consider removing the 'standard' final paragraph in section 1, since it is not necessary to follow such a short paper.