End-to-End Neural Relation Extraction with Global Optimization
Neural networks have recently shown promising results for relation extraction. State-of-the-art models cast the task as an end-to-end problem, solved incrementally using a local classifier. Yet previous work using statistical models have demonstrated that global optimization can achieve better performances compared to local classification. We build a globally optimized neural model for end-to-end relation extraction, proposing novel LSTM features in order to better learn representations. Experiments show that our model is highly effective, achieving the best performances on two standard benchmarks.
Neural networks have recently shown promising results for relation extraction. State-of-the-art models cast the task as an end-to-end problem, solved incrementally using a local classifier. Yet previous work using statistical models have demonstrated that global optimization can achieve better performances compared to local classification. We build a globally optimized neural model for end-to-end relation extraction, proposing novel LSTM features in order to better learn representations. Experiments show that our model is highly effective, achieving the best performances on two standard benchmarks.
1 Introduction
Extracting entities (Florian et al., 2006, 2010) and relations (Zhao and Grishman, 2005; Jiang and Zhai, 2007; Sun et al., 2011; Plank and Moschitti, 2013) from unstructured texts have been two central tasks in information extraction (Grishman, 1997; Doddington et al., 2004). Traditional approaches to relation extraction take entity recognition as a predecessor step in a pipeline (Zelenko et al., 2003; Chan and Roth, 2011), predicting relations between given entities.
In recent years, there has been a surge of interest in performing end-to-end relation extraction, jointly recognizing entities and relations given free text inputs (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016; Gupta et al., 2016). End-to-end learning prevents error propagation in the pipeline approach, and allows cross-task dependencies to be modeled explicitly for entity recognition. As a result, it gives better relation extraction accuracies compared to pipelines.
Miwa and Bansal (2016) were among the first to use neural networks for end-to-end relation extraction, showing highly promising results. In particular, they used a bidirectional LSTM (Graves et al., 2013) to learn hidden word representations under a sentential context, and further leveraged a tree-structured LSTMs (Tai et al., 2015) to encode syntactic information, given the output of a parser. The resulting representations are then used for making local decisions for entity and relation extraction incrementally, leading to much improved results compared with the best statistical model (Li and Ji, 2014). This demonstrates the strength of neural representation learning for endto-end relation extraction.
On the other hand, Miwa and Bansal (2016)’s model is trained locally, without considering structural correspondences between incremental decisions. This is unlike existing statistical methods, which utilize well-studied structured prediction methods to address the problem (Li and Ji, 2014; Miwa and Sasaki, 2014). As has been commonly understood, learning local decisions for structured prediction can lead to label bias (Lafferty et al., 2001), which prevents globally optimal structures from receiving optimal scores by the model. We address this potential issue by building a structural neural model for end-to-end relation extraction, following a recent line of efforts on globally optimized models for neural structured prediction (Zhou et al., 2015; Watanabe and Sumita, 2015; Andor et al., 2016; Wiseman and Rush, 2016).
In particular, we follow Miwa and Bansal (2016), casting the task as an end-to-end tablefilling problem. This is different from the actionbased method of Li and Ji (2014), yet has shown to be more flexible and accurate (Miwa and Sasaki, 2014). We take a different approach to representation learning, addressing two potential limitations of Miwa and Bansal (2016).
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
First, Miwa and Bansal (2016) rely on external syntactic parsers for obtaining syntactic information, which is crucial for relation extraction (Culotta and Sorensen, 2004; Zhou et al., 2005; Bunescu and Mooney, 2005; Qian et al., 2008). However, parsing errors can lead to encoding inaccuracies of tree-LSTMs, thereby hurting relation extraction potentially. We take an alternative approach to integrating syntactic information, by taking the hidden LSTM layers of a bi-affine attention parser (Dozat and Manning, 2016) to augment input representations. Pretrained for parsing, such hidden layers contain rich syntactic information on each word, yet do not explicitly represent parsing decisions, thereby avoiding potential issues caused by incorrect parses.
Our method is also free from a particular syntactic formalism, such as dependency grammar, constituent grammar or CCG, requiring only hidden representations on word that contain syntactic information. In contrast, the method of Miwa and Bansal (2016) must consider tree LSTM formulations that are specific to grammar formalisms, which can be very different (Tai et al., 2015).
Second, Miwa and Bansal (2016) did not explicitly learn the representation of segments when predicting entity boundaries or making relation classification decisions, which can be intuitively highly useful. We take the LSTM-Minus method of Wang and Chang (2016), modelling a segment as the difference between its last and first LSTM hidden vectors. This method is highly efficient, yet gives as accurate results as compared to more complex neural network structures to model a span of words (Cross and Huang, 2016).
Evaluation on two benchmark datasets shows that our method outperforms previous methods of Miwa and Bansal (2016), Li and Ji (2014) and Miwa and Sasaki (2014), giving the best reported results on both benchmarks. Our code is available under GPL at https://github.com/〈anonymized〉.
2 Model
2.1 Task Definition
As shown in Figure 1, the goal of relation extraction is to mine relations from raw texts. It consists of two sub-tasks, namely entity detection, which recognizes valid entities, and relation classification, which determines the relation categories over entity pairs. We follow recent studies and recognize entities and relations as one single task.
2.2 Method
We follow Miwa and Sasaki (2014) and Gupta et al. (2016), treating relation extraction as a tablefilling problem, performing entity detection and relation classification using a single incremental model, which is similar in spirit to Miwa and Bansal (2016). Figure 2 shows an example of the table-filling process.
Formally, given a sentence w1w2 · · ·wn, we maintain a table Tn×n, where T (i, j) denotes the relation between wi and wj . When i = j, T (i, j) denotes an entity boundary label. We map entity words into labels under the BILOU (Begin, Inside, Last, Outside, Unit) scheme, assuming that there are no overlapping entities in one sentence (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016). Only the upper triangular table is necessary for indicating the relations.
We adopt the close-first left-to-right strategy (Miwa and Sasaki, 2014) to map the twodimensional table into a sequence, in order to fill the table incrementally. During the table-filling process, we take two label sets for entity detection (i = j) and relation classification (i < j), respectively. The labels for entity detection include {B*, I-*, L-*, O, U-* }, where * denotes the entity type, and the labels for relation classification are {−→∗ ,←−∗ }, where * denotes the relation category.
At each step, given a partially-filled table T , we determine the most suitable label l for the next step (order in Figure 2) using a scoring function:
score(T, l) = WlhT , (1)
where Wl is a model parameter and hT is the vector representation of T . Based on the function, we aim to find the best label sequence l1 · · · lm, where m = n(n+1)2 , and the resulting sequence of partially-filled tables is T0T1 · · ·Tm, where Ti = FILL(Ti−1, li), and T0 is an empty table. Different from previous work, we investigate a structural
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Associated Press writer Patrick McDowell in Kuwait City
Associated Press writer Patrick McDowell
in Kuwait
City
1 B-ORG 9 ⊥ 16 ⊥ 22 ⊥ 27 ⊥ 31 ⊥ 34 ⊥ 36 ⊥ 2 L-ORG 10←−−−−−ORG-AFF 17 ⊥ 23 ⊥ 28 ⊥ 32 ⊥ 35 ⊥
3 U-PER 11 ⊥ 18 ⊥ 24 ⊥ 29 ⊥ 33 ⊥ 4 B-PER 12 ⊥ 19 ⊥ 25 ⊥ 30 ⊥
5 L-PER 13 ⊥ 20 ⊥ 26−−−→PHYS 6 O 14 ⊥ 21 ⊥
7 B-GPE 15 ⊥ 8 L-GPE
Figure 2: Table-filling example, where numbers indicate the filling order.
hw
⊕ ewe′w et hchar ⊕ ⊕ CNN
character sequence
Figure 3: Word representations.
model that is optimized for the label sequence l1 · · · lm globally, rather than for each li locally.
2.3 Representation Learning
At the ith step, we determine the label li of the next table slot based on the current hypothesis Ti−1. Following Miwa and Bansal (2016), we use a neural network to learn the vector representation of Ti−1, and then use Equation 1 to rank candidate next labels. There are two types of input features, including the word sequence w1w2 · · ·wn, and the readily filled label sequence l1l2 · · · li−1. We build a neural network to represent Ti−1.
2.3.1 Word Representation
Shown in Figure 3, we represent each word wi by hwi using its word form, POS tag and characters. Based on the word form, we use two different embeddings, one being obtained by using a randomly initialized look-up table Ew, tuned during training and represented by ew, and the other being a pretrained external word embedding from E′w, which is fixed and represented by e′w.
1 For a POS tag t, its embedding et is obtained from a look-up table Et similar to Ew.
The above two components have also been used by Miwa and Bansal (2016). In addition, we enhance the word representation by its character sequence (Ballesteros et al., 2015; Lample et al., 2016), using a convolution neural network (CNN) to derive a character-based word representation hchar, which has been demonstrated effective for
1We use the set of pre-trained glove word embeddings available at http://nlp.stanford.edu/data/glove.6B.zip as external word embeddings.
hi−1...... hi ...... hj ......
hseg = hj − hi−1
Figure 4: Segment representation.
several NLP tasks (dos Santos and Gatti, 2014). We obtain the final hwi based on a non-linear feedforward layer on e′w ⊕ ew ⊕ et ⊕ hchar, where ⊕ denotes concatenation.
2.3.2 Label Representation
In addition to the word sequence, the history label sequence l1l2 · · · li−1, especially the labels representing detected entities, is also useful disambiguation. For example, the previous entity boundary label can be helpful to deciding the boundary label of the current word. During relation classification, the types of the entities involved can indicate the relation category between them. We exploit the diagonal label sequence of partial table T , which denotes entity boundaries of words, to enhance the representation learning. We obtain a word’s entity boundary label embedding el by a randomly initialized looking-up table El.
2.3.3 LSTM Features
We follow Miwa and Bansal (2016), learning global context representations using LSTMs. Three basic LSTM structures are used: a leftto-right word LSTM ( −−−−→ LSTMw), a right-to-left word LSTM ( ←−−−− LSTMw) and a left-to-right entity boundary label LSTM ( −−−−→ LSTMe). Each LSTM derives a sequence of hidden vectors for inputs. For example, for w1w2 · · ·wn, −−−−→ LSTMw gives hw,→1 h w,→ 2 · · ·h w,→ n .
Different from Miwa and Bansal (2016), who use the output hidden vectors {hi} of LSTMs to represent words, we exploit segment representations as well. In particular, for a segment of text [i, j], the representation is computed by using LSTM-Minus (Wang and Chang, 2016), shown by Figure 4, where hj − hi−1 in a left-to-right LSTM
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
...... ...... ......
...... ...... ......
...... ......
hT
−−−−→ LSTMw
←−−−− LSTMw
−−−−→ LSTMe
...... j − 1 j ...... i− 1 i i+ 1 ......
concatenate
feed-forward
(a) entity detection
...... ...... ...... ...... ......
...... ...... ...... ...... ......
...... ...... ...... ...... ......
hT
−−−−→ LSTMw
←−−−− LSTMw
−−−−→ LSTMe
left entityi middle entityj right
concatenate
feed-forward
(b) relation classification
Figure 5: Feature representation.
and hi − hj+1 in a right-to-left LSTM are used to represent the segment [i, j]. The segment representations can reflect entities in a sentence, and thus can be potentially useful for both entity detection and relation extraction.
2.3.4 Feature Representation
We use separate feature representations for entity detection and relation classification, both of which are induced according to the above three LSTM structures. In particular, we first extract a set of base neural features, and then concatenate them and feed them into a non-linear neural layer for entity detection and relation classification, respectively. Figure 5 shows the overall representation.
[Entity Detection] Figure 5(a) shows the feature representation method for the entity detection actions. First, we extract six feature vectors from the three basic LSTMs, three of which are word features, namely hw,→i , h w,← i and h e,→ i−1 , and the remaining are segment features, namely hw,→[j,i−1], hw,←[j,i−1] and h e,→ [j,i−1], where j denotes the start po-
sition of the previous entity.2 The six vectors are concatenated and then fed into a non-linear layer for entity detection.
[Relation Classification] Figure 5(b) shows the feature representation method for relation classification. Similar to entity detection, we extract a set of features from the basic LSTMs (
−−−−→ LSTMw,←−−−−
LSTMw and −−−−→ LSTMe), and then concatenate them for a non-linear classification layer. The differences between relation classification with entity detection lie in the range of hidden layers from LSTMs. For relation classification between i and j, we split each LSTM into five segments according to the two entities ended with i and j. Formally, let [s(i), i] and [s(j), j] denote the two entities above, where s(·) denotes the start position of an entity, the resulted segments are [0, s(i)− 1] (i.e., left, in Figure 5(b)), [s(i), i] (i.e., entityi), [i + 1, s(j) − 1] (i.e., middle), [s(j), j] (i.e., entityj) and [j + 1, n] (i.e., right), respectively. For the word LSTMs, we extract all five segment features, while the entity label LSTM, we only use the segment features of entityi and entityj .
2.3.5 Syntactic Features
Previous work has shown that syntactic features are useful for relation extraction (Zhou et al., 2005). For example, the shortest dependency path has been used by several relation extraction models (Bunescu and Mooney, 2005; Miwa and Bansal, 2016). Here we propose a novel method to integrate syntax, without need for prior knowledge on concrete syntactic structures.
In particular, many state-of-the-art syntactic parsers use encoder-decoder neural models (Buys and Blunsom, 2015; Kiperwasser and Goldberg, 2016), where the encoder represents the features of the input sentences. For example, LSTM over the input word/tag sequences has been used frequently (Kiperwasser and Goldberg, 2016). The decoder can also leverage partially-parsed results, such as features from partial syntactic trees. Table 1 shows the encoder structures of three state-ofthe-art dependency parsers.
Our method is to dump the encoder source representations of state-of-the-art parsers, and then use them directly as part of input embeddings in our proposed model. Denoting the dumped syntactic features on each word as hsyn1 h syn 2 · · ·h syn n , we feed them into a non-linear neural layer, 2The non-entity word is treated as a special unit entity to extract segmental features.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Models Encoder LAS S-LSTM (2015) 1-Layer LSTM 90.9
K&G (2016) 2-Layer Bi-LSTM 91.9 D&M (2016) 4-Layer Bi-LSTM 93.8
Table 1: The encoder structures and performances of three state-of-the-art dependency parsers, where S-LSTM (2015) refers to Dyer et al. (2015), K&G (2016) refers to the best parser of Kiperwasser and Goldberg (2016), D&M (2016) refers to Dozat and Manning (2016), and LAS (labeled attachment score) is the major evaluation metric for dependency parsing.
and then generate two LSTMs (bi-directional) based on the outputs, namely
−−−−→ LSTMsyn and←−−−− LSTMsyn, respectively, augmenting the original three LSTMs into five LSTMs. Features are extracted from the two new LSTMs in the same way as from the basic bi-directional word LSTMs.
We exploit the parser of Dozat and Manning (2016) to extract syntactic features, since it achieves the current best performance for dependency parsing. Our method can be easily generalized to the use of other parsers, which are potentially useful for our task as well. For example, we can use a constituent parser in the same way.
2.4 Training and Search
2.4.1 Local Optimization
Previous work (Miwa and Bansal, 2016; Gupta et al., 2016) trains model parameters by modeling each step for labeling one input sentence separately. Given a partial table T , we first obtain its neural representation hT , and then compute the next label scores {l1, l2, · · · , ls} using Equation 1. The output scores are regularized into a probability distribution {pl1 , pl2 , · · · , pls} by using a softmax layer. Our training objective is to minimize the cross-entropy loss between this output distribution with the gold-standard distribution:
loss(T, lgi ,Θ) = − log plgi , (2)
where lgi is the gold-standard next label for T , and Θ is the set of all model parameters. We refer this training method as local optimization, because it maximizes the score of the gold-standard label at each step locally.
During the decoding phase, the greedy search strategy is applied in consistence with the training. At each step, we find the highest-scored label
Algorithm 1 Beam-search. agenda← { (empty table, score=0.0) } for i in 1 · · ·max-step
next scored tables← { } for scored table in agenda
labels← NEXTLABELS(scored table) for next label in labels
new← FILL(scored table, next label) ADDITEM(next scored tables, new)
agenda← TOP-B(next scored tables, B)
based on the current partial table, before going on to the next step.
2.4.2 Global Optimization
We exploit the global optimization strategy of Andor et al. (2016), maximizing the cumulative score of the gold-standard label sequence for one sentence as a unit. Global optimization has achieved success for several NLP tasks under the neural setting (Zhou et al., 2015; Watanabe and Sumita, 2015). For relation extraction, global learning gives the best performances under the discrete setting (Li and Ji, 2014; Miwa and Sasaki, 2014). We study such models here under the neural setting.
Given a label sequence of l1l2 · · · li, the score of Ti is defined as follows:
score(Ti) = i∑
j=0
score(Tj−1, lj)
= score(Ti−1) + score(Ti−1, li),
(3)
where score(T0) = 0 and score(Ti−1, li) is computed by Equation 1. By this definition, we maximize the scores of all gold-standard partial tables.
Again cross-entropy loss is used to perform model updates. At each step i, the objective function is defined by:
loss(x, T gi ,Θ) = − log pT gi
= − log score(T gi )∑ T ′i score(T ′i ) ,
(4)
where x denotes the input sentence, T gi denotes the gold-standard state at step i, and T ′i are all partial tables that can be reached at step i.
The major challenge is to compute pT gi , because we cannot traverse all partial tables that are valid at step i, since their count increases exponentially by the step number. We follow Andor
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
et al. (2016), approximating the probability by using beam search and early-update.
Shown in Algorithm 1, we use standard beam search, using an agenda to maintain B highestscored partially-filled tables at each step. When each action of table filling is taken, all hypotheses in the agenda are expanded by enumerating the next labels, and the B highest-scored resulting tables are used to replace the agenda for the next step. Search begins with the agenda containing an empty table, and finishes when all cells of the tables in the agenda have been filled. When the beam size is 1, the algorithm is the same as greedy decoding. When the beam size is larger than 1, however, error propagation from greedy steps can be alleviated. For training, the same beam search algorithm is applied to training examples, and early-update (Collins and Roark, 2004) is used to fix search errors.
3 Experiments
3.1 Data and Evaluation
We evaluate the proposed model on two datasets, namely the ACE05 data and the corpus of Roth and Yih (2004) (CONLL04), respectively. The ACE05 data defines seven coarse-grained entity types and six coarse-grained relation categories, while the CONLL04 data defines four entity types and five relation categories.
For the ACE05 dataset, we follow Li and Ji (2014) and Miwa and Bansal (2016), splitting and preprocessing the dataset into training, development and test sets.3 For CONLL04, we follow Miwa and Sasaki (2014) to split and preprocess the dataset into training and test corpus, and divide 10% of the training corpus for development.
We use the micro F1-measure as the major metric to evaluate model performances, treating an entity as correct when its head region and type are both exactly matched,4 and regard a relation as correct when the argument entities and the relation category are all correct.
3.2 Parameter Tuning
We update all model parameters by back propagation using Adam (Kingma and Ba, 2014) with a learning rate 10−3, using gradient clipping by
3https://github.com/tticoin/LSTM-ER/. 4For the ACE05 dataset, the head region is defined by the corpus, and for the CONLL04 dataset, the head region covers the entire scope of an entity.
a max norm 10 and l2-regularization by a parameter 10−5. The dimension sizes of various vectors in neural network structure are shown in Table 2. All the hyper-parameters are tuned by development experiments. All experiments are conducted under gcc version 4.9.4 (Ubuntu 4.9.4- 2ubuntu1 14.04.1), on an Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz.
Online training is used to learn model parameters, traversing over the entire training examples by 300 iterations. We select the best iteration model according to the development results. In particular, we exploit pre-training techniques to learn better model parameters (Wiseman and Rush, 2016). For the local model, we follow Miwa and Bansal (2016), training parameters only for entity detection during the first 20 iterations. For the global model, we pretrain our model using local optimization for 40 iterations, before conducting beam global optimization.
3.3 Development Experiments
We conduct several development experiments on the ACE05 development dataset.
3.3.1 Feature Ablation Tests
We consider the baseline system with no syntactic features using local training. Compared with Miwa and Bansal (2016), we introduce characterlevel features, and in addition exploit segmental features for entity detection. Feature ablation experiments are conducted for the two types of features. As shown in Table 3, without character-level features, the F-scores of entity detection and relation classification decrease 0.6% and 0.7%, respectively. Without segmental features for entity
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
655
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model Beam F1 Speed Local 1 50.9 95.6
Local(+SS) 1 51.2 95.1
Global 1 51.4 95.3 3 51.8 52.0 5 52.6 36.9
Table 4: Comparisons between local and global models, where SS denotes scheduled sampling, and speed is measured by the number of sentences per second.
detection, the baseline model loses 1.3% in entity detection, which results in an error propagation of 1.1% for relation classification. The results demonstrate that the two types of new features we use are useful for relation extraction.
3.3.2 Local v.s. Global
We study the influence of training strategies for the relation extraction model without using syntactic features. For the local model, we apply the scheduled sampling strategy (Bengio et al., 2015), which has been shown to improve the performances by Miwa and Bansal (2016).
Table 4 shows the relation F1 scores. Scheduled sampling achieves improved F-measure scores for the local model. With the same greedy search strategy, the globally normalized model gives slightly better results than the local model with scheduled sampling. The performance of the global model increases with a larger beam size. However, the decoding speed becomes intolerably slow when the beam size increases beyond 5. Thus we exploit a beam size of 5 for global training considering both performance and efficiency.
3.3.3 Syntactic Features
We examine the effectiveness of the proposed syntactic features. Table 5 shows the developmental results using both local and global optimization. The proposed features improve the relation performances significantly under both settings, where the p-values are below 10−4 by using pairwise ttest, demonstrating that our use of syntactic features is highly effective for relation extraction.
3.4 Final Results
Table 6 shows the final results on the test datasets of ACE05 and CONLL04. We show several topperforming systems in the table as well, where M&B (2016) refers to Miwa and Bansal (2016), who exploit end-to-end LSTM neural networks
with local optimization, and L&J (2014) and M&S (2014) refer to Li and Ji (2014) and Miwa and Sasaki (2014), respectively, which are both globally optimized models using discrete features, giving the top F-scores among statistical models.5
Overall, neural models give better performances than statistical models, and global optimization can give improved performances as well. Our final model achieves the best performances on both datasets. Compared with the best reported results, our model gives improvements of 1.9% on ACE05, and 6.8% on CONLL04.
3.5 Analysis
We conduct analysis on the ACE05 test dataset in order to understand our models in depth. We focus on two major contributions by our model, first examining the influences of global optimization, and then studying the gains by using the proposed syntactic features.
Global optimization aims to find the best label sequences, rather than the best label locally at each step. Thus intuitively global optimization should give better accuracies at the sentence level. We verify this by examining the sentence-level accuracies, where one sentence is regarded as correct when all the labels in the resulted table are correct. Figure 6 shows the result, which is consistent with our intuition. On the one hand, the sentence-
5Gupta et al. (2016) proposed a locally optimized model but used a different test dataset from CONLL04 and a different evaluation method, reporting entity and relation F-scores of 93.6% and 72.1%, respectively. Their results are not directly comparable to the results in Table 6. In particular, they regard an entity as correct if at least one token is tagged correctly, which influences the results significantly since multiword entities accounts for over 50% of all entities.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
5 10 15 20 25 ≥30 20
40
60
80 100 A cc ur ac y( % )
global local
Figure 6: Sentence-level accuracies with respect to sentence length.
1 2 3 4 5 ≥6 30
40
50
60
70
Fsc
or e(
% )
+syn -syn
Figure 7: F-scores with respect to the distance between entity pairs.
level accuracies of the globally normalized model are consistently better than the local model. On the other hand, the accuracy decreases sharply as the sentence length increases, with the local model suffering more severely from larger sentences.
To understand the effectiveness of the proposed syntactic features, we examine the relation Fscores with respect to entity distances. Miwa and Bansal (2016) exploit the shortest dependency path, which can make the distance between two entities closer compared with their sequential distance, thus facilitating relation extraction. We verify whether the proposed syntactic features can benefit our model similarly. As shown in Figure 7, the F-scores of entity-pairs with large distances see apparent improvements, demonstrating that our use of syntactic features has a similar effect compared to the shortest dependency path.
4 Related Work
Entity recognition (Florian et al., 2004, 2006; Ratinov and Roth, 2009; Florian et al., 2010; Kuru et al., 2016) and relation extraction (Zhao and Grishman, 2005; Jiang and Zhai, 2007; Zhou et al., 2007; Qian and Zhou, 2010; Chan and Roth, 2010; Sun et al., 2011; Plank and Moschitti, 2013; Verga et al., 2016) have received much attention in the NLP community. The dominant methods treat the two tasks separately, where relation extraction is performed assuming that entity boundaries have been given (Zelenko et al., 2003; Miwa et al.,
2009; Chan and Roth, 2011; Lin et al., 2016). Several studies find that extracting entities and relations jointly can benefit both tasks. Early work conducts joint inference for separate models (Ji and Grishman, 2005; Roth and Yih, 2004, 2007). Recent work shows that joint learning and decoding with a single model brings more benefits for the two tasks (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016; Gupta et al., 2016), and we follow this line of work in the study.
LSTM features have been extensively exploited for NLP tasks, including tagging (Huang et al., 2015; Lample et al., 2016), parsing (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016), relation classification (Xu et al., 2015; Vu et al., 2016; Miwa and Bansal, 2016) and sentiment analysis (Li et al., 2015; Teng et al., 2016). Based on the output of LSTM structures, Wang and Chang (2016) introduce segmental features, and apply it to dependency parsing. The same method is applied to constituent parsing by Cross and Huang (2016). We exploit this segmental representation for relation extraction.
Global optimization and normalization has been successfully applied on many NLP tasks that involve structural prediction (Lafferty et al., 2001; Collins, 2002; McDonald et al., 2010; Zhang and Clark, 2011), using traditional discrete features. For neural models, it has recently received increasing interests (Zhou et al., 2015; Andor et al., 2016; Xu, 2016; Wiseman and Rush, 2016), and improved performances can be achieved with global optimization accompanied by beam search. Our work is in line with these efforts. To our knowledge, we are the first to apply globally optimized neural models for end-to-end relation extraction, achieving the best results on standard benchmarks.
5 Conclusion
We proposed a novel relation extraction model using neural network, based on the table-filling framework proposed by Miwa and Sasaki (2014). Feature representations are learned from several LSTM structures over the inputs, and a simple method is used to integrate syntactic information into our model without the need of parser outputs. In addition, global optimization is taken to make use of structural information more effectively. Compared with previous work, our final model achieved the best performances on two benchmark datasets.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:   - The paper is clearly written and well-structured.    - The system newly applied several techniques including global optimization to end-to-end neural relation extraction, and the direct incorporation of the parser representation is interesting.   - The proposed system has achieved the state-of-the-art performance on both ACE05 and CONLL04 data sets.   - The authors include several analyses.  - Weaknesses:   - The approach is incremental and seems like just a combination of existing methods.     - The improvements on the performance (1.2 percent points on dev) are relatively small, and no significance test results are provided.  - General Discussion:  - Major comments:   - The model employed a recent parser and glove word embeddings. How did they affect the relation extraction performance?   - In prediction, how did the authors deal with illegal predictions?  - Minor comments:   - Local optimization is not completely  local . It  considers structural correspondences between incremental decisions,  so this explanation in the introduction is misleading.   - Points in Figures 6 and 7 should be connected with straight lines, not curves.   - How are entities represented in  -segment ?   - Some citations are incomplete. Kingma et al. (2014) is accepted to ICLR, and Li et al. (2014) misses pages.


Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics
The existing word representation methods mostly limit their information source to word co-occurrence statistics. In this paper, we introduce ngrams into four representation methods: SGNS, GloVe, PPMI matrix and its SVD factorization. Comprehensive experiments are conducted on word analogy and similarity tasks. The results show that improved word representations are learned from ngram cooccurrence statistics. We also demonstrate that the trained ngram representations are useful in many aspects such as finding antonyms and collocations. Besides, a novel approach of building co-occurrence matrix is proposed to alleviate the hardware burden brought by ngrams.
1 Introduction
Recently, deep learning approaches have achieved state-of-the-art results on a range of NLP tasks. One of the most fundamental work in this field is word embedding, where low-dimensional word representations are learned from unlabeled corpus. The trained word embeddings reflect semantic and syntactic information of words. They are not only useful in tasks of lexical semantics (e.g. finding similar words), but also widely used as the input of the downstream tasks, such as text classification (Kim, 2014) and tagging (Collobert et al., 2011; Pennington et al., 2014).
Word2vec is one of the most popular word embedding models (Mikolov et al., 2013b,a). It is trained upon <word, context> pairs in the local window. Word2vec gains the reputation by its amazing effectiveness and efficiency. It achieves state-of-the-art results on a range of linguistic tasks with only a fraction of time compared with
previous techniques. A challenger of word2vec is GloVe (Pennington et al., 2014). Instead of training on <word, context> pairs, GloVe directly uses word co-occurrence matrix. They claim that the change brings the improvement over word2vec on both accuracy and speed. Levy and Goldberg (2014b) further reveal that the attractive properties observed in word embeddings are not restricted to neural models. They use traditional bag-ofcontexts (concretely, PPMI matrix) to represent words, and achieve comparable results with the above neural embedding models.
The relationships among different representation methods are intricate. A preliminary conclusion is obtained in (Levy et al., 2015), which states that none of them consistently outperform the other methods. One should not feel surprised with the conclusion, because these methods all exploit word co-occurrence statistics as the information source and no one goes beyond that. To learn improved word representations, we extend the information source from co-occurrence of ‘wordword’ type to co-occurrence of ‘ngram-ngram’ type. The idea of using ngrams is well supported by language modeling, one of the oldest problems studied in statistical NLP. In language models, co-occurrence of words and ngrams is used to predict the next word (Kneser and Ney, 1995; Katz, 1987). Actually, the idea of word embedding models roots in language models. They are closely related but with different purposes. Word embedding models aim at learning word representations instead of word prediction. Since ngram is a vital part in language modeling, we are inspired to integrate ngram statistical information into recent word representation methods for better performance.
The idea of using ngram is intuitive. However, there is still rare work using ngrams in recent representation methods. In this paper, we in-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
troduce ngrams into SGNS, GloVe, PPMI and its SVD factorization. To evaluate the ngram-based models, comprehensive experiments are conducted on word analogy and similarity tasks. The results demonstrate that the improved word representations are learned from ngram co-occurrence statistics. Besides that, we qualitatively evaluate the trained ngram representations. We show that they are able to reflect ngrams’ meanings and syntactic patterns (e.g. ‘be + past participle’ pattern). The high-quality ngram representations are useful in many ways. For example, ngrams in negative form (e.g. ‘not interesting’) can be used for finding antonyms (e.g. ‘boring’).
Finally, a novel method is proposed to build ngram co-occurrence matrix. Our method reduces the disk I/O as much as possible, largely alleviating the costs brought by ngrams. We unify different representation methods in a pipeline. The source code is organized as ngram2vec toolkit and released at http://github.com/.
2 Related Work
SGNS, GloVe, PPMI and its SVD factorization are used as baselines. The information used by them does not go beyond word co-occurrence statistics. However, their approaches to using the information are totally different. We review these methods in the following 3 sections. In section 2.4, we revisit the use of ngram in deep learning context.
2.1 SGNS
Skip-gram with negative sampling (SGNS) is a model in the word2vec toolkit (Mikolov et al., 2013b,a). Its training procedure follows the majority of neural embedding models (Bengio et al., 2003): (1) Scan the corpus and use <word, context> pairs in the local window as training samples. (2) Train the models to make words useful for predicting contexts (or in reverse). The details of SGNS is discussed in Section 3.1. Compared to previous neural embedding models, SGNS speeds up the training process, reducing the training time from days or weeks to hours. Also, the trained embeddings possess valuable properties. They are able to reflect relations between the two words accurately, which is evaluated by a fancy task called word analogy.
Due to the above advantages, many models are proposed on the basis of word2vec. For example, Faruqui et al. (2015) introduce knowledge in lexi-
cal resources into the word2vec. Zhao et al. (2016) extend the contexts from the local window to the entire texts. Li et al. (2015) use supervised information to guide the training. Dependency parsetree is used for defining context in (Levy and Goldberg, 2014a). Sub-word information is considered in (Sun et al., 2016; Soricut and Och, 2015).
2.2 GloVe
Different from typical neural embedding models which are trained on <word, context> pairs, GloVe learns word representation on the basis of co-occurrence matrix (Pennington et al., 2014). GloVe breaks traditional ‘words predict contexts’ paradigm. Its objective is to reconstruct non-zero values in the matrix. The direct use of matrix is reported to bring more improved results and higher speed. However, There is still dispute about the advantages of GloVe over word2vec (Levy et al., 2015; Schnabel et al., 2015). GloVe and other embedding models are essentially based on word cooccurrence statistics of the corpus. The <word, context> pairs and matrix can be converted to each other. Suzuki and Nagata (2015) try to unify GloVe and SGNS in one framework.
2.3 PPMI & SVD
When we are satisfied with the embeddings’ fancy properties, a natural question is raised: where the properties come from. One conjecture is that it’s due to the neural networks. However, Levy and Goldberg (2014c) reveal that SGNS is just factoring PMI matrix implicitly. The experiments also confirm this idea. Levy and Goldberg (2014b) show that positive PMI (PPMI) matrix still rivals the newly proposed embedding models on a range of linguistic tasks. Properties like word analogy are not restricted to neural models. At last, we factorize PPMI matrix via SVD factorization, a classic method for generating low-dimensional vectors from sparse matrix (Deerwester et al., 1990).
2.4 Ngram in Deep Learning
In the deep learning literature, ngram has shown to be useful in generating text representations. Recently, convolutional neural networks (CNNs) are reported to perform well on a range of NLP tasks (Blunsom et al., 2014; Hu et al., 2014; Severyn and Moschitti, 2015). CNNs are essentially using ngram information to represent texts. They use 1-D convolutional layers to extract ngram features and the distinct features are selected by max-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
pooling layers. In (Li et al., 2016), ngram embedding is introduced into Paragraph Vector model, where text embedding is trained to be useful to predict ngrams in the text. In the word embedding literature, a work that is related to ngram is from (Mikolov et al., 2013b), where phrases are embedded into vectors. It should be noted that phrases are different from ngrams. Phrases have clear semantics and the number of phrases is much less than the number of ngrams. Using phrase embedding has little impact on word embedding’s quality.
3 Model
In this section, we introduce ngrams into SGNS, GloVe, PPMI and SVD. Section 3.1 reviews the SGNS. Section 3.2 and 3.3 show the details of introducing ngrams into SGNS. In section 3.4, we show the way of using ngrams in GloVe, PPMI and SVD, and propose a novel way of building ngram co-occurrence matrix.
3.1 Word Predicts Word: the Revisit of SGNS
First we establish some notations. The raw input is the corpus T = {w1,w2,......,w|T |}. W and C denote the word and context vocabulary. θ is the parameters to be optimized. SGNS’s parameters involve two parts: word embedding matrix and context embedding matrix. With embedding ~w ∈ Rd, the total number of parameters is (|W|+|C|)*d.
The SGNS’s objective is to maximize the conditional probabilities of contexts given center words:
|T |∑ t=1 [ ∑ c∈C(wt) log p(c|wt; θ) ] (1)
where C(wt) = {wi, t− win ≤ i ≤ t+ win and i 6= t} and win denotes the window size. As illustrated in figure 1, the center word ‘written’ predicts its surrounding words ‘Potter’, ‘is’, ‘by’ and ‘J.K.’. In this paper, negative sampling (Mikolov et al., 2013b) is used to approximate the conditional probability:
p(c|w) = σ(~wT~c) k∏
j=1
E cj∼Pn(C)
σ(−~wT ~cj) (2)
where σ is sigmoid function. k samples (from c1 to ck) are drawn from context distribution raised to the n power.
Figure 1: Illustration of ‘word predicts word’.
3.2 Word Predicts Ngram
In this section, we introduce ngrams into context vocabulary. We treat each ngram as a normal word and give it a unique embedding. During the training, the center word should not only predict its surrounding words, but also predict its surrounding ngrams. As shown in figure 2, center word ‘written’ predicts the bi-grams in the local window such as ‘by J.K.’. The objective of ‘word predicts ngram’ is similar with original SGNS. The only difference is the definition of the C(w). In ngram case, C(w) is formally defined as follows:
C(wt) = N⋃
n=1
{wi:i+n|wi:i+n is not wt AND
t− win ≤ i ≤ t+ win− n+ 1} (3)
where wi:i+n denotes the ngram wiwi+1...wi+n−1 and N is the order of context ngram. Two points need to be noticed from the above definition. The first is how to determine the distance between word and ngram. In this paper, we use the distance between word and ngram’s far-end word. As show in figure 2, the distance between ‘written’ and ‘Harry Potter’ is 3. As a result, ‘Harry Potter’ is not included in the center word’s context. This distance definition ensures that the ngram models don’t use the information beyond the pre-specified window, which guarantees the fair comparisons with the baselines. Another point is whether the overlap of word and ngram is allowed or not. In the overlap situation, ngrams are used as context even they contain the center word. As the example in figure 2 shows, ngram ‘is written’ and ‘written by’ are predicted by the center word ‘written’. In the non-overlap case, these ngrams are excluded. The properties of word embeddings are different when overlap is allowed or not, which is discussed in experiments section.
3.3 Ngram Predicts Ngram
We further extend the model to introduce ngrams into center word vocabulary. During the training, center ngrams (including words) predict their
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 2: Illustration of ‘word predicts ngram’.
Figure 3: Illustration of ‘ngram predicts ngram’.
surrounding ngrams. As shown in figure 3, center bi-gram ‘is written’ predicts its surrounding words and bi-grams. The objective of ‘ngram predicts ngram’ is as follows:
|T |∑ t=1 Nw∑ nw=1 [ ∑ c∈C(wt:t+nw ) log p(c|wt:t+nw ; θ) ] (4)
where Nw is the order of center ngram. The definition of C(wt:t+nw) is as follows:
Nc⋃ nc=1 {wi:i+nc |wi:i+nc is not wt:t+nw AND
t− win+ nw − 1 ≤ i ≤ t+ win− nc + 1} (5)
where Nc is the order of context ngram. To this end, the word embeddings are not only affected by the ngrams in the context, but also indirectly affected by co-occurrence statistics of ‘ngramngram’ type in the corpus.
SGNS is proven to be equivalent with factorizing pointwise mutual information (PMI) matrix (Levy and Goldberg, 2014c). Following their work, we can easily show that models in section 3.2 and 3.3 are implicitly factoring PMI of ‘wordngram’ and ‘ngram-ngram’ type. They are just the representation method discussed in the following section, where ngrams are introduced into positive PMI (PPMI) matrix.
3.4 Co-occurrence Matrix Construction
Introducing ngrams into GloVe, PPMI and SVD is straightforward: the only change is to replace word co-occurrence matrices with ngram ones. In the above three sections, we have discussed the
way of taking out <ngram, context> pairs from the corpus. Afterwards, we build co-occurrence matrix upon these pairs. The rest steps are identical with the original baseline models.
However, building co-occurrence matrix is not an easy task as it apparently looks like. The introduction of ngrams brings huge burdens on the hardware. The matrix construction cost is closely related to the number of pairs (#Pairs). Table 1 shows the detailed statistics of corpus wiki2010 1. We can observe that #Pairs is huge when bi-grams are considered.
To speed up the process of building ngram co-occurrence matrix, we take advantages of ‘mixture’ strategy (Pennington et al., 2014) and ‘stripes’ strategy (Dyer et al., 2008; Lin, 2008). The two strategies optimize the process in different aspects. Computational cost is reduced significantly when they are used together.
When the words (or ngrams) are sorted in descending order by frequency, the matrix’s topleft corner is dense while the rest part is sparse. Based on this observation, the ‘mixture’ of two data structures are used for storing matrix. Elements in the top-left corner are stored in a 2D array, which stays in memory. The rest of the elements are stored in the form of <ngram, H> (‘stripes’ strategy), where H<context, count> is an associative array recording the times the ngram and context co-occurs. Compared with storing <ngram, context> pairs explicitly, the ‘stripes’ strategy provides more opportunities to aggregate pairs outside of the top-left corner.
Algorithm 1 shows the way of using the ‘mixture’ and ‘stripes’ strategies together. In the first
1http://nlp.stanford.edu/data/WestburyLab.wikicorp.201004.txt.bz2
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
stage, pairs are stored in different data structures according to topLeft function. Intermediate results are written to temporary files when memory is full. In the second stage, we merge these sorted temporary files to generate co-occurrence matrix. The getSmallest function takes out the pair <ngram, H> with the smallest key from temporary files. In practice, algorithm 1 is efficient. Instead of using computer clusters (Lin, 2008), we can build the matrix of ‘bi bi’ type even in a laptop. It only requires 12GB to store temporary files (win=2, subsampling=0, memory size=4GB), which is much smaller than the implementations in (Pennington et al., 2014; Levy et al., 2015) . More detailed analysis about these strategies can be found in the ngram2vec toolkit and the attachment along with our paper 2.
Algorithm 1: An algorithm for building ngram co-occurrence matrix
Input : Pairs P , Sorted vocabulary V Output: Sorted and aggregated pairs
1 The 2D array A[ ][ ]; 2 The dictionary D < ngram,H >; 3 The temporary files array tfs[ ]; fid=1; 4 for pair p < n, c > in P do 5 if topLeft(n, c) == 1 then 6 A[getId(n)][getId(c)] += 1; 7 else 8 D{n}{c} += 1; 9 if Memory is full or P is empty then
10 Sort D by key (ngram); 11 Write D to tfs[fid]; 12 fid += 1; 13 end 14 end 15 end 16 Write A to tfs[0] in the form of < ngram,H >; 17 old = getSmallest(tfs) ; 18 while !(All files in tfs are empty) do 19 new = getSmallest(tfs) ; 20 if old.ngram == new.ngram then 21 old = < old.ngram,merge(old.H, new.H) >; 22 else 23 Write old to disk; 24 old = new 25 end 26 end
4 Experiments
4.1 Datasets
The datasets used in this paper is the same with (Levy et al., 2015), including six similarity and two analogy datasets. In similarity task, a scalar
2We upload the implementation of algorithm 1 along with the paper
(e.g. a score from 0 to 10) is used to measure the relation between the two words. The scalar only reflects the strength of the relation, not the type. For example, the ‘train, car’ pair is given the score of 6.31. In our view, relations are too complex to be reflected by a single score. Suppose that when one is asked to describe the relations between ‘Athens’ and ‘Greece’, rather than giving a score, a reasonable answer is probably ‘they are both place names and Athens is the capital of Greece’. Besides that, we have to compare pairs with different types in similarity task. Considering the two pairs: ‘train, car’ and ‘movie, star’. It is hard to say which pair has closer relations (higher score) (Schnabel et al., 2015).
In analogy task, relations between the two words are reflected by a vector, which is usually obtained by the difference between word embeddings. Different from scalar, the vector provides more accurate descriptions of relations. For example, capital-country relation is encoded in vec(Athens)-vec(Greece), vec(Tokyo)vec(Japan) and so on. More concretely, the questions in the analogy task are in the form of ‘a is to b as c is to d’. ‘d’ is an unknown word in the test phase. To correctly answer the questions, the models should embed the two relations, vec(a)vec(b) and vec(c)-vec(d), into similar positions in the space. Following (Levy and Goldberg, 2014b), both additive (add) and multiplicative (mul) functions are used for finding word ‘d’. The latter one is more suitable for sparse representation.
4.2 Pipeline and Hyper-parameter Setting
We implement SGNS, GloVe, PPMI and SVD in a pipeline, allowing the reuse of code and intermediate results. Figure 4 illustrates the overview of the pipeline. Firstly, <ngram, context> pairs are extracted from the corpus as the input of SGNS. Afterwards, we build the co-occurrence ma-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Win Type Google Tot. / Sem. / Syn. MSRAdd Mul Add Mul
2
uni uni .488 / .425 / .540 .513 / .457 / .559 .477 .498 overlap uni bi .579 / .640 / .528 .610 / .670 / .560 .465 .496bi bi .645 / .729 / .576 .665 / .739 / .603 .527 .550 non-overlap uni bi .589 / .606 / .575 .610 / .625 / .598 .469 .505bi bi .651 / .708 / .604 .682 / .725 / .646 .525 .557
5
uni uni .587 / .591 / .585 .607 / .611 / .604 .445 .463 overlap uni bi .635 / .697 / .583 .638 / .696 / .590 .448 .420bi bi .649 / .711 / .598 .660 / .714 / .616 .463 .480 non-overlap uni bi .653 / .666 / .643 .670 / .676 / .665 .491 .516bi bi .661 / .687 / .639 .671 / .690 / .656 .512 .536
Table 2: Performance of (ngram) SGNS on analogy datasets.
Win Type Sim. Rel. Bruni Radinsky Luong Hill
2 uni uni .707 .565 .696 .585 .450 .405 uni bi .714 .568 .688 .616 .471 .428 bi bi .735 .594 .698 .644 .468 .413 5 uni uni .739 .637 .771 .651 .457 .396 uni bi .763 .624 .773 .663 .471 .416 bi bi .778 .647 .767 .664 .473 .413
Table 3: Performance of (ngram) SGNS on similarity datasets.
trix upon the pairs. GloVe and PPMI learn word representations on the basis of co-occurrence matrix. SVD factorizes the PPMI matrix to obtain low-dimensional representation.
Most hyper-parameters come from ‘corpus2pairs’ part and four representation models. ‘corpus2pairs’ part determines the source of information for the subsequent models. We basically follow the default setting in (Levy et al., 2015): low-frequency words (ngrams) are removed with a threshold of 100. High-frequency words (ngrams) are removed with sub-sampling at the degree of 1e-5 3. Window size is set to 2 and 5. Clean strategy (Levy et al., 2015) is used to ensure no information beyond pre-specified window is included. Non-overlap setting is used in default since it produces much less pairs. For hyper-parameters in the models, we use the embeddings of 300 dimensions. The rest strictly follow the baseline models 4.
We only consider uni-grams (words) and bigrams in our experiments. Bi-grams are introduced into context (uni bi) or both context and center ‘word’ (bi bi). We suspect that the higherorder grams are so sparse that may deteriorate the performance. The implementation of higher-order
3Sub-sampling is not used in GloVe, which follows its original setting.
4http://bitbucket.org/omerlevy/ hyperwords for SGNS, PPMI and SVD; http://nlp.stanford.edu/projects/glove/ for GloVe.
models and their results will be released with ngram2vec toolkit.
4.3 Ngrams on SGNS
SGNS is a highly popular word embedding model. Even compared with its challengers such as GloVe, SGNS is reported to have more robust performance with faster training speed. Table 2 lists the results on analogy datasets. We can observe that the introduction of bi-grams gives the huge promotions at different hyper-parameter settings. The SGNS of ‘bi bi’ type provides the highest results. Bi-grams are very effective for capturing semantic information. Around 20 and 10 percent improvements are witnessed on semantic questions at window sizes of 2 and 5. For syntactic questions, the improvements are still significant. Around 5 percent improvements are achieved on Google (syntactic) and MSR datasets. In traditional language models, ngram is the vital part for predicting next words. Our results confirm the effectiveness of ngrams again on recent word embedding models and more advanced analogy task.
The effect of overlap is large on analogy datasets. Semantic questions prefer the overlap setting. Around 3 percent increase is witnessed compared with non-overlap setting. While in syntactic case, non-overlap setting performs better by a margin of around 5 percent.
Table 3 illustrates the SGNS’s performance on similarity task. The use of bi-grams is effective on
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Win Type Google MSR Sim. Rel. Bruni Radinsky Luong HillAdd Mul Add Mul
2 uni uni .488 / .571 / .418 .626 / .739 / .533 .276 .441 .681 .559 .670 .621 .404 .366uni bi .464 / .580 / .367 .733 / .833 / .649 .286 .553 .673 .532 .681 .635 .401 .377 5 uni uni .508 / .635 / .403 .588 / .757 / .447 .238 .332 .732 .680 .722 .626 .421 .329uni bi .509 / .682 / .365 .650 / .819 / .508 .239 .416 .750 .695 .708 .658 .411 .371
Table 4: Performance of (ngram) PPMI on analogy and similarity datasets.
Win Type Google MSR Sim. Rel. Bruni Radinsky Luong HillAdd Mul Add Mul
2 uni uni .535 / .599 / .482 .540 / .610 / .481 .444 .445 .681 .529 .698 .608 .381 .351uni bi .543 / .601 / .493 .549 / .612 / .496 .464 .472 .686 .545 .695 .631 .389 .352 5 uni uni .625 / .689 / .572 .626 / .696 / .568 .476 .490 .747 .600 .735 .657 .389 .347uni bi .631 / .699 / .575 .633 / .703 / .574 .477 .504 .752 .610 .737 .631 .395 .342
Table 5: Performance of (ngram) GloVe on analogy and similarity datasets.
Win Type Google MSR Sim. Rel. Bruni Radinsky Luong HillAdd Mul Add Mul
2 uni uni .429 / .385 / .465 .448 / .392 / .495 .345 .375 .729 .611 .732 .637 .499 .369uni bi .410 / .338 / .471 .446 / .364 / .514 .370 .411 .731 .608 .721 .637 .513 .378 5 uni uni .436 / .415 / .454 .461 / .438 / .481 .328 .332 .744 .629 .756 .634 .501 .337uni bi .486 / .425 / .536 .514 / .460 / .559 .349 .391 .761 .642 .751 .626 .522 .357
Table 6: Performance of (ngram) SVD on analogy and similarity datasets.
most cases. However, the improvements brought by bi-grams are not as big as the case in analogy task. The SGNS of ‘uni bi’ type outperforms the ‘bi bi’ type on most datasets.
4.4 Ngrams on GloVe, PPMI, SVD
In this section, we only report the results of models of ‘uni uni’ and ‘uni bi’ types. Using ngram-ngram co-occurrence statistics bring little improvements but immense costs. Levy and Goldberg (2014b) demonstrate that sparse representation (PPMI matrix) can still achieve competitive results on many linguistic tasks, challenging the dominance of neural embedding models. Table 4 lists the results of PPMI. PPMI prefers Multiplicative (Mul) evalution. To this end, we focus on analyzing the results on Mul columns. When bi-grams are used, significant improvements are witnessed on analogy task. On Google dataset, bi-grams bring around 8 percent increase on the total accuracies. At window size 2, the accuracies even reach 73.3/83.3/64.9. They are pretty high numbers considering only default setting is used. On MSR dataset, around 10 percent improvements are achieved. Though bi-grams are very effective on analogy task, the improvements on similarity task are not witnessed. The PPMI of ‘uni bi’ type improves the results of 3 in 6 datasets.
Table 5 and 6 list the GloVe and SVD’s results. In GloVe, consistent but minor improvements are achieved on analogy task. Little improvement is witnessed on similarity task. In SVD, bi-grams
sometimes lead to worse results in both analogy and similarity tasks. That’s not what we expected before the experiments. We thought ngram would be very effective in GloVe and SVD just like the cases in SGNS, since these models use exactly the same source of information. Our preliminary conjecture is that the default hyper-parameter setting should be blamed. We strictly follow the hyper-parameters used in baseline models, making no adjustments to cater to the introduction of ngrams. Besides that, some common techniques such as dynamic window, decreasing weighting function, dirty sub-sampling are discarded. The relationships between ngrams and various hyperparameters require further exploration. Though trivial, it may lead to much better results and give researchers better understanding of different representation methods. That will be the focus of our future work.
4.5 Qualitative Evaluations of Ngram Embedding
In this section, we analyze the properties of ngram embeddings trained on SGNS of ‘bi bi’ type. Ideally, the trained ngram embeddings should reflect ngrams’ semantic meanings. For example, the vec(wasn’t able) should be closed to vec(unable). The vec(is written) should be closed to vec(write) and vec(book). Also, the trained ngram embeddings should preserve the ngrams’ syntactic patterns. For example, ‘was written’ is in the form of ‘be + past participle’ and the nearest neighbors
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Pattern Target Word Bi-gram
Negative Form wasn’t able unable(.745), couldn’t(.723), didn’t(.680) was unable(.832), didn’t manage(.799), never managed(.786) don’t need don’t(.773), dont(.751), needn’t(.715) dont need(.790), don’t have(.785), dont want(.769) not enough enough(.708), insufficient(.701), sufficient(.629) not sufficient(.750), wasn’t enough(.729), was insufficient(0.685) Adj. Modifier heavy rain torrential(.844), downpours(.780), rain(.766) torrential rain(.829), heavy rainfall(.799), heavy snow(.799)
strong supporter supporter(.828), proponent(.733), admirer(.602) staunch supporter(.870), vocal supporter(.810), a supporter(.808) high quality high-quality(.867), quality(.744), inexpensive(.622) good quality(.813), top quality(.751), superior quality(.736)
Passive Voice was written written(.793), penned(.675), co-written(.629) were written(.785), is written(.744), written by(.739)
was sent sent(.844), dispatched(.661), went(.630) then sent(.779), later sent(.776), was dispatched(.774) was pulled pulled(.730), yanked(.629), limped(.593) were pulled(.706), pulled from(.691), was ripped(.682)
Perfect Tense has achieved achieved(.683), achieves(.680), achieving(.625) has attained(.775), has enjoyed(.741), has gained(.733) has impacted interconnectedness(.679), pervade(.676) have impacted(.838), is affecting(.773), have shaped(.772) has published authored(.722), publishes(.705), coauthored(.791) has authored(.852), has edited(.795), has written(.791) Phrasal Verb give off exude(.796), fluoresce(.789), emit(.754) gave off(.837), giving off(.820), and emit(.816)
make up comprise(.726), constitute(.616), make(.541) makes up(.705), making up(.702), comprise the(.672) picked up picked(.870), snagged(.544), scooped(.538) later picked(.712), and picked(.682), then picked(.681)
Common Sense highest mountain muztagh(.669), prokletije(.664), cadair(.658) highest peak(.873), tallest mountain(.857), highest summit(.830) avian influenza h5n1(.870), zoonotic(.812), adenovirus(.806) avian flu(.885), the h5n1(.870), flu virus(.868) computer vision human-computer(.789), holography(.767) image processing(.850), object recognition(.818)
Table 7: Target bi-grams and their nearest neighbours associated with similarity scores.
should possess similar patterns, such as ‘is written’ and ‘was transcribed’.
Table 7 lists the target ngrams and their top nearest neighbours. We divide the target ngrams into six groups according to their patterns. We can observe that the returned words and ngrams are very intuitive. As might be expected, synonyms of the target ngrams are returned in top positions (e.g. ‘give off’ and ‘emit’; ‘heavy rain’ and ‘downpours’). Besides that, from the results of the first group, it can be observed that bi-gram in negative form ‘not X’ is useful for finding the antonym of word ‘X’. The trained n-gram embeddings also preserve some common sense. For example, the returned results of ‘highest mountain’ is a list of mountain names (with a few exceptions such as ‘unclimbed’). In terms of syntactic patterns, we can observe that in most cases, the returned ngrams are in the similar form of target ngrams. In general, the trained embeddings basically reflect the semantic meanings and syntactic patterns of ngrams.
With high-quality ngram embeddings, we have the opportunity to do more interesting things in our future work. For example, we will construct a antonym dataset to evaluate ngram embeddings systematically. Besides that, we will find more scenarios for using ngram embeddings. In our view, ngram embeddings may be useful in many NLP tasks. For example, Johnson and Zhang (2015) use one-hot ngram representation as the input of CNN. Li et al. (2016) use the ngram embeddings to represent texts. Intuitively, initializing these models with pre-trained ngram embeddings may further
improve the accuracies.
5 Conclusion
We introduce ngrams into four representation methods. The experimental results demonstrate ngrams’ effectiveness for learning improved word representations. In addition, we find that the trained ngram embeddings are able to reflect their semantic meanings and syntactic patterns. To alleviate the costs brought by ngrams, we propose a novel way of building co-occurrence matrix, enabling the ngram-based models to run on cheap hardware.

This paper modifies existing word embedding algorithms (GloVe, Skip Gram, PPMI, SVD) to include ngram-ngram cooccurance statistics. To deal with the large computational costs of storing such expensive matrices, the authors propose an algorithm that uses two different strategies to collect counts.    - Strengths:  * The proposed work seems like a natural extension of existing work on learning word embeddings. By integrating bigram information, one can expect to capture richer syntactic and semantic information.  - Weaknesses:  * While the authors propose learning embeddings for bigrams (bi_bi case), they actually do not evaluate the embeddings for the learned bigrams except for the qualitative evaluation in Table 7. A more quantitative evaluation on paraphrasing or other related tasks that can include bigram representations could have been a good contribution.  * The evaluation and the results are not very convincing - the results do not show consistent trends, and some of the improvements are not necessarily statistically significant.  * The paper reads clunkily due to significant grammar and spelling errors, and needs a major editing pass.  - General Discussion:  This paper is an extension of standard embedding learning techniques to include information from bigram-bigram coocurance. While the work is interesting and a natural extension of existing work, the evaluation and methods leaves some open questions. Apart from the ones mentioned in the weaknesses, some minor questions for the authors :  * Why is there significant difference between the overlap and non-overlap cases? I would be more interested in finding out more than the quantitative difference shown on the tasks.  I have read the author response. I look forward to seeing the revised version of the paper.