End-to-End Neural Relation Extraction with Global Optimization
Neural networks have recently shown promising results for relation extraction. State-of-the-art models cast the task as an end-to-end problem, solved incrementally using a local classifier. Yet previous work using statistical models have demonstrated that global optimization can achieve better performances compared to local classification. We build a globally optimized neural model for end-to-end relation extraction, proposing novel LSTM features in order to better learn representations. Experiments show that our model is highly effective, achieving the best performances on two standard benchmarks.
Neural networks have recently shown promising results for relation extraction. State-of-the-art models cast the task as an end-to-end problem, solved incrementally using a local classifier. Yet previous work using statistical models have demonstrated that global optimization can achieve better performances compared to local classification. We build a globally optimized neural model for end-to-end relation extraction, proposing novel LSTM features in order to better learn representations. Experiments show that our model is highly effective, achieving the best performances on two standard benchmarks.
1 Introduction
Extracting entities (Florian et al., 2006, 2010) and relations (Zhao and Grishman, 2005; Jiang and Zhai, 2007; Sun et al., 2011; Plank and Moschitti, 2013) from unstructured texts have been two central tasks in information extraction (Grishman, 1997; Doddington et al., 2004). Traditional approaches to relation extraction take entity recognition as a predecessor step in a pipeline (Zelenko et al., 2003; Chan and Roth, 2011), predicting relations between given entities.
In recent years, there has been a surge of interest in performing end-to-end relation extraction, jointly recognizing entities and relations given free text inputs (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016; Gupta et al., 2016). End-to-end learning prevents error propagation in the pipeline approach, and allows cross-task dependencies to be modeled explicitly for entity recognition. As a result, it gives better relation extraction accuracies compared to pipelines.
Miwa and Bansal (2016) were among the first to use neural networks for end-to-end relation extraction, showing highly promising results. In particular, they used a bidirectional LSTM (Graves et al., 2013) to learn hidden word representations under a sentential context, and further leveraged a tree-structured LSTMs (Tai et al., 2015) to encode syntactic information, given the output of a parser. The resulting representations are then used for making local decisions for entity and relation extraction incrementally, leading to much improved results compared with the best statistical model (Li and Ji, 2014). This demonstrates the strength of neural representation learning for endto-end relation extraction.
On the other hand, Miwa and Bansal (2016)’s model is trained locally, without considering structural correspondences between incremental decisions. This is unlike existing statistical methods, which utilize well-studied structured prediction methods to address the problem (Li and Ji, 2014; Miwa and Sasaki, 2014). As has been commonly understood, learning local decisions for structured prediction can lead to label bias (Lafferty et al., 2001), which prevents globally optimal structures from receiving optimal scores by the model. We address this potential issue by building a structural neural model for end-to-end relation extraction, following a recent line of efforts on globally optimized models for neural structured prediction (Zhou et al., 2015; Watanabe and Sumita, 2015; Andor et al., 2016; Wiseman and Rush, 2016).
In particular, we follow Miwa and Bansal (2016), casting the task as an end-to-end tablefilling problem. This is different from the actionbased method of Li and Ji (2014), yet has shown to be more flexible and accurate (Miwa and Sasaki, 2014). We take a different approach to representation learning, addressing two potential limitations of Miwa and Bansal (2016).
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
First, Miwa and Bansal (2016) rely on external syntactic parsers for obtaining syntactic information, which is crucial for relation extraction (Culotta and Sorensen, 2004; Zhou et al., 2005; Bunescu and Mooney, 2005; Qian et al., 2008). However, parsing errors can lead to encoding inaccuracies of tree-LSTMs, thereby hurting relation extraction potentially. We take an alternative approach to integrating syntactic information, by taking the hidden LSTM layers of a bi-affine attention parser (Dozat and Manning, 2016) to augment input representations. Pretrained for parsing, such hidden layers contain rich syntactic information on each word, yet do not explicitly represent parsing decisions, thereby avoiding potential issues caused by incorrect parses.
Our method is also free from a particular syntactic formalism, such as dependency grammar, constituent grammar or CCG, requiring only hidden representations on word that contain syntactic information. In contrast, the method of Miwa and Bansal (2016) must consider tree LSTM formulations that are specific to grammar formalisms, which can be very different (Tai et al., 2015).
Second, Miwa and Bansal (2016) did not explicitly learn the representation of segments when predicting entity boundaries or making relation classification decisions, which can be intuitively highly useful. We take the LSTM-Minus method of Wang and Chang (2016), modelling a segment as the difference between its last and first LSTM hidden vectors. This method is highly efficient, yet gives as accurate results as compared to more complex neural network structures to model a span of words (Cross and Huang, 2016).
Evaluation on two benchmark datasets shows that our method outperforms previous methods of Miwa and Bansal (2016), Li and Ji (2014) and Miwa and Sasaki (2014), giving the best reported results on both benchmarks. Our code is available under GPL at https://github.com/〈anonymized〉.
2 Model
2.1 Task Definition
As shown in Figure 1, the goal of relation extraction is to mine relations from raw texts. It consists of two sub-tasks, namely entity detection, which recognizes valid entities, and relation classification, which determines the relation categories over entity pairs. We follow recent studies and recognize entities and relations as one single task.
2.2 Method
We follow Miwa and Sasaki (2014) and Gupta et al. (2016), treating relation extraction as a tablefilling problem, performing entity detection and relation classification using a single incremental model, which is similar in spirit to Miwa and Bansal (2016). Figure 2 shows an example of the table-filling process.
Formally, given a sentence w1w2 · · ·wn, we maintain a table Tn×n, where T (i, j) denotes the relation between wi and wj . When i = j, T (i, j) denotes an entity boundary label. We map entity words into labels under the BILOU (Begin, Inside, Last, Outside, Unit) scheme, assuming that there are no overlapping entities in one sentence (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016). Only the upper triangular table is necessary for indicating the relations.
We adopt the close-first left-to-right strategy (Miwa and Sasaki, 2014) to map the twodimensional table into a sequence, in order to fill the table incrementally. During the table-filling process, we take two label sets for entity detection (i = j) and relation classification (i < j), respectively. The labels for entity detection include {B*, I-*, L-*, O, U-* }, where * denotes the entity type, and the labels for relation classification are {−→∗ ,←−∗ }, where * denotes the relation category.
At each step, given a partially-filled table T , we determine the most suitable label l for the next step (order in Figure 2) using a scoring function:
score(T, l) = WlhT , (1)
where Wl is a model parameter and hT is the vector representation of T . Based on the function, we aim to find the best label sequence l1 · · · lm, where m = n(n+1)2 , and the resulting sequence of partially-filled tables is T0T1 · · ·Tm, where Ti = FILL(Ti−1, li), and T0 is an empty table. Different from previous work, we investigate a structural
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Associated Press writer Patrick McDowell in Kuwait City
Associated Press writer Patrick McDowell
in Kuwait
City
1 B-ORG 9 ⊥ 16 ⊥ 22 ⊥ 27 ⊥ 31 ⊥ 34 ⊥ 36 ⊥ 2 L-ORG 10←−−−−−ORG-AFF 17 ⊥ 23 ⊥ 28 ⊥ 32 ⊥ 35 ⊥
3 U-PER 11 ⊥ 18 ⊥ 24 ⊥ 29 ⊥ 33 ⊥ 4 B-PER 12 ⊥ 19 ⊥ 25 ⊥ 30 ⊥
5 L-PER 13 ⊥ 20 ⊥ 26−−−→PHYS 6 O 14 ⊥ 21 ⊥
7 B-GPE 15 ⊥ 8 L-GPE
Figure 2: Table-filling example, where numbers indicate the filling order.
hw
⊕ ewe′w et hchar ⊕ ⊕ CNN
character sequence
Figure 3: Word representations.
model that is optimized for the label sequence l1 · · · lm globally, rather than for each li locally.
2.3 Representation Learning
At the ith step, we determine the label li of the next table slot based on the current hypothesis Ti−1. Following Miwa and Bansal (2016), we use a neural network to learn the vector representation of Ti−1, and then use Equation 1 to rank candidate next labels. There are two types of input features, including the word sequence w1w2 · · ·wn, and the readily filled label sequence l1l2 · · · li−1. We build a neural network to represent Ti−1.
2.3.1 Word Representation
Shown in Figure 3, we represent each word wi by hwi using its word form, POS tag and characters. Based on the word form, we use two different embeddings, one being obtained by using a randomly initialized look-up table Ew, tuned during training and represented by ew, and the other being a pretrained external word embedding from E′w, which is fixed and represented by e′w.
1 For a POS tag t, its embedding et is obtained from a look-up table Et similar to Ew.
The above two components have also been used by Miwa and Bansal (2016). In addition, we enhance the word representation by its character sequence (Ballesteros et al., 2015; Lample et al., 2016), using a convolution neural network (CNN) to derive a character-based word representation hchar, which has been demonstrated effective for
1We use the set of pre-trained glove word embeddings available at http://nlp.stanford.edu/data/glove.6B.zip as external word embeddings.
hi−1...... hi ...... hj ......
hseg = hj − hi−1
Figure 4: Segment representation.
several NLP tasks (dos Santos and Gatti, 2014). We obtain the final hwi based on a non-linear feedforward layer on e′w ⊕ ew ⊕ et ⊕ hchar, where ⊕ denotes concatenation.
2.3.2 Label Representation
In addition to the word sequence, the history label sequence l1l2 · · · li−1, especially the labels representing detected entities, is also useful disambiguation. For example, the previous entity boundary label can be helpful to deciding the boundary label of the current word. During relation classification, the types of the entities involved can indicate the relation category between them. We exploit the diagonal label sequence of partial table T , which denotes entity boundaries of words, to enhance the representation learning. We obtain a word’s entity boundary label embedding el by a randomly initialized looking-up table El.
2.3.3 LSTM Features
We follow Miwa and Bansal (2016), learning global context representations using LSTMs. Three basic LSTM structures are used: a leftto-right word LSTM ( −−−−→ LSTMw), a right-to-left word LSTM ( ←−−−− LSTMw) and a left-to-right entity boundary label LSTM ( −−−−→ LSTMe). Each LSTM derives a sequence of hidden vectors for inputs. For example, for w1w2 · · ·wn, −−−−→ LSTMw gives hw,→1 h w,→ 2 · · ·h w,→ n .
Different from Miwa and Bansal (2016), who use the output hidden vectors {hi} of LSTMs to represent words, we exploit segment representations as well. In particular, for a segment of text [i, j], the representation is computed by using LSTM-Minus (Wang and Chang, 2016), shown by Figure 4, where hj − hi−1 in a left-to-right LSTM
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
...... ...... ......
...... ...... ......
...... ......
hT
−−−−→ LSTMw
←−−−− LSTMw
−−−−→ LSTMe
...... j − 1 j ...... i− 1 i i+ 1 ......
concatenate
feed-forward
(a) entity detection
...... ...... ...... ...... ......
...... ...... ...... ...... ......
...... ...... ...... ...... ......
hT
−−−−→ LSTMw
←−−−− LSTMw
−−−−→ LSTMe
left entityi middle entityj right
concatenate
feed-forward
(b) relation classification
Figure 5: Feature representation.
and hi − hj+1 in a right-to-left LSTM are used to represent the segment [i, j]. The segment representations can reflect entities in a sentence, and thus can be potentially useful for both entity detection and relation extraction.
2.3.4 Feature Representation
We use separate feature representations for entity detection and relation classification, both of which are induced according to the above three LSTM structures. In particular, we first extract a set of base neural features, and then concatenate them and feed them into a non-linear neural layer for entity detection and relation classification, respectively. Figure 5 shows the overall representation.
[Entity Detection] Figure 5(a) shows the feature representation method for the entity detection actions. First, we extract six feature vectors from the three basic LSTMs, three of which are word features, namely hw,→i , h w,← i and h e,→ i−1 , and the remaining are segment features, namely hw,→[j,i−1], hw,←[j,i−1] and h e,→ [j,i−1], where j denotes the start po-
sition of the previous entity.2 The six vectors are concatenated and then fed into a non-linear layer for entity detection.
[Relation Classification] Figure 5(b) shows the feature representation method for relation classification. Similar to entity detection, we extract a set of features from the basic LSTMs (
−−−−→ LSTMw,←−−−−
LSTMw and −−−−→ LSTMe), and then concatenate them for a non-linear classification layer. The differences between relation classification with entity detection lie in the range of hidden layers from LSTMs. For relation classification between i and j, we split each LSTM into five segments according to the two entities ended with i and j. Formally, let [s(i), i] and [s(j), j] denote the two entities above, where s(·) denotes the start position of an entity, the resulted segments are [0, s(i)− 1] (i.e., left, in Figure 5(b)), [s(i), i] (i.e., entityi), [i + 1, s(j) − 1] (i.e., middle), [s(j), j] (i.e., entityj) and [j + 1, n] (i.e., right), respectively. For the word LSTMs, we extract all five segment features, while the entity label LSTM, we only use the segment features of entityi and entityj .
2.3.5 Syntactic Features
Previous work has shown that syntactic features are useful for relation extraction (Zhou et al., 2005). For example, the shortest dependency path has been used by several relation extraction models (Bunescu and Mooney, 2005; Miwa and Bansal, 2016). Here we propose a novel method to integrate syntax, without need for prior knowledge on concrete syntactic structures.
In particular, many state-of-the-art syntactic parsers use encoder-decoder neural models (Buys and Blunsom, 2015; Kiperwasser and Goldberg, 2016), where the encoder represents the features of the input sentences. For example, LSTM over the input word/tag sequences has been used frequently (Kiperwasser and Goldberg, 2016). The decoder can also leverage partially-parsed results, such as features from partial syntactic trees. Table 1 shows the encoder structures of three state-ofthe-art dependency parsers.
Our method is to dump the encoder source representations of state-of-the-art parsers, and then use them directly as part of input embeddings in our proposed model. Denoting the dumped syntactic features on each word as hsyn1 h syn 2 · · ·h syn n , we feed them into a non-linear neural layer, 2The non-entity word is treated as a special unit entity to extract segmental features.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Models Encoder LAS S-LSTM (2015) 1-Layer LSTM 90.9
K&G (2016) 2-Layer Bi-LSTM 91.9 D&M (2016) 4-Layer Bi-LSTM 93.8
Table 1: The encoder structures and performances of three state-of-the-art dependency parsers, where S-LSTM (2015) refers to Dyer et al. (2015), K&G (2016) refers to the best parser of Kiperwasser and Goldberg (2016), D&M (2016) refers to Dozat and Manning (2016), and LAS (labeled attachment score) is the major evaluation metric for dependency parsing.
and then generate two LSTMs (bi-directional) based on the outputs, namely
−−−−→ LSTMsyn and←−−−− LSTMsyn, respectively, augmenting the original three LSTMs into five LSTMs. Features are extracted from the two new LSTMs in the same way as from the basic bi-directional word LSTMs.
We exploit the parser of Dozat and Manning (2016) to extract syntactic features, since it achieves the current best performance for dependency parsing. Our method can be easily generalized to the use of other parsers, which are potentially useful for our task as well. For example, we can use a constituent parser in the same way.
2.4 Training and Search
2.4.1 Local Optimization
Previous work (Miwa and Bansal, 2016; Gupta et al., 2016) trains model parameters by modeling each step for labeling one input sentence separately. Given a partial table T , we first obtain its neural representation hT , and then compute the next label scores {l1, l2, · · · , ls} using Equation 1. The output scores are regularized into a probability distribution {pl1 , pl2 , · · · , pls} by using a softmax layer. Our training objective is to minimize the cross-entropy loss between this output distribution with the gold-standard distribution:
loss(T, lgi ,Θ) = − log plgi , (2)
where lgi is the gold-standard next label for T , and Θ is the set of all model parameters. We refer this training method as local optimization, because it maximizes the score of the gold-standard label at each step locally.
During the decoding phase, the greedy search strategy is applied in consistence with the training. At each step, we find the highest-scored label
Algorithm 1 Beam-search. agenda← { (empty table, score=0.0) } for i in 1 · · ·max-step
next scored tables← { } for scored table in agenda
labels← NEXTLABELS(scored table) for next label in labels
new← FILL(scored table, next label) ADDITEM(next scored tables, new)
agenda← TOP-B(next scored tables, B)
based on the current partial table, before going on to the next step.
2.4.2 Global Optimization
We exploit the global optimization strategy of Andor et al. (2016), maximizing the cumulative score of the gold-standard label sequence for one sentence as a unit. Global optimization has achieved success for several NLP tasks under the neural setting (Zhou et al., 2015; Watanabe and Sumita, 2015). For relation extraction, global learning gives the best performances under the discrete setting (Li and Ji, 2014; Miwa and Sasaki, 2014). We study such models here under the neural setting.
Given a label sequence of l1l2 · · · li, the score of Ti is defined as follows:
score(Ti) = i∑
j=0
score(Tj−1, lj)
= score(Ti−1) + score(Ti−1, li),
(3)
where score(T0) = 0 and score(Ti−1, li) is computed by Equation 1. By this definition, we maximize the scores of all gold-standard partial tables.
Again cross-entropy loss is used to perform model updates. At each step i, the objective function is defined by:
loss(x, T gi ,Θ) = − log pT gi
= − log score(T gi )∑ T ′i score(T ′i ) ,
(4)
where x denotes the input sentence, T gi denotes the gold-standard state at step i, and T ′i are all partial tables that can be reached at step i.
The major challenge is to compute pT gi , because we cannot traverse all partial tables that are valid at step i, since their count increases exponentially by the step number. We follow Andor
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
et al. (2016), approximating the probability by using beam search and early-update.
Shown in Algorithm 1, we use standard beam search, using an agenda to maintain B highestscored partially-filled tables at each step. When each action of table filling is taken, all hypotheses in the agenda are expanded by enumerating the next labels, and the B highest-scored resulting tables are used to replace the agenda for the next step. Search begins with the agenda containing an empty table, and finishes when all cells of the tables in the agenda have been filled. When the beam size is 1, the algorithm is the same as greedy decoding. When the beam size is larger than 1, however, error propagation from greedy steps can be alleviated. For training, the same beam search algorithm is applied to training examples, and early-update (Collins and Roark, 2004) is used to fix search errors.
3 Experiments
3.1 Data and Evaluation
We evaluate the proposed model on two datasets, namely the ACE05 data and the corpus of Roth and Yih (2004) (CONLL04), respectively. The ACE05 data defines seven coarse-grained entity types and six coarse-grained relation categories, while the CONLL04 data defines four entity types and five relation categories.
For the ACE05 dataset, we follow Li and Ji (2014) and Miwa and Bansal (2016), splitting and preprocessing the dataset into training, development and test sets.3 For CONLL04, we follow Miwa and Sasaki (2014) to split and preprocess the dataset into training and test corpus, and divide 10% of the training corpus for development.
We use the micro F1-measure as the major metric to evaluate model performances, treating an entity as correct when its head region and type are both exactly matched,4 and regard a relation as correct when the argument entities and the relation category are all correct.
3.2 Parameter Tuning
We update all model parameters by back propagation using Adam (Kingma and Ba, 2014) with a learning rate 10−3, using gradient clipping by
3https://github.com/tticoin/LSTM-ER/. 4For the ACE05 dataset, the head region is defined by the corpus, and for the CONLL04 dataset, the head region covers the entire scope of an entity.
a max norm 10 and l2-regularization by a parameter 10−5. The dimension sizes of various vectors in neural network structure are shown in Table 2. All the hyper-parameters are tuned by development experiments. All experiments are conducted under gcc version 4.9.4 (Ubuntu 4.9.4- 2ubuntu1 14.04.1), on an Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz.
Online training is used to learn model parameters, traversing over the entire training examples by 300 iterations. We select the best iteration model according to the development results. In particular, we exploit pre-training techniques to learn better model parameters (Wiseman and Rush, 2016). For the local model, we follow Miwa and Bansal (2016), training parameters only for entity detection during the first 20 iterations. For the global model, we pretrain our model using local optimization for 40 iterations, before conducting beam global optimization.
3.3 Development Experiments
We conduct several development experiments on the ACE05 development dataset.
3.3.1 Feature Ablation Tests
We consider the baseline system with no syntactic features using local training. Compared with Miwa and Bansal (2016), we introduce characterlevel features, and in addition exploit segmental features for entity detection. Feature ablation experiments are conducted for the two types of features. As shown in Table 3, without character-level features, the F-scores of entity detection and relation classification decrease 0.6% and 0.7%, respectively. Without segmental features for entity
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
655
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Model Beam F1 Speed Local 1 50.9 95.6
Local(+SS) 1 51.2 95.1
Global 1 51.4 95.3 3 51.8 52.0 5 52.6 36.9
Table 4: Comparisons between local and global models, where SS denotes scheduled sampling, and speed is measured by the number of sentences per second.
detection, the baseline model loses 1.3% in entity detection, which results in an error propagation of 1.1% for relation classification. The results demonstrate that the two types of new features we use are useful for relation extraction.
3.3.2 Local v.s. Global
We study the influence of training strategies for the relation extraction model without using syntactic features. For the local model, we apply the scheduled sampling strategy (Bengio et al., 2015), which has been shown to improve the performances by Miwa and Bansal (2016).
Table 4 shows the relation F1 scores. Scheduled sampling achieves improved F-measure scores for the local model. With the same greedy search strategy, the globally normalized model gives slightly better results than the local model with scheduled sampling. The performance of the global model increases with a larger beam size. However, the decoding speed becomes intolerably slow when the beam size increases beyond 5. Thus we exploit a beam size of 5 for global training considering both performance and efficiency.
3.3.3 Syntactic Features
We examine the effectiveness of the proposed syntactic features. Table 5 shows the developmental results using both local and global optimization. The proposed features improve the relation performances significantly under both settings, where the p-values are below 10−4 by using pairwise ttest, demonstrating that our use of syntactic features is highly effective for relation extraction.
3.4 Final Results
Table 6 shows the final results on the test datasets of ACE05 and CONLL04. We show several topperforming systems in the table as well, where M&B (2016) refers to Miwa and Bansal (2016), who exploit end-to-end LSTM neural networks
with local optimization, and L&J (2014) and M&S (2014) refer to Li and Ji (2014) and Miwa and Sasaki (2014), respectively, which are both globally optimized models using discrete features, giving the top F-scores among statistical models.5
Overall, neural models give better performances than statistical models, and global optimization can give improved performances as well. Our final model achieves the best performances on both datasets. Compared with the best reported results, our model gives improvements of 1.9% on ACE05, and 6.8% on CONLL04.
3.5 Analysis
We conduct analysis on the ACE05 test dataset in order to understand our models in depth. We focus on two major contributions by our model, first examining the influences of global optimization, and then studying the gains by using the proposed syntactic features.
Global optimization aims to find the best label sequences, rather than the best label locally at each step. Thus intuitively global optimization should give better accuracies at the sentence level. We verify this by examining the sentence-level accuracies, where one sentence is regarded as correct when all the labels in the resulted table are correct. Figure 6 shows the result, which is consistent with our intuition. On the one hand, the sentence-
5Gupta et al. (2016) proposed a locally optimized model but used a different test dataset from CONLL04 and a different evaluation method, reporting entity and relation F-scores of 93.6% and 72.1%, respectively. Their results are not directly comparable to the results in Table 6. In particular, they regard an entity as correct if at least one token is tagged correctly, which influences the results significantly since multiword entities accounts for over 50% of all entities.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
5 10 15 20 25 ≥30 20
40
60
80 100 A cc ur ac y( % )
global local
Figure 6: Sentence-level accuracies with respect to sentence length.
1 2 3 4 5 ≥6 30
40
50
60
70
Fsc
or e(
% )
+syn -syn
Figure 7: F-scores with respect to the distance between entity pairs.
level accuracies of the globally normalized model are consistently better than the local model. On the other hand, the accuracy decreases sharply as the sentence length increases, with the local model suffering more severely from larger sentences.
To understand the effectiveness of the proposed syntactic features, we examine the relation Fscores with respect to entity distances. Miwa and Bansal (2016) exploit the shortest dependency path, which can make the distance between two entities closer compared with their sequential distance, thus facilitating relation extraction. We verify whether the proposed syntactic features can benefit our model similarly. As shown in Figure 7, the F-scores of entity-pairs with large distances see apparent improvements, demonstrating that our use of syntactic features has a similar effect compared to the shortest dependency path.
4 Related Work
Entity recognition (Florian et al., 2004, 2006; Ratinov and Roth, 2009; Florian et al., 2010; Kuru et al., 2016) and relation extraction (Zhao and Grishman, 2005; Jiang and Zhai, 2007; Zhou et al., 2007; Qian and Zhou, 2010; Chan and Roth, 2010; Sun et al., 2011; Plank and Moschitti, 2013; Verga et al., 2016) have received much attention in the NLP community. The dominant methods treat the two tasks separately, where relation extraction is performed assuming that entity boundaries have been given (Zelenko et al., 2003; Miwa et al.,
2009; Chan and Roth, 2011; Lin et al., 2016). Several studies find that extracting entities and relations jointly can benefit both tasks. Early work conducts joint inference for separate models (Ji and Grishman, 2005; Roth and Yih, 2004, 2007). Recent work shows that joint learning and decoding with a single model brings more benefits for the two tasks (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016; Gupta et al., 2016), and we follow this line of work in the study.
LSTM features have been extensively exploited for NLP tasks, including tagging (Huang et al., 2015; Lample et al., 2016), parsing (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016), relation classification (Xu et al., 2015; Vu et al., 2016; Miwa and Bansal, 2016) and sentiment analysis (Li et al., 2015; Teng et al., 2016). Based on the output of LSTM structures, Wang and Chang (2016) introduce segmental features, and apply it to dependency parsing. The same method is applied to constituent parsing by Cross and Huang (2016). We exploit this segmental representation for relation extraction.
Global optimization and normalization has been successfully applied on many NLP tasks that involve structural prediction (Lafferty et al., 2001; Collins, 2002; McDonald et al., 2010; Zhang and Clark, 2011), using traditional discrete features. For neural models, it has recently received increasing interests (Zhou et al., 2015; Andor et al., 2016; Xu, 2016; Wiseman and Rush, 2016), and improved performances can be achieved with global optimization accompanied by beam search. Our work is in line with these efforts. To our knowledge, we are the first to apply globally optimized neural models for end-to-end relation extraction, achieving the best results on standard benchmarks.
5 Conclusion
We proposed a novel relation extraction model using neural network, based on the table-filling framework proposed by Miwa and Sasaki (2014). Feature representations are learned from several LSTM structures over the inputs, and a simple method is used to integrate syntactic information into our model without the need of parser outputs. In addition, global optimization is taken to make use of structural information more effectively. Compared with previous work, our final model achieved the best performances on two benchmark datasets.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths:   - The paper is clearly written and well-structured.    - The system newly applied several techniques including global optimization to end-to-end neural relation extraction, and the direct incorporation of the parser representation is interesting.   - The proposed system has achieved the state-of-the-art performance on both ACE05 and CONLL04 data sets.   - The authors include several analyses.  - Weaknesses:   - The approach is incremental and seems like just a combination of existing methods.     - The improvements on the performance (1.2 percent points on dev) are relatively small, and no significance test results are provided.  - General Discussion:  - Major comments:   - The model employed a recent parser and glove word embeddings. How did they affect the relation extraction performance?   - In prediction, how did the authors deal with illegal predictions?  - Minor comments:   - Local optimization is not completely  local . It  considers structural correspondences between incremental decisions,  so this explanation in the introduction is misleading.   - Points in Figures 6 and 7 should be connected with straight lines, not curves.   - How are entities represented in  -segment ?   - Some citations are incomplete. Kingma et al. (2014) is accepted to ICLR, and Li et al. (2014) misses pages.