MinIE: Minimizing Facts in Open Information Extraction
The goal of open information extraction (OIE) is to extract surface relations and their arguments from naturallanguage text in an unsupervised, domainindependent manner. In this paper, we explore how overly-specific extractions can be reduced in OIE systems without producing uninformative or inaccurate results. We propose MinIE, an OIE system that produces minimized, annotated extractions. At its heart, MinIE rewrites OIE extractions by (1) identifying and removing parts that are considered overly specific; (2) representing information about polarity, modality, attribution, and quantities with suitable annotations instead of in the actual extraction. We conducted an experimental study with several real-world datasets and found that MinIE achieves competitive or higher precision and recall than most prior systems, while at the same time producing much shorter extractions.
1 Introduction
The goal of open information extraction (OIE) is to extract surface relations and their arguments from natural-language text in an unsupervised, domain-independent manner (Banko et al., 2007). In contrast to traditional IE systems, OIE systems do not require an upfront specification of the target schema (e.g., target relations) but instead represent extractions in the form of surface subject-relationobject triples, which are convenient for representing facts in a structured, machine-readable manner (Suchanek et al., 2007; Auer et al., 2007; Bizer et al., 2009). The extractions of OIE systems are useful for tasks such as information retrieval (Löser et al., 2012), question answering (Fader et al., 2014), knowledge-base extension (Dong et al., 2014; Riedel et al., 2013; Petroni
et al., 2015) as well as text comprehension, word similarity, word analogy (Stanovsky et al., 2015).
Consider for example the sentence “Superman was born on Krypton.” An OIE system aims to extract the triple (Superman, was born on, Krypton); in fact, most of the available systems will correctly produce this extraction. A key challenge in OIE is to avoid overly-specific extractions— while simultaneously producing informative and accurate results—when sentences get more complex. To see this, consider the more involved sentence “Pinocchio believes that the hero Superman was not actually born on beautiful Krypton.” Table 1 shows the corresponding extractions of various OIE systems.1 Many of the extractions are overly-specific in that the constituents contain specific modifiers or even form complete clauses. In this paper, we explore techniques to minimize OIE extractions and propose a new OIE system called MinIE. Table 1 also shows the output of (two variants of) MinIE for the example sentence.
At its heart, MinIE rewrites OIE extractions by (1) identifying and removing parts that are considered overly specific and (2) removing information about polarity, modality, attribution, and quantities. To retain the original meaning to the extent possible, MinIE provides suitable annotations for each extraction. The idea of using annotations has already been explored by OLLIE (Schmitz et al., 2012), which uses annotations for attribution and clausal modifiers. MinIE follows OLLIE, but uses additional and more expressive annotations. For example, MinIE detects negations in the relation, removes them from the extraction, and adds a “negative polarity” (-) annotation. In fact, MinIE considers the relations was born on and was not born on equivalent up to polarity. To the best of our knowledge, MinIE is the first OIE system to
1Suprisingly to us, this particular sentence is problematic for many prior systems. In practice, these systems generally perform better than indicated in this example.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Pinocchio believes that the hero Superman was not actually born on beautiful Krypton.
1 OLLIE (Pinocchio, believes that, the hero [...] beautiful Krypton) 2 (Superman, was not actually born on, beautiful Krypton) 3 (Superman, was not actually born on beautiful Krypton in, the hero) 4 ClausIE (Pinocchio, believes, that the hero [...] beautiful Krypton) 5 (the hero Superman, was not born, on beautiful Krypton) 6 (the hero Superman, was not born, on beautiful Krypton actually)
Stanford OIE No extractions 7 MinIE-S (Superman, was born on, beautiful Krypton) A.: fact (- [not], CT), attribution (Pinocchio, +, PS [believes]), relation (was actually born on) 8 (Superman, ”is”, hero) A.: fact (+, CT) 9 MinIE-C (Superman, was born on, Krypton)
A.: fact (- [not], CT), attribution (Pinocchio, +, PS [believes]), relation (was act. born on), argument (beau. K.) 10 (Superman, ”is”, hero)
A.: fact (+, CT)
Table 1: Example extractions and annotations from various OIE systems (“A”=annotation, “+”=positive polarity, “-”=negative polarity, “PS”=possibility, “CT”=certainty)
produce such negative-polarity extractions. The absence of negative evidence is a major concern for reasoning with OIE extractions—e.g., (Dong et al., 2014) proposes to use a “local closed world assumption” and (Riedel et al., 2013; Petroni et al., 2015) use “negative sampling” methods—, and MinIE’s annotations help to alleviate this problem.
Apart from polarity annotations, MinIE also provides annotations for modality, attribution, quantities, and for other rewrites that have been performed. Generally, minimizing OIE extractions is inherently limited in scope due to the absence of domain knowledge. Thus MinIE does not and cannot fully minimize its extractions in all cases. Instead, MinIE supports multiple minimization modes, which differ in their aggressiveness. Table 1 shows the output of MinIE’s safe mode (which is least aggressive) and its collocation mode (which is more aggressive). We conducted an experimental study with several realworld datasets and found that the various modes of MinIE produced much shorter extractions than prior systems, while simultaneously achieving competitive or higher precision. MinIE sometimes fell behind prior systems in terms of the total number of extractions. We found that those prior system can output a large number of redundant extractions; if redundant extractions were discounted, MinIE was competitive.
2 Related work
The concept of OIE was introduced by (Banko et al., 2007). Since then, many different OIE systems have been proposed in the literature. Early systems (Fader et al., 2011) relied on basic NLP techniques—such as POS tagging and chunking—, whereas all recent systems make use
of dependency parsing (Gamallo et al., 2012; Wu and Weld, 2010). Most systems represent their extractions in the form of triples, although some systems also produce n-ary extractions (Akbik and Löser, 2012; Del Corro and Gemulla, 2013) or use nested representations (Bast and Haussmann, 2013; Bhutani et al., 2016). Some systems focus on noun-mediated extractions (Yahya et al., 2014).
A general challenge in OIE is to avoid both uninformative and overly specific extractions. ReVerb (Fader et al., 2011) proposed to avoid overlyspecific relations by making use of lexical constraints: relations that occur infrequently in a large corpus were considered overly specific. MinIE’s frequency mode makes use of a similar constraint, but also applies it to subjects and arguments. CSDIE (Bast and Haussmann, 2013) introduced the notion of nested facts (termed minimal in their paper) and produce extractions with “pointers” to other extractions. NestIE (Bhutani et al., 2016) takes up this idea. MinIE currently does not handle nested extractions.
Perhaps the closest system in spirit to MinIE is Stanford OIE (Angeli et al., 2015), which also uses aggressive minimization. Stanford OIE deletes all subconstituents connected by certain typed dependencies (e.g., amod). For some typed dependencies (e.g., prep or dobj), the system makes use of a frequency constraint along the lines of ReVerb. MinIE differs from Stanford OIE in that it (1) separates out polarity, modality, attribution, and quantities; (2) uses different, more principled (and more precise) modes of minimization.
The concept of annotated OIE extractions was introduced by OLLIE (Schmitz et al., 2012). OLLIE uses two types of annotations: an attribution – describing the the supplier of the triple, and a
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
clause modifier – describing clauses that modify the extraction. MinIE picks up the attribution annotation, and uses additional techniques to provide the attribution’s polarity and modality. The clause modifier is an alternative approach to CSD-IE’s notion of nested facts and not present in MinIE. MinIE’s factuality and quantity annotations have not been provided by existing OIE systems.
3 Overview
The goal of MinIE is to provide minimized, annotated OIE extractions. While the techniques employed by MinIE can in principle be integrated into any OIE system, we built MinIE on top of ClausIE (Del Corro and Gemulla, 2013). We chose ClausIE because (1) it separates the identification of the constituents of the extractions from the generation of propositions, (2) it detects clause types, which are also useful for MinIE, and (3) it is a state-of-the-art OIE system.
As ClausIE, MinIE focuses on extractions obtained from individual clauses (with the exception of attribution; see Sec. 5.3). Each clause is broken up into its constituents: one subject (S), one verb (V) and optionally an indirect object (O), a direct object (O), a complement (C) and one or more adverbials (A). ClausIE then identifies the clause type, which indicates which of the constituents are obligatory and which optional from a syntactic point of view. (Quirk et al., 1985) identified the seven clause types SV, SVA, SVC, SVO, SVOO, SVOA, and SVOC; here letters refer to obligatory constituents and each clause can be accompanied by additional optional adverbials.
The general architecture of MinIE is outlined in Fig. 1. Each input sentence is run through ClausIE and a separate extractor for implicit facts (Sec. 4.2). ClausIE’s relations are enriched with information from other constituents (Sec. 4.1) in order to make them more informative. We refer to the resulting extractions as input extractions. MinIE then rewrites each input extraction to a basic extraction by detecting and eliminating information about polarity (Sec. 5.1), modality (Sec. 5.2), attribution (Sec. 5.3), and quantities (Sec. 5.4). To further minimize basic extractions, MinIE can be run in various modes (Sec. 6) with increasing levels of aggressiveness: MinIE-S(afe), MinIE-F(requency), MinIEC(ollocations), and MinIE-A(ggressive). The modes differ in the amount of minimizations being applied and which additional resources are being used. The result of this phase is a minimized
extraction. Finally, MinIE outputs each minimized extraction along with annotations that provide information about the minimization process. Some annotations (such as polarity) are crucial to correctly represent the extraction, others (such as original relation) give additional information about which parts have been removed during minimization.
4 Input Extractions
Before we discuss how we minimize extractions, we summarize how MinIE obtains its input extractions.
4.1 Enriching Relations
As mentioned before, MinIE uses ClausIE as its underlying OIE system. The relations extracted by ClausIE consist of only verbs and negation particles (cf. Tab. 1). (Fader et al., 2011) argues that such an approach can lead to uninformative relations. For example, from the sentence “Faust made a deal with the Devil”, ClausIE extracts triple (Faust, made, a deal with the Devil), whereas (Fader et al., 2011) argue that the extraction (Faust, made a deal with, the Devil) has a more informative relation. Indeed, the relation made is highly polysemous (49 synsets in WordNet), whereas made a deal with is not.
In MinIE, we follow (Fader et al., 2011) and try to produce more informative relations. To do so, we aim to decide which constituents of the input sentence should be pushed into the relation. Our approach is inspired by the syntactic patterns of (Fader et al., 2011), but additionally makes use of the clause type to obtain meaningful relations in a principled way. This allows us, for example, to handle cases with multiple adverbials or with adverbials that do not start with a preposition. Note that the relations produced in this step may sometimes be considered overly specific; they will be minimized further in subsequent steps.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
SVA. If the adverbial starts with a preposition, we push the preposition into the relation. For example, we rewrite (Superman, lives, in Metropolis) to (Superman, lives in, Metropolis). This allows us to distinguish live in from relations such as live during, live until, live through, and so on.
SVOO, SVOC. We generally push the indirect object (SVOO) or direct object (SVOC) into the relation. In both cases, the verb requires two additional constituents: we use the first one to enrich the relation and the second one as an argument. For example, we rewrite (Superman, declared, the city safe) to (Superman, declared the city, safe). As this example indicates, this rewrite is somewhat unsatisfying; further exploration is an interesting direction for future work.
SVOA. If the adverbial consists of a single adverb, we push it to the relation and use the object as an argument. Otherwise, we proceed as in SVOC, but additionally push the starting preposition (if present) of the adverbial to the relation. For example, (Ana, turned, the light off) becomes (Ana, turned off, the light), and (The doorman, leads, visitors to their destination) becomes (The doorman, leads visitors to, their destination).
Optional adverbials. If the clause contains optional adverbials, ClausIE creates one extraction per optional adverbial and one without any optional adverbial. We process the former extractions as if the adverbial were obligatory. For example, the extraction (Faust, made, a deal with the Devil) becomes (Faust, made a deal with, the Devil). Here the actual clause type is SVO, but we process it as if it were SVOA.
Infinitive forms. If the argument starts with a to-infinitive verb, we move it to the relation. For example, (Superman, needs, to defeat Lex) becomes (Superman, needs to defeat, Lex).
4.2 Implicit Extractions
ClausIE produces non-verb-mediated extractions from appositions and possesives. We refer to these extractions as implicit extractions. In MinIE, we use implicit extractions as a signal for the minimization of non-implicit ones (see Sec. 6.1). To make the minimization more effective, we include additional patterns identified in prior work. We use the explicit patterns of FINET (Del Corro et al., 2015) to detect explicit type mentions. For example, if the sentence contains president Barack Obama, we obtain (Barack Obama, is, president). We also include certain patterns involving named entities: pattern ORG IN LOC for extraction (ORG, is IN, LOC); pattern “Mr.” PER for
Sentence Factuality
S. does live in Metropolis. (+, CT) S. does not live in M. (– [not], CT)
S. does probably live in M. (+, PS [probably]) S. probably does not live in M. (– [not], PS [probably])
(PER, is, male) (similarly, Ms. or Mrs.); and pattern ORG POS? NP PER for (PER, is NP of, ORG) from RelNoun (Pal and Mausam, 2016).
5 Basic Minimization
The basic minimization step of MinIE detects and eliminates information about polarity (Sec. 5.1), modality (Sec. 5.2), attribution (Sec. 5.3), and quantities (Sec. 5.4). Our focus is on methods that are both domain-independent and (considered) safe to use.
5.1 Polarity
MinIE annotates each extraction with information about its factuality. Following (Saurı́ and Pustejovsky, 2012), we represent the factuality of an extraction with two pieces of information: polarity (+ or -) and modality (CT or PS; for certainty or possibility, resp.). Tab. 2 lists some examples.
The polarity indicates whether or not a triple occured in negated form. In order to assign a polarity value to a triple, we aim to detect whether the relation indicates a negative polarity. If so, we assign negative polarity to the whole triple. We detect negations using a small lexicon of negation words (consisting of the words no, not, never, none). Whenever a word from the lexicon is detected, it is dropped from the relation and the triple is annotated with negative polarity (-) and the negation word. In Tab. 2, the extractions from sentences 2 and 4 are annotated as negative.
We found that this simple approach successfully spots many negations present in the input relations. Note that whenever a negation is missed, MinIE still produces correct results because such negations are retained in the triple. For example, if a negations occurs in the subject or argument of the extraction, MinIE misses it. E.g., from sentence “No people were hurt in the fire”, MinIE extracts (Q1 people, were hurt in; fire) with quantity Q1 = no (see Sec. 5.4). This extraction is correct, but can be further minimized to (people; were hurt in; fire) with a negative polarity annotation. We consider such advanced minimizations too danger-
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ous to use in the absence of domain knowledge. Generally, negation detection is a hard problem and involves questions such as negation scope resolution, focus detection, and double negation (Blanco and Moldovan, 2011). MinIE does not address these problems, but restricts attention to the easy cases.
5.2 Modality
The modality of a triple indicates whether the triple is considered a certainty (CT) or a possibility (PS) in the clause in which it occurs. We proceed similarly to the detection of negations and consider a triple certain unless we find evidence indicating possibility.
To find such evidence, MinIE searches the relation for (1) modal verbs such as may or can, (2) possibility-indicating words, and (3) certain infinitive verb phrases. For (2) and (3), we make use of a small domain-indepependent lexicon. Our lexicon is based on the lexicon of (Saurı́ and Pustejovsky, 2012) and the words in the corresponding WordNet synsets. It mainly contains adverbs such as probably, possibly, maybe, likely and infinitive verb phrases such as is going to, is planning to, or intends to. Whenever words indicating possibility are detected, we remove these words from the triple and annotate the triple as possible (PS) along with the words just removed. For example, sentences 3 and 4 in Tab. 2 are annotated PS with the possibility-indicating word probably.
As with negation detection, MinIE’s approach focuses on the easy cases and uses only the CT and PS levels. A more advanced approach is taken by (Saurı́ and Pustejovsky, 2012).
5.3 Attribution
The attribution of a triple is the supplier of information given in the input sentence, if any. We adapt our attribution annotation from the notion of source in (Saurı́ and Pustejovsky, 2012), i.e., the attribution consists of a supplier of information along with a factuality (polarity and modality). The factuality is independent from the factuality of the extracted triple; it indicates whether the supplier expresses a negation or a possibility. Tab. 1 shows some examples.
We extract attributions from subordinate clauses as well as from an “according to” pattern. For the former, MinIE searches for extractions that contain entire clauses as arguments. We then compare the relation against a domain-independent dictionary of relations indicating attributions (e.g., say
or believe). If we find a match, we create an attribution annotation and use the subject of the extraction as the supplier of information. Each entry in the attribution dictionary is annotated with a modality. For example, relations such as know, say, or write express certainty, whereas relations such as believe or doubt express possibility. If the relation is modified by a negation word (see Sec. 5.1), we mark the attribution with negative polarity (e.g., never said that). After the attribution has been established, we run ClausIE on the main clause and add the attribution to each extracted triple. The second source of attributions is obtained via “according to” adverbial phrases. To detect those, we search for adverbials that start with according to and take whatever follows as the supplier with factuality (+,CT). The remaining part of the clause is processed as before.
5.4 Quantities
A quantity is a phrase that expresses an amount of something. It either modifies a noun phrase (e.g. 9 cats) or is an independent complement (e.g. I have 3). Quantities include cardinals (9), determiners (all) or whole phrases (almost 10). Whenever we detect a quantity, we replace it by a placeholder Q and add an annotation with the original quantity. The goal of this step is to unify extractions that only differ in quantities. For example, the arguments 9 cats, ten cats, all the cats and almost about 100 cats are all rewritten to Q cats, only the quantity annotation differs.
We detect quantities by looking for numbers (NER types such as NUMBER or PERCENT) or words expressing quantities (such as all, some, many). We then extend these words via relevant typed dependencies, such as quantity modifiers (quantmod) and or adverbial modifiers (advmod).
6 Minimization Modes
Once basic minimization has been performed, MinIE further minimizes extractions by dropping additional words. Since such minimization is risky, MinIE employs various minimization modes with different levels of aggressiveness. We discuss these modes in turn, from the safest mode to the most aggressive one.
MinIE represents each constituent of a basic extraction by its words, its dependency structure, its part-of-speech tags, and its named entities (detected by a named-entity recognizer). In general, each mode defines a set of stable subconstituents, which will always be fully retained, and subse-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
quently searches for candidate words to drop outside of the stable subconstituents. Whenever a word is dropped from a constituent, we add an annotation with the original, unmodified constituent. In all of MinIE’s modes, named entites are considered stable subconstituents.
6.1 Safe Mode (MinIE-S)
MinIE’s safe mode only drops word which we consider universally safe to drop. We first drop all constituents that are covered by the implicit extractions discussed in Sec. 4.2; e.g., types of named entities, locations of organizations, and the words “Mr.”, “Ms.”, and “Mrs.” before persons. We then drop all determiners, all possessive pronouns, all adverbs modifying the verb in the relation, and all adjectives and adverbials modifying persons. An exception to these rules is given by named entities, which we consider as stable subconstituents.
Note that this procedure cannot be considered safe when used on input extractions. We consider it safe, however, when applied to basic extractions. In particular, all determiners, pronouns, and adverbs indicating negation, modality, or quantities are already processed during basic minimization and are consequently retained in annotations. Moreover, due to the way we construct input extractions, each relation has at most one adverb modifying its verb. The safe mode thus only performs simple rewrites such as the great city to great city, his presidency to presidency, had also to had, and the eloquent president Mr. Barack Obama to Barack Obama.
6.2 Frequency Mode (MinIE-F) and Collocations Mode (MinIE-C)
The frequency and collocations modes make use of a dictionary D of stable constituents. The two modes differ only in the choice of D. We first discuss how the dictionary is being used, and subsequently how we can construct it.
Both modes first perform all the minimizations of the safe mode and then search for maximal noun phrases of the form P ≡ [adverbial|adjective]+ [noun+|ner]. For each instance of P , we drop a certain subset of its words. For example, a suitable minimization for very infamous cold war symbol (i.e., the Berlin wall) is cold war symbol, i.e., we consider cold as essential to the meaning of the constituent and very infamous as overly specific. The decision of what is considered essential and what overly specific is informed by dictionary D. Note that in order to minimize mistakes, we con-
sider for dropping only words in instances of pattern P . In particular, we do not touch subconstituents that contain prepositions because these are notoriously difficult to handle (e.g., we do not want to minimize Bill of Rights to Bill).
Our main goal is to retain phrases that occur in D, even if they occur in different order or with additional modifiers. To achieve this goal, we proceed as follows for each instance I of P . We first mark all nouns (or the named entity) as stable. Afterwards, we create a set of potentially stable subconstituents (PSS). Each PSS is queried against dictionary D. If it occurs in D, all of its words are marked as stable. Once all PSS have been processed, we drop all words from I that are not marked stable. In our example, if D = {cold war}, we obtain the minimization cold war symbol as desired.
To generate the set of PSS, we enumerate all subconstituents of I that are syntactically valid. For example, infamous symbol or cold infamous war are syntactically valid, whereas very symbol or very cold war are not. Conceptually,2 we enumerate all subsequences of I and check whether (1) at least one noun (or the named entity) is retained, and (2) whenever an adverbial or adjective has been dropped, so are all of its modifiers. For each such subsequence, we generate all permutations of adverbial and adjective modifiers originating from the same node in the dependency structure. This step ensures that the order of modifiers in I does influence whether or not a word is marked stable. The set of PSS for very infamous cold war symbol contains 22 entries.
It remains to discuss the construction of dictionary D. In MinIE’s collocations mode, D is provided externally and contains a set of collocations. In our experimental study, we included all multi-word expressions from WordNet and Wiktionary, which we consider domain-independent. In practice, applications can extend the dictionary using suitable collocations, either from domaindependent dictionaries or by using methods to automatically extract colloctions from a text corpus (Gries, 2013). In MinIE’s frequency mode, we process the entire corpus using the safe mode and additionally include all frequent (e.g., frequency ≥ 10) subjects, relations, and arguments into D. This lexical constraint is inspired by (Fader et al., 2011); the rationale is that everything sufficiently frequent is not considered overly specific.
2In our implementation, we generate both instances of P as well as the set of PSS directly from the dependency structure of the constituent.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
6.3 Aggressive Mode (MinIE-A)
The modes discussed so far aimed to be conservative. MinIE-A proceeds the other way around: all words for which we are not sure that they need to be retained are dropped. We mainly include this mode as a baseline and to study how effective such an approach is empirically.
For every word in a constituent of a basic extraction, we drop all adverbial, adjective, possessive, and temporal modifiers (recursively along with their modifiers). We also drop prepositional attachments (e.g., man with apples becomes man), quantifiers modifying nouns, auxiliary modifiers to the main verb (e.g., have escalated becomes escalated), and all compound nouns that have a different named-entity type than their head word (e.g., European Union offical becomes offical). In most cases, after applying these steps, only a single word, named entity, or a sequence of nouns remains for subject and argument constituents.
7 Experimental Study
The goal of our experimental study was to investigate the differences in the various modes of MinIE w.r.t. precision, recall, and extraction length as well as to compare it to popular prior methods.
7.1 Experimental Setup
All source code, datasets, extractions, labels, and labelling guidelines will be made publically available. We also submitted them to SoftConf.
Datasets. We used as datasets (1) a random sample of 10,000 sentences from the New York Times Annotated Corpus (NYT-10k), (2) a random sample of 200 sentences from the same corpus (NYT), and (3) a random sample of 200 sentences from Wikipedia (Wiki). Datasets NYT and Wiki were used in the evaluation of ClausIE and NestIE.
Methods. We used ClausIE, OLLIE, and Stanford OIE as baseline systems. We adapted the publically available version of ClausIE to Stanford CoreNLP 3.5.1 (Manning et al., 2014) and implemented MinIE on top. For MinIE-F, we built the dictionary using the entire NYT corpus.
Labeling. We labeled every extraction of NYT and Wiki with two labels: one for the triple and one for the attribution. A triple is labeled as correct if it is entailed by its corresponding clause; here factuality annotations are taken into account. For example, all triples except #3 of Tab. 1 are considered correct. The attribution is labeled incorrect if there is an attribution which is neither
present in the triple nor in the attribution annotation. In Tab. 1, the attribution is incorrect for extractions 2, 3, 5, and 6. We only label attribution when the fact triple is labeled correct.3 Further details can be found in the labelling guidelines.
Measures. For each system, we measured the total number of extractions (termed recall), the fraction of correct triples (factual precision), and the fraction of correct triples with correct attributions (attribution precision). We also determined the mean word count per extraction (µ) and its standard deviation (σ); we use these numbers as a proxy for minimality. Finally, we observed that some systems produced a large number of redundant extractions. To obtain insight into the amount of redundancy, we also report the number of nonredundant extractions. For simplicity, we consider a triple t1 redundant if it appears as subsequence in some other triple t2 produced by the same extractor from the same sentence (e.g., extraction 5 in Tab. 1 is redundant given extraction 6).
7.2 Extraction Statistics
In our first experiment, we used the larger NYT10k dataset (which we did not label). Tab. 3 summarizes statistics of the extraction produced by each system. For MinIE, we show the fraction of negative polarity and possibility annotations for triples only (i.e., we exclude the attribution annotation).
In terms of recall, MinIE and Stanford OIE were roughly on par; OLLIE fell behind and ClausIE went ahead. The reason why ClausIE has more extractions than MinIE is that different (partly redundant) extractions from ClausIE may lead to the same minimized extraction. This is also also the reason why recall drops in the more aggressive modes of MinIE. We also determined the number of non-redundant extractions produced by each system and found that most system’s only produce a moderate number of redundant extractions. A notable exception is Stanford OIE, which produces many extraction variants by dropping different subsets of words.
We observed that all modes of MinIE achieved significantly smaller extractions than ClausIE (its underlying OIE system), and that the average extraction length indeed dropped as we used more aggressive modes. Only MinIE-A produced shorter extractions than Stanford OIE. The main reason for the short extraction length of Stanford
3In fact, it’s meaningless to attribute incorrect extractions to suppliers.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
OLLIE ClausIE Stanford MinIE-S MinIE-F MinIE-C MinIE-A
# non-redundant extr. 21,277 41,174 16,779 37,638 37,698 37,277 36,155 # extractions 24,316 58,420 43,365 45,649 45,386 45,322 43,035
µ± σ 9.3± 5.2 10.9± 7 6.6± 3.0 7.2± 4.2 6.9± 4.0 6.8± 4.0 4.7± 1.9 with attributions 6.8% - - 10.7% 10.69% 10.7% 10.8%
with negative polarity - - - 3.7% 3.7% 3.7% 3.8% with possibility - - - 9.9% 9.9% 9.9% 9.7%
with quantity - - - 17.8% 17.8% 17.8% 1.9%
Table 3: Results on the unlabeled NYT-10k dataset (µ=avg. extraction length, σ=standard deviation)
OLLIE ClausIE Stanford MinIE-S MinIE-F MinIE-C MinIE-A
NYT # non-redundant 306/435 621/928 218/364 623/791 618/792 609/785 514/748 # extractions 353/497 890/1300 649/1052 730/923 717/914 713/914 588/861 factual prec. (0.71) (0.68) (0.62) (0.79) (0.78) (0.78) (0.68) attr. prec. (0.92) - - (0.94) (0.94) (0.93) (0.93) Wiki # non-redundant 343/491 561/777 261/431 558/676 - 540/674 459/663
# extractions 399/565 728/1002 812/1516 683/822 - 653/819 528/788 factual prec. (0.71) (0.73) (0.54) (0.83) - (0.80) (0.67)
attr. prec. (0.98) - - (0.98) - (0.98) (0.98)
Table 4: Results on the labeled NYT and Wiki datasets
OIE is its aggressive creation of (shorter) redundant extractions.
Only OLLIE and MinIE make use of annotations. The fraction of extracted attribution annotations was significantly smaller for OLLIE than for MinIE, mainly because OLLIE’s attribution detection is limited to the ccomp dependency relation. The statistics also indicate that the basic minimizations of MinIE fire quite frequently, with the notable exception of negations.
7.3 Precision
In our second experiment, we compared the precision of the various systems on the smaller NYT and Wiki datasets. We did not run MinIE-F on Wiki because it was trained on the full NYT corpus. Our results are summarized in Tab. 4, where we give recall in the form of #correct/#total.
We found that Stanford OIE had the lowest factual precision throughout, producing many incorrect and many redundant extractions (e.g., Stanford OIE produced 400 extractions from five sentences on NYT). For MinIE, the factual precision dropped as expected when we use more aggressive modes. Interestingly, the drop in precision between MinIE-S, MinIE-F, and MinIE-C was quite low, even though extractions get shorter. The aggressive minimization of MinIE-A led to a more severe drop in precision. Surprisingly to us, even MinIE’s aggressive mode achieved higher precision than Stanford OIE. ClausIE and OLLIE had similar factual precision. Note that MinIE-S had higher precision than ClausIE. Reasons include that MinIE-S produces additional high-precision implicit extractions, breaks up very long and thus
error-prone extractions, and uses a different version of Stanford CoreNLP.
As for attribution precision, most of the sentences in our samples did not contain attributions; these numbers should thus be taken with a grain of salt. OLLIE and MinIE achieved similar results, even though MinIE additionally annotated attributions with factuality information.
7.4 MinIE’s Errors
For all modes, errors in dependency parsing transfer over to errors in MinIE, which we believe is the main source of errors in MinIE-S. For MinIE-C and MinIE-F, we sometimes drop adjectives which in fact form collocations (e.g., “assistant director”) with the noun they are modifying, but the collocation is not present in the dictionary and thus broken up. This problem can be addressed by using richer collocation dictionaries. Another source of error stems from the named-entity recognizer (e.g., the first word of the named entity Personal Ensign was not recognized so that MinIE incorrectly extracts (”Ensign”, ”is”, ”Personal”)).
8 Conclusions
We presented MinIE, a system that minimizes and annotates OIE extractions. We believe that the use of minimized extractions with suitable annotations are a promising direction for OIE. The techniques presented here can be seen as a step towards this goal. Future directions include additional annotation types (e.g. temporal/spatial), improved techniques for basic minimizations, better handling of collocations, and use of nested representations.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  [+] Well motivated, tackles an interesting problem;  [+] Clearly written and structured, accompanied by documented code and dataset;  [+] Encouraging results.  - Weaknesses:  [-] Limited to completely deterministic, hand-engineered minimization rules;  [-] Some relevant literature on OIE neglected;  [-] Sound but not thorough experimental evaluation.  - General Discussion:  This paper tackles a practical issue of most OIE systems, i.e. redundant, uninformative and inaccurate extractions. The proposed approach, dubbed MinOIE, is designed to actually  minimize  extractions by removing overly specific portions and turning them into structured annotations of various types (similarly to OLLIE). The authors put MinIE on top of a state-of-the-art OIE system (ClausIE) and test it on two publicly available datasets, showing that it effectively leads to more concise extractions compared to standard OIE approaches, while at the same time retaining accuracy.  Overall, this work focuses on an interesting (and perhaps underinvestigated) aspect of OIE in a sound and principled way. The paper is clearly written, sufficiently detailed, and accompanied by supplementary material and a neat Java implementation. My main concern is, however, with the entirely static, deterministic and rule-based structure of MinIE. Even though I understand that a handful of manually engineered rules is technically the best strategy when precision is key, these approaches are typically very hard to scale, e.g. in terms of languages (a recent trend of OIE, see Faruqui and Kumar, 2015; Falke et al., 2016). In other words, I think that this contribution somehow falls short of novelty and substance in proposing a pipeline of engineered rules that are mostly inspired by other OIE systems (such as ClausIE or ReVerb); for instance, I would have really appreciated an attempt to learn these minimization rules instead of hard-coding them.  Furthermore, the authors completely ignore a recent research thread on  semantically-informed d OIE (Nakashole et al., 2012; Moro and Navigli, 2012; 2013; Delli Bovi et al., 2015) where traditional extractions are augmented with links to underlying knowledge bases and sense inventories (Wikipedia, Wikidata, Yago, BabelNet). These contributions are not only relevant in terms of related literature: in fact, having text fragments (or constituents) explicitly linked to a knowledge base would reduce the need for ad-hoc minimization rules such as those in Sections 6.1 and 6.2. In the example with  Bill of Rights  provided by the authors (line 554), an OIE pipeline with a proper Entity Linking module would recognize automatically the phrase as mention of a registered entity, regardless of the shape of its subconstituents. Also, an underlying sense inventory would seamlessly incorporate the external information about collocations and multi-word expressions used in Section 6.2: not by chance, the authors rely on WordNet and Wiktionary to compile their dictionary of collocations.  Finally, some remarks on the experimental evaluation:  - Despite the claim of generality of MinIE, the authors choose to experiment only with ClausIE as underlying OIE system (most likely the optimal match). It would have been very interesting to see if the improvement brought by MinIE is consistent also with other OIE systems, in order to actually assess its flexibility as a post-processing tool.  - Among the test datasets used in Section 7, I would have included the recent OIE benchmark of Stanovsky and Dagan (2016), where results are reported also for comparison systems not included in this paper (TextRunner, WOIE, KrakeN).  References:  - Manaal Faruqui and Shankar Kumar. Multilingual Open Relation Extraction using Cross-lingual Projection. NAACL-HLT, 2015.  - Tobias Falke, Gabriel Stanovsky, Iryna Gurevych and Ido Dagan. Porting an Open Information Extraction System from English to German. EMNLP 2016.  - Ndapandula Nakashole, Gerhard Weikum and Fabian Suchanek. PATTY: A Taxonomy of Relational Patterns with Semantic Types. EMNLP 2012.  - Andrea Moro, Roberto Navigli. WiSeNet: Building a Wikipedia-based Semantic Network with Ontologized Relations. CIKM 2012.  - Andrea Moro, Roberto Navigli. Integrating Syntactic and Semantic Analysis into the Open Information Extraction Paradigm. IJCAI 2013.  - Claudio Delli Bovi, Luca Telesca and Roberto Navigli. Large-Scale Information Extraction from Textual Definitions through Deep Syntactic and Semantic Analysis. TACL vol. 3, 2015.  - Gabriel Stanovsky and Ido Dagan. Creating a Large Benchmark for Open Information Extraction. EMNLP 2016.


In this paper we study the problem of answering cloze-style questions over documents. Our model, the Gated-Attention (GA) Reader, integrates a multi-hop architecture with a novel attention mechanism, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection. The GA Reader obtains state-of-the-art results on three benchmarks for this task–the CNN & Daily Mail news stories and the Who Did What dataset. The effectiveness of multiplicative interaction is demonstrated by an ablation study, and by comparing to alternative compositional operators for implementing the gated-attention.
1 Introduction
A recent trend to measure progress towards machine reading is to test a system’s ability to answer questions about a document it has to comprehend. Towards this end, several large-scale datasets of cloze-style questions over a context document have been introduced recently, which allow the training of supervised machine learning systems (Hermann et al., 2015; Hill et al., 2015; Onishi et al., 2016). Such datasets can be easily constructed automatically and the unambiguous nature of their queries provides an objective benchmark to measure a system’s performance at text comprehension.
Deep learning models have been shown to outperform traditional shallow approaches on text comprehension tasks (Hermann et al., 2015). The
success of many recent models can be attributed primarily to two factors: (1) Multi-hop architectures allow a (Weston et al., 2014; Sordoni et al., 2016; Shen et al., 2016), model to scan the document and the question iteratively for multiple passes. (2) Attention mechanisms, (Chen et al., 2016; Hermann et al., 2015) borrowed from the machine translation literature (Bahdanau et al., 2014), allow the model to focus on appropriate subparts of the context document. Intuitively, the multi-hop architecture allows the reader to incrementally refine token representations, and the attention mechanism re-weights different parts in the document according to their relevance to the query.
The effectiveness of multi-hop reasoning and attentions have been explored orthogonally so far in the literature. In this paper, we focus on combining both in a complementary manner, by designing a novel attention mechanism which gates the evolving token representations across hops. More specifically, unlike existing models where the query attention is applied either token-wise (Hermann et al., 2015; Kadlec et al., 2016; Chen et al., 2016; Hill et al., 2015) or sentence-wise (Weston et al., 2014; Sukhbaatar et al., 2015) to allow weighted aggregation, the Gated-Attention (GA) module proposed in this work allows the query to directly interact with each dimension of the token embeddings at the semantic-level, and is applied layer-wise as information filters during the multi-hop representation learning process. Such a fine-grained attention enables our model to learn conditional token representations w.r.t. the given question, leading to accurate answer selections.
We show in our experiments that the proposed GA reader, despite its relative simplicity, consistently improves over a variety of strong baselines on three benchmark datasets Our key contribution, the GA module, provides a significant im-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
provement for large datasets. Qualitatively, visualization of the attentions at intermediate layers of the GA reader shows that in each layer the GA reader attends to distinct salient aspects of the query which help in determining the answer.
2 Related Work
The cloze-style QA task involves tuples of the form (d, q, a, C), where d is a document (context), q is a query over the contents of d, in which a phrase is replaced with a placeholder, and a is the answer to q, which comes from a set of candidates C. In this work we consider datasets where each candidate c ∈ C has at least one token which also appears in the document. The task can then be described as: given a document-query pair (d, q), find a ∈ C which answers q. Below we provide an overview of representative neural network architectures which have been applied to this problem.
LSTMs with Attention: Several architectures introduced in (Hermann et al., 2015) employ LSTM units to compute a combined document-query representation g(d, q), which is used to rank the candidate answers. These include the DeepLSTM Reader which performs a single forward pass through the concatenated (document, query) pair to obtain g(d, q); the Attentive Reader which first computes a document vector d(q) by a weighted aggregation of words according to attentions based on q, and then combines d(q) and q to obtain their joint representation g(d(q), q); and the Impatient Reader where the document representation is built incrementally. The architecture of the Attentive Reader has been simplified recently in Stanford Attentive Reader, where shallower recurrent units were used with a bilinear form for the query-document attention (Chen et al., 2016).
Attention Sum: The Attention-Sum (AS) Reader (Kadlec et al., 2016) uses two bidirectional GRU networks (Cho et al., 2014) to encode both d and q into vectors. A probability distribution over the entities in d is obtained by computing dot products between q and the entity embeddings and taking a softmax. Then, an aggregation scheme named pointer-sum attention is further applied to sum the probabilities of the same entity, so that frequent entities the document will be favored compared to rare ones. Building on the AS Reader, the Attention-over-Attention (AoA) Reader (Cui et al., 2016) introduces a two-way attention mechanism where the query and the doc-
ument are mutually attentive to each other. Mulit-hop Architectures: Memory Networks (MemNets) were proposed in (Weston et al., 2014), where each sentence in the document is encoded to a memory by aggregating nearby words. Attention over the memory slots given the query is used to compute an overall memory and to renew the query representation over multiple iterations, allowing certain types of reasoning over the salient facts in the memory and the query. Neural Semantic Encoders (NSE) (Munkhdalai & Yu, 2016a) extended MemNets by introducing a write operation which can evolve the memory over time during the course of reading. Iterative reasoning has been found effective in several more recent models, including the Iterative Attentive Reader (Sordoni et al., 2016) and ReasoNet (Shen et al., 2016). The latter allows dynamic reasoning steps and is trained with reinforcement learning.
Other related works include Dynamic Entity Representation network (DER) (Kobayashi et al., 2016), which builds dynamic representations of the candidate answers while reading the document, and accumulates the information about an entity by max-pooling; EpiReader (Trischler et al., 2016) consists of two networks, where one proposes a small set of candidate answers, and the other reranks the proposed candidates conditioned on the query and the context; Bi-Directional Attention Flow network (BiDAF) (Seo et al., 2016) adopts a multi-stage hierarchical architecture along with a flow-based attention mechanism; (Bajgar et al., 2016) showed a 10% improvement on the CBT corpus (Hill et al., 2015) by training the AS Reader on an augmented training set of about 14 million examples, making a case for the community to exploit data abundance. The focus of this paper, however, is on designing models which exploit the available data efficiently.
3 Gated-Attention Reader
Our proposed GA readers perform multiple hops over the document (context), similar to the Memory Networks architecture (Sukhbaatar et al., 2015). Multi-hop architectures mimic the multistep comprehension process of human readers, and have shown promising results in several recent models for text comprehension (Sordoni et al., 2016; Kumar et al., 2015; Shen et al., 2016). The contextual representations in GA readers, namely the embeddings of words in the document, are it-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
eratively refined across hops until reaching a final attention-sum module (Kadlec et al., 2016) which maps the contextual representations in the last hop to a probability distribution over candidate answers.
The attention mechanism has been introduced recently to model human focus, leading to significant improvement in machine translation and image captioning (Bahdanau et al., 2014; Mnih et al., 2014). In reading comprehension tasks, ideally, the semantic meanings carried by the contextual embeddings should be aware of the query across hops. As an example, human readers are able to keep the question in mind during multiple passes of reading, to successively mask away information irrelevant to the query. However, existing neural network readers are restricted to either attend to tokens (Hermann et al., 2015; Chen et al., 2016) or entire sentences (Weston et al., 2014), with the assumption that certain sub-parts of the document are more important than others. In contrast, we propose a finer-grained model which attends to components of the semantic representation being built up by the GRU. The new attention mechanism, called gated-attention, is implemented via multiplicative interactions between the query and the contextual embeddings, and is applied per hop to act as fine-grained information filters during the multi-step reasoning. The filters weigh individual components of the vector representation of each token in the document separately.
The design of gated-attention layers is motivated by the effectiveness of multiplicative interaction among vector-space representations, e.g., in various types of recurrent units (Hochreiter & Schmidhuber, 1997; Wu et al., 2016) and in relational learning (Yang et al., 2014; Kiros et al., 2014). While other types of compositional operators are possible, such as concatenation or addition (Mitchell & Lapata, 2008), we find that multiplication has strong empirical performance (section 4.3), where query representations naturally serve as information filters across hops.
3.1 Model Details
Several components of the model use a Gated Recurrent Unit (GRU) (Cho et al., 2014) which maps an input sequence X = [x1, x2, . . . , xT ] to an
ouput sequence H = [h1, h2, . . . , hT ] as follows:
rt = σ(Wrxt + Urht−1 + br),
zt = σ(Wzxt + Uzht−1 + bz),
h̃t = tanh(Whxt + Uh(rt ht−1) + bh), ht = (1− zt) ht−1 + zt h̃t.
where denotes the Hadamard product or the element-wise multiplication. rt and zt are called the reset and update gates respectively, and h̃t the candidate output. A Bi-directional GRU (BiGRU) processes the sequence in both forward and backward directions to produce two sequences [hf1 , h f 2 , . . . , h f T ] and [h b 1, h b 2, . . . , h b T ], which are concatenated at the output ←→ GRU(X) = [hf1‖h b T , . . . , h f T ‖h b 1] (1)
where ←→ GRU(X) denotes the full output of the Bi-GRU obtained by concatenating each forward state hfi and backward state h b T−i+1 at step i given the inputX . Note ←→ GRU(X) is a matrix in R2nh×T where nh is the number of hidden units in GRU. Let X(0) = [x(0)1 , x (0) 2 , . . . x (0) |D|] denote the token embeddings of the document, which are also inputs at layer 1 for the document reader below, and Y = [y1, y2, . . . y|Q|] denote the token embeddings of the query. Here |D| and |Q| denote the document and query lengths respectively.
3.1.1 Multi-Hop Architecture
Fig. 1 illustrates the Gated-Attention (GA) reader. The model reads the document and the query over K horizontal layers, where layer k receives the contextual embeddings X(k−1) of the document from the previous layer. The document embeddings are transformed by taking the full output of a document Bi-GRU (indicated in blue in Fig. 1):
D(k) = ←→ GRU (k)
D (X (k−1)) (2)
At the same time, a layer-specific query representation is computed as the full output of a separate query Bi-GRU (indicated in green in Figure 1):
Q(k) = ←→ GRU (k)
Q (Y ) (3)
Next, Gated-Attention is applied to D(k) and Q(k) to compute inputs for the next layer X(k).
X(k) = GA(D(k), Q(k)) (4)
where GA is defined in the following subsection.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 1: Gated-Attention Reader. Dashed lines represent dropout connections.
3.1.2 Gated-Attention Module
For brevity, let us drop the superscript k in this subsection as we are focusing on a particular layer. For each token di in D, the GA module forms a token-specific representation of the query q̃i using soft attention, and then multiplies the query representation element-wise with the document token representation. Specifically, for i = 1, . . . , |D|:
αi = softmax(Q>di) (5)
q̃i = Qαi
xi = di q̃i (6)
In equation (6) we use the multiplication operator to model the interactions between di and q̃i. In the experiments section, we also report results for other choices of gating functions, including addition xi = di + q̃i and concatenation xi = di‖q̃i.
3.1.3 Answer Prediction
Let q(K)` = q f ` ‖q b T−`+1 be an intermediate output of the final layer query Bi-GRU at the location ` of the cloze token in the query, and D(K) = ←→ GRU (K)
D (X (K−1)) be the full output of final layer
document Bi-GRU. To obtain the probability that a particular token in the document answers the query, we take an inner-product between these two, and pass through a softmax layer:
s = softmax((q(K)` ) TD(K)) (7)
where vector s defines a probability distribution over the |D| tokens in the document. The probability of a particular candidate c ∈ C as being the
answer is then computed by aggregating the probabilities of all document tokens which appear in c and renormalizing over the candidates:
Pr(c|d, q) ∝ ∑
i∈I(c,d)
si (8)
where I(c, d) is the set of positions where a token in c appears in the document d. This aggregation operation is the same as the pointer sum attention applied in the AS Reader (Kadlec et al., 2016).
Finally, the candidate with maximum probability is selected as the predicted answer:
a∗ = argmaxc∈C Pr(c|d, q). (9)
During the training phase, model parameters of GA are updated w.r.t. a cross-entropy loss between the predicted probabilities and the true answers.
3.1.4 Further Enhancements
Character-level Embeddings: Given a token w from the document or query, its vector space representation is computed as x = L(w)||C(w). L(w) retrieves the word-embedding forw from a lookup table L ∈ R|V |×nl , whose rows hold a vector for each unique token in the vocabulary. We also utilize a character composition model C(w) which generates an orthographic embedding of the token. Such embeddings have been previously shown to be helpful for tasks like Named Entity Recognition (Yang et al., 2016) and dealing with OOV tokens at test time (Dhingra et al., 2016). The embedding C(w) is generated by taking the final outputs zfnc and zbnc of a Bi-GRU applied to embeddings from
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
a lookup table of characters in the token, and applying a linear transformation:
z = zfnc ||z b nc C(w) =Wz + b
Question Evidence Common Word Feature (qecomm): (Li et al., 2016) recently proposed a simple token level indicator feature which significantly boosts reading comprehension performance in some cases. For each token in the document we construct a one-hot vector fi ∈ {0, 1}2 indicating its presence in the query. It can be incorporated into the GA reader by assigning a feature lookup table F ∈ RnF×2 (we use nF = 2), taking the feature embedding ei = fTi F and appending it to the inputs of the last layer document BiGRU as, x(K)i ‖fi for all i. We conducted several experiments both with and without this feature and observed some interesting trends, which are discussed below. Henceforth, we refer to this feature as the qe-comm feature or just feature.
4 Experiments and Results
4.1 Datasets
We evaluate the GA reader on five large-scale datasets recently proposed in the literature. The first two, CNN and Daily Mail news stories1 consist of articles from the popular CNN and Daily Mail websites (Hermann et al., 2015). A query over each article is formed by removing an entity from the short summary which follows the article. Further, entities within each article were anonymized to make the task purely a comprehension one. N-gram statistics, for instance, computed over the entire corpus are no longer useful in such an anonymized corpus.
The next two datasets are formed from two different subsets of the Children’s Book Test (CBT)2 (Hill et al., 2015). Documents consist of 20 contiguous sentences from the body of a popular children’s book, and queries are formed by deleting a token from the 21st sentence. We only focus on subsets where the deleted token is either a common noun (CN) or named entity (NE) since simple language models already give human-level performance on the other types (cf. (Hill et al., 2015)).
The final dataset is Who Did What3 (WDW) (Onishi et al., 2016), constructed from the LDC
1 https://github.com/deepmind/rc-data 2 http://www.thespermwhale.com/jaseweston/babi/
CBTest.tgz 3 https://tticnlp.github.io/who_did_what/
English Gigaword newswire corpus. First, article pairs which appeared around the same time and with overlapping entities are chosen, and then one article forms the document and a cloze query is constructed from the other. Missing tokens are always person named entities. Questions which are easily answered by simple baselines are filtered out, to make the task more challenging. There are two versions of the training set—a small but focused “Strict” version and a large but noisy “Relaxed” version. We report results on both settings which share the same validation and test sets. Statistics of all the datasets used in our experiments are summarized in the Appendix (Table 4).
4.2 Performance Comparison
Tables 1 and 3 show a comparison of the performance of GA Reader with previously published results on WDW and CNN, Daily Mail, CBT datasets respectively. The numbers reported for GA Reader are for single best models, though we compare to both ensembles and single models from prior work. GA Reader-- refers to an earlier version of the model, unpublished but described in a preprint, with the following differences—(1) it does not utilize token-specific attentions within the GA module, as described in equation (5), (2) it does not use a character composition model, (3) it is initialized with word embeddings pretrained on the corpus itself rather than GloVe. A detailed analysis of these differences is studied in the next section. Here we present 4 variants of the latest GA Reader, using combinations of whether the qe-comm feature is used (+feature) or not, and whether the word lookup table L(w) is updated during training or fixed to its initial value. Other hyperparameters are listed in Appendix A.
Interestingly, we observe that feature engineering leads to significant improvements for WDW and CBT datasets, but not for CNN and Daily Mail datasets. We note that anonymization of the latter datasets means that there is already some feature engineering (it adds hints about whether a token is an entity), and these are much larger than the other four. In machine learning it is common to see the effect of feature engineering diminish with increasing data size. Similarly, fixing the word embeddings provides an improvement for the WDW and CBT, but not for CNN and Daily Mail. This is not surprising given that the latter datasets are larger and less prone to overfitting.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Table 1: Validation/Test accuracy (%) on WDW dataset for both “Strict” and “Relaxed” settings. Results with “†” are cf previously published works.
Model Strict Relaxed
Val Test Val Test
Human † – 84 – –
Attentive Reader † – 53 – 55 AS Reader † – 57 – 59 Stanford AR † – 64 – 65 NSE † 66.5 66.2 67.0 66.7
GA-- † – 57 – 60.0 GA (update L(w)) 67.8 67.0 67.0 66.6 GA (fix L(w)) 68.3 68.0 69.6 69.1 GA (+feature, update L(w)) 70.1 69.5 70.9 71.0 GA (+feature, fix L(w)) 71.6 71.2 72.6 72.6
Comparing with prior work, on the WDW dataset the basic version of the GA Reader outperforms all previously published models when trained on the Strict setting. By adding the qecomm feature the performance increases by 3.2% and 3.5% on the Strict and Relaxed settings respectively to set a new state of the art on this dataset. On the CNN and Daily Mail datasets the GA Reader leads to an improvement of 3.2% and 4.3% respectively over the best previous single models. They also outperform previous ensemble models, setting a new state of that art for both datasets. For CBT-NE, GA Reader with the qecomm feature outperforms all previous single and ensemble models except the AS Reader trained on the much larger BookTest Corpus (Bajgar et al., 2016). Lastly, on CBT-CN the GA Reader with the qe-comm feature outperforms all previously published single models except the NSE, and AS Reader trained on a larger corpus.
4.3 GA Reader Analysis
In this section we do an ablation study to see the effect of Gated Attention. We compare the GA Reader as described here to a model which is exactly the same in all aspects, except that it passes document embeddings D(k) in each layer directly to the inputs of the next layer without using the GA module. In other words X(k) = D(k) for all k > 0. This model ends up using only one query GRU at the output layer for selecting the answer from the document. We compare these two variants both with and without the qe-comm feature on CNN and WDW datasets for three subsets of
the training data - 50%, 75% and 100%. Test set accuracies for these settings are shown in Figure 2. On CNN when tested without feature engineering, we observe that GA provides a significant boost in performance compared to without GA. When tested with the feature it still gives an improvement, but the improvement is significant only with 100% training data. On WDW-Strict, which is a third of the size of CNN, without the feature we see an improvement when using GA versus without using GA, which becomes significant as the training set size increases. When tested with the feature on WDW, for a small data size without GA does better than with GA, but as the dataset size increases they become equivalent. We conclude that GA provides a boost in the absence of feature engineering, or as the training set size increases.
Next we look at the question of how to gate intermediate document reader states from the query, i.e. what operation to use in equation 6. Table 2 (top) shows the performance on WDW dataset for three common choices – sum (x = d + q), concatenate (x = d‖q) and multiply (x = d q). Empirically we find element-wise multiplication does significantly better than the other two, which justifies our motivation to “filter” out document features which are irrelevant to the query.
At the bottom of Table 2 we show the effect of varying the number of hops K of the GA Reader on the final performance. We note that for K = 1, our model is equivalent to the AS Reader without any GA modules. We see a steep and steady rise in accuracy as the number of hops is increased from K = 1 to 3, which remains constant beyond
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Table 3: Validation/Test accuracy (%) on CNN, Daily Mail and CBT. Results marked with “†” are cf previously published works. Results marked with “‡” were obtained by training on a larger training set. Best performance on standard training sets is in bold, and on larger training sets in italics.
Model CNN Daily Mail CBT-NE CBT-CN
Val Test Val Test Val Test Val Test
Humans (query) † – – – – – 52.0 – 64.4 Humans (context + query) † – – – – – 81.6 – 81.6
LSTMs (context + query) † – – – – 51.2 41.8 62.6 56.0 Deep LSTM Reader † 55.0 57.0 63.3 62.2 – – – – Attentive Reader † 61.6 63.0 70.5 69.0 – – – – Impatient Reader † 61.8 63.8 69.0 68.0 – – – – MemNets † 63.4 66.8 – – 70.4 66.6 64.2 63.0 AS Reader † 68.6 69.5 75.0 73.9 73.8 68.6 68.8 63.4 DER Network † 71.3 72.9 – – – – – – Stanford AR (relabeling) † 73.8 73.6 77.6 76.6 – – – – Iterative Attentive Reader † 72.6 73.3 – – 75.2 68.6 72.1 69.2 EpiReader † 73.4 74.0 – – 75.3 69.7 71.5 67.4 AoA Reader † 73.1 74.4 – – 77.8 72.0 72.2 69.4 ReasoNet † 72.9 74.7 77.6 76.6 – – – – NSE † – – – – 78.2 73.2 74.3 71.9 BiDAF † 76.3 76.9 80.3 79.6 – – – –
MemNets (ensemble) † 66.2 69.4 – – – – – – AS Reader (ensemble) † 73.9 75.4 78.7 77.7 76.2 71.0 71.1 68.9 Stanford AR (relabeling,ensemble) † 77.2 77.6 80.2 79.2 – – – – Iterative Attentive Reader (ensemble) † 75.2 76.1 – – 76.9 72.0 74.1 71.0 EpiReader (ensemble) † – – – – 76.6 71.8 73.6 70.6
AS Reader (+BookTest) † ‡ – – – – 80.5 76.2 83.2 80.8 AS Reader (+BookTest,ensemble) † ‡ – – – – 82.3 78.4 85.7 83.7
GA-- 73.0 73.8 76.7 75.7 74.9 69.0 69.0 63.9 GA (update L(w)) 77.9 77.9 81.5 80.9 76.7 70.1 69.8 67.3 GA (fix L(w)) 77.9 77.8 80.4 79.6 77.2 71.4 71.6 68.0 GA (+feature, update L(w)) 77.3 76.9 80.7 80.0 77.2 73.3 73.0 69.8 GA (+feature, fix L(w)) 76.7 77.4 80.0 79.3 78.5 74.9 74.4 70.7
that. This is a common trend in machine learning as model complexity is increased, however we note that a multi-hop architecture is important to achieve a high performance for this task, and provide further evidence for this in the next section.
Lastly, we perform an ablation study for the three components of the GA Reader which were absent in the preprint version (GA Reader--). See Appendix B for more details.
4.4 Attention Visualization
To gain an insight into the reading process employed by the model we analyzed the attention distributions at intermediate layers of the reader. Figure 3 shows an example from the validation set of
WDW dataset (several more are in the Appendix). In each figure, the left and middle plots visualize attention over the query (equation 5) for candidates in the document after layers 1 & 2 respectively. The right plot shows attention over candidates in the document of cloze placeholder (XXX) in the query at the final layer. The full document, query and correct answer are shown at the bottom.
A generic pattern observed in these examples is that in intermediate layers, candidates in the document (shown along rows) tend to pick out salient tokens in the query which provide clues about the cloze, and in the final layer the candidate with the highest match with these tokens is
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Figure 2: Performance in accuracy with and without the Gated-Attention module over different training sizes. p-values for an exact one-sided Mcnemar’s test are given inside the parentheses for each setting.
0.66
0.68
0.7
0.72
0.74
0.76
0.78
50% (<0.01) 75% (<0.01) 100% (<0.01)
CNN (w/o qe-comm feature)
No Gating With Gating
0.66
0.68
0.7
0.72
0.74
0.76
0.78
50% (0.07)
75% (0.13)
100% (<0.01)
CNN (w qe-comm feature)
No Gating With Gating
0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7
50% (0.28)
75% (<0.01) 100% (<0.01)
WDW (w/o qe-comm feature)
No Gating With Gating
0.6 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7
50% (<0.01)
75% (0.42)
100% (0.27)
WDW (w qe-comm feature)
No Gating With Gating
selected as the answer. In Figure 3 there is a high attention of the correct answer on financial regulatory standards in the first layer, and on us president in the second layer. The incorrect answer, in contrast, only attends to one of these aspects, and hence receives a lower score in the final layer despite the n-gram overlap it has with the cloze token in the query. Importantly, different layers tend to focus on different tokens in the query, supporting the hypothesis that the multihop architecture of GA Reader is able to combine distinct pieces of information to answer the query.
5 Conclusion
We presented the Gated-Attention reader for answering cloze-style questions over documents. The GA reader features a novel multiplicative gating mechanism, combined with a multi-hop ar-
chitecture. Our model achieves the state-of-theart performance on several large-scale benchmark datasets with more than 4% improvements over competitive baselines. Our model design is backed up by an ablation study showing statistically significant improvements of using Gated Attention as information filters. We also showed empirically that multiplicative gating is superior to addition and concatenation operations for implementing gated-attentions, though a theoretical justification remains part of future research goals. Analysis of document and query attentions in intermediate layers of the reader further reveals that the model iteratively attends to different aspects of the query to arrive at the final answer. In this paper we have focused on text comprehension, but we believe that the Gated-Attention mechanism may benefit other tasks as well where multiple sources of information interact.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
B Ablation Study for Model Components
Table 6 shows accuracy on WDW by removing one component at a time.
Table 6: Ablation study on WDW dataset, without using the qe-comm feature and with fixed L(w). Results marked with † are cf (Onishi et al., 2016).
Model Accuracy
Val Test
GA 68.3 68.0 −char 66.9 66.9 −token-attentions (eq. 5) 65.7 65.0 −glove, +corpus 64.0 62.5
GA--† – 57
The steepest reduction is observed when we replace pretrained GloVe vectors with those pretrained on the corpus itself. GloVe vectors were trained on a large corpus of about 6 billion tokens (Pennington et al., 2014), and provide an important source of prior knowledge for the model. Note that the strongest baseline on WDW, NSE (Munkhdalai & Yu, 2016b), also uses pretrained GloVe vectors, hence the comparison is fair in that respect. Next, we observe a substantial drop when removing token-specific attentions over the query in the GA module, which allow gating individual tokens in the document only by parts of the query relevant to that token rather than the overall query representation. Finally, removing the character embeddings, which were only used for WDW and CBT, leads to a reduction of about 1% in the performance.
C Attention Plots
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
Table 4: Dataset statistics.
CNN Daily Mail CBT-NE CBT-CN WDW-Strict WDW-Relaxed
# train 380,298 879,450 108,719 120,769 127,786 185,978 # validation 3,924 64,835 2,000 2,000 10,000 10,000
# test 3,198 53,182 2,500 2,500 10,000 10,000 # vocab 118,497 208,045 53,063 53,185 347,406 308,602 max doc length 2,000 2,000 1,338 1,338 3,085 3,085
Table 5: Hyperparameter settings for each dataset. dim() indicates hidden state size of GRU.
Hyperparameter CNN Daily Mail CBT-NE CBT-CN WDW-Strict WDW-Relaxed
Dropout 0.2 0.1 0.4 0.4 0.3 0.3 dim( ←→ GRU∗) 256 256 128 128 128 128
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449

This paper presents a gated attention mechanism for machine reading.  A key idea is to extend Attention Sum Reader (Kadlec et al. 2016) to multi-hop reasoning by fine-grained gated filter.  It's interesting and intuitive for machine reading.  I like the idea along with significant improvement on benchmark datasets, but also have major concerns to get it published in ACL.  - The proposed GA mechanism looks promising, but not enough to convince the importance of this technique over other state-of-the-art systems, because engineering tricks presented 3.1.4 boost a lot on accuracy and are blended in the result.  - Incomplete bibliography: Nearly all published work in reference section refers arxiv preprint version.  This makes me (and future readers) suspicious if this work thoroughly compares with prior work. Please make them complete if the published version is available.   - Result from unpublished work (GA): GA baseline in table 1 and 3 is mentioned as previous work that is unpublished preprint.  I don't think this is necessary at all. Alternately, I would like the author to replace it with vanilla GA (or variant of the proposed model for baseline).  It doesn't make sense that result from the preprint which will end up being the same as this ACL submission is presented in the same manuscript.  For fair blind-review, I didn't search on arvix archive though.  - Conflict on table 1 and 2: GA-- (table 1) is the same as K=1(AS) in table 2, and GA (fix L(w)) is for K=3 in table 2.  Does this mean that GA-- is actually AS Reader?  It's not clear that GA-- is re-implementation of AS.  I assumed K=1 (AS) in table 2 uses also GloVe initialization and token-attention, but it doesn't seem in GA--.   - I wish the proposed method compared with prior work in related work section (i.e. what's differ from related work).  - Fig 2 shows benefit of gated attention (which translates multi-hop architecture), and it's very impressive. It would be great to see any qualitative example with comparison.