Identifying 1950s American Jazz Composers: Fine-Grained IsA Extraction via Modifier Composition
We present a method for populating fine-grained classes (e.g., “1950s American jazz composers”) with instances (e.g., Charles Mingus). While stateof-the-art methods tend to treat class labels as single lexical units, the proposed method considers each of the individual modifiers in the class label relative to the head. An evaluation on the task of reconstructing Wikipedia category pages demonstrates a >10 point increase in AUC, over a strong baseline relying on widely-used Hearst patterns.
1 000
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Substantial attention has been paid to automatically acquiring taxonomic knowledge, like that “Charles Mingus” is a “composer”, from text (Snow et al., 2006; Shwartz et al., 2016). The majority of approaches for extracting such “IsA” relations rely on lexical patterns as the primary signal of whether an instance belongs to a class: for example, observing a pattern like “X such as Y” is a strong indication that Y is an instance of class X (Hearst, 1992).
Methods based on these “Hearst patterns” assume that class labels can be treated as atomic lexicalized units. This assumption has several significant weakness. First, in order to recognize an instance of a class, these patternbased methods require that the entire class label be observed verbatim in text. The requirement is reasonable for class labels containing a single word, but in practice, there are many possible fine-grained classes: not only “composers” but also “1950s American jazz composers”. The probability that a given label will appear in its entirety within one of the
expected patterns is very low, even in large amounts of text. Second, when class labels are treated as though they cannot be decomposed, every class label must be modeled independently, even those containing overlapping words (“American jazz composer”, “French jazz composer”). As a result, the number of meaning representations to be learned is exponential in the length of the class label, and quickly becomes intractable. Thus, compositional models of taxonomic relations are necessary for better language understanding.
We introduce a compositional approach for reasoning about fine-grained class labels. Our approach is based on the notion from formal semantics in which modifiers (“1950s”) correspond to properties which differentiate instances of a subclass (“1950s composers”) from instances of the superclass (“composers”) (Heim and Kratzer, 1998). Our method consists of two stages: interpreting each modifier relative to the head (“composers active during 1950s”), and using the interpretations to identify instances of the class from text (Figure 1). Our main contributions are: 1) a compositional method for IsA extraction, which in-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
volves a novel application of noun-phrase paraphrasing methods to the task of semantic taxonomy induction and 2) the operationalization of a formal semantics framework to address two aspects of semantics that are often kept separate in NLP: assigning intrinsic “meaning” to a phrase, and reasoning about that phrase in a truth-theoretic context.
2 Related Work
Noun Phrase Interpretation. Compound noun phrases (“jazz composer”) communicate implicit semantic relations between modifiers and the head. Many efforts to provide semantic interpretations of such phrases rely on matching the compound to pre-defined patterns or semantic ontologies (Fares et al., 2015; Séaghdha and Copestake, 2007; Tratz and Hovy, 2010; Surtani and Paul, 2015; Choi et al., 2015). Recently, interpretations may take the form of arbitrary natural language predicates (Hendrickx et al., 2013). Most approaches are supervised, comparing unseen noun compounds to the most similar phrase seen in training (Wijaya and Gianfortoni, 2011; Nulty and Costello, 2013; Van de Cruys et al., 2013). Other unsupervised approaches apply information extraction techniques to paraphrase noun compounds (Kim and Nakov, 2011; Xavier and de Lima, 2014; Pasca, 2015). They focus exclusively on providing good paraphrases for an input noun compound. To our knowledge, ours is the first attempt to use these interpretations for the downstream task of IsA relation extraction.
Semantic Taxonomy Induction. Most efforts to learn taxonomic relations from text build on the seminal work of Hearst (1992), which observes that certain textual patterns– e.g., “X and other Y”–are high-precision indicators of whether X is a member of class Y. Recent work focuses on learning such patterns automatically from corpora (Snow et al., 2006; Shwartz et al., 2016). These IsA extraction techniques provide a key step for the more general task of knowledge base population. The “universal schema” approach (Riedel et al., 2013; Kirschnick et al., 2016; Verga et al., 2016), which infers relations using matrix factorization, often includes Hearst patterns as input features. Graphical (Bansal et al., 2014)
and joint inference models (Movshovitz-Attias and Cohen, 2015) typically require Hearst patterns to define an inventory of possible classes. A separate line of work avoids Hearst patterns by instead exploiting semi-structured data from HTML markup (Wang and Cohen, 2009; Dalvi et al., 2012; Pasupat and Liang, 2014). These approaches all share the limitation that, in practice, in order for a class to be populated with instances, the entire class label has to have been observed verbatim in text. This requirement limits the ability to handle arbitrarily fine-grained classes. Our work addresses this limitation by modeling fine-grained class labels compositionally. Thus the proposed method can combine evidence from multiple sentences, and can perform IsA extraction without requiring any example instances of a given class.1
3 Modifiers as Functions
Formalization. In formal semantics, modification is modeled as function application. Specifically, let MH be a class label consisting of a head H, which we assume to be a common noun, preceded by a modifier M . We use J·K to represent the “interpretation function” which maps a linguistic expression to its denotation in the world. The interpretation of a common noun is the set of entities2 in the universe U which are denoted by the noun (Heim and Kratzer, 1998):
JHK = {e ∈ U | e is a H} (1)
The interpretation of a modifier M is a function that maps between sets of entities. That is, modifiers select a subset3 of the input set:
JMK(H) = {e ∈ H | e satisfies M} (2)
This formalization leaves open how one decides whether or not “e satisfiesM”. This nontrivial, as the meaning of a modifier can vary depending on the class it is modifying: if e is a “good student”, e is not necessarily a “good
1Pasupat and Liang (2014) also focuses on zero-shot IsA extraction, but exploits HTML document structure, rather than reasoning compositionally.
2We use “entities” and “instances” interchangeably;“entities” is standard terminology in linguistics.
3As does virtually all previous work in information extraction, we assume that modifiers are subsective, acknowledging the limitations (Kamp and Partee, 1995).
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
person”, making it difficult to model whether “e satisfies good” in general. We therefore reframe the above equation, so that the decision of whether “e satisfies M” is made by calling a binary function φM , parameterized by the class H within which e is being considered:
JMK(H) = {e ∈ H | φM (H, e)} (3)
Conceptually, φM captures the core “meaning” of the modifier M , which is the set of properties that differentiate members of the output class MH from members of the more general input class H. This formal semantics framework has two important consequences. First, the modifier has an intrinsic “meaning”. The properties entailed by the modifier are independent of the particular state of the world. This makes it possible to make inferences about “1950s composers” even if no 1950s composers have been observed. Second, the modifier is a function that can be applied in a truth-theoretic setting. That is, applying “1950s” to the set of “composers” returns exactly the set of “1950s composers”.
Computational Approaches. While the notion of modifiers as functions has been incorporated into computational models previously, prior work focuses on either assigning an intrinsic meaning to M or on operationalizing M in a truth-theoretic sense, but not on doing both simultaneously. For example, Young et al. (2014) focuses exclusively on the subset selection aspect of modification. That is, given a set of instances H and a modifier M , their method could return the subset MH. However, their method does not model the meaning of the modifier itself, so that, e.g., if there were no red cars in their model of the world, the phrase “red cars” would have no meaning. In contrast, Baroni and Zamparelli (2010) models the meaning of modifiers explicitly as functions which map between vector-space representations of nouns. However, their model focuses on similarity between class labels–e.g., to say that “important routes” is similar to “major roads”–and it is not obvious how the method could be operationalized in order to identify instances of those classes. A contribution of our work is to model the semantics of M intrinsically, but in a way that permits application in the
model theoretic setting. We learn an explicit model of the “meaning” of a modifier M relative to a head H, represented as a distribution over properties which differentiate the members of the class MH from those of the class H. We then use this representation to identify the subset of instances of H which constitute the subclass MH.
4 Learning Modifier Interpretations
4.1 Setup
For each modifier M , we would like to learn the function φM from Eq. 3. Doing so makes it possible, given H and an instance e ∈ H, to decide whether e has the properties required to be an instance of MH. In general, there is no systematic way to determine the implied relation between M and H, as modifiers can arguably express any semantic relation, given the right context (Weiskopf, 2007). We therefore model the semantic relation between M and H as a distribution over properties which could potentially define the subclass MH ⊆ H. We will refer to this distribution as a “property profile” for M relative to H. We make the assumption that relations between M and H that are discussed more often are more likely to capture the important properties of the subclass MH. This assumption is not perfect (Section 4.4) but has given good results for paraphrasing noun phrases (Nakov and Hearst, 2013; Pasca, 2015). Our method for learning property profiles is based on the unsupervised method proposed by Pasca (2015), which uses query logs as a source of common sense knowledge, and rewrites noun compounds by matching MH (“American composers”) to queries of the form “H(.∗)M” (“composers from America”).
4.2 Inputs
We assume two inputs: 1) an IsA repository, O, containing 〈e, C〉 tuples where C is a category and e is an instance of C, and 2) a fact repository, D, containing 〈s, p, o, w〉 tuples where s and o are noun phrases, p is a predicate, and w is a confidence that p expresses a true relation between s and o. Both O and D are extracted from a sample of around 1 billion Web documents in English. The supplementary material gives additional details.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Good property profiles Bad property profiles rice dish French violinist Led Zeppelin song still life painter child actor risk manager * serve with rice * live in France Led Zeppelin write * * known for still life * have child * take risk * include rice * born in France Led Zeppelin play * * paint still life * expect child * be at risk * consist of rice * speak French Led Zeppelin have * still life be by * * play child * be aware of risk
Table 1: Example property profiles learned by observing predicates that relate instances of class H to modifier M (I2). Results are similar when using the class label H directly (I1). We spell out inverted predicates (Section 4.2) so wildcards (*) may appear as subjects or objects.
We instantiate O with an IsA repository constructed by applying Hearst patterns to the Web documents. Instances are represented as automatically-disambiguated entity mentions4 which, when possible, are resolved to Wikipedia pages. Classes are represented as (non-disambiguated) natural language strings. We instantiate D with a large repository of facts extracted using in-house implementations of ReVerb (Fader et al., 2011) and OLLIE (Mausam et al., 2012). The predicates are extracted as natural language strings. Subjects and objects may be either disambiguated entity references or natural language strings. Every tuple is included in both the forward and the reverse direction. E.g. 〈jazz, perform at, venue〉 also appears as 〈venue,←perform at, jazz〉, where ← is a special character signifying inverted predicates. These inverted predicates simplify the following definitions. In total, O contains 1.1M tuples and D contains 30M tuples.
4.3 Building Property Profiles
Properties. Let I be a function which takes as input a noun phrase MH and returns a property profile for M relative to H. We define a “property” to be an SPO tuple in which the subject position5 is a wildcard, e.g. 〈∗, born in,America〉. Any instance which fills the wildcard slot then “has” the property. We expand adjectival modifiers to encompass nominalized forms using a nominalization dictionary extracted from WordNet (Miller, 1995). If MH is “American composer” and we require a tuple to have the form 〈H, p,M,w〉, we will include tuples in which the third element is either “American” or “America”.
4“Entity mentions” may be individuals, like “Barack Obama”, but may also be concepts like “jazz”.
5Inverse predicates capture properties in which the wildcard is conceptually the object of the relation, but occupies the subject slot in the tuple. For example, 〈venue,←perform at, jazz〉 captures that a “jazz venue” is a “venue” e such that “jazz performed at e”.
Relating M to H Directly. We first build property profiles by taking the predicate and object from any tuple in D in which the subject is the head and the object is the modifier:
I1(MH) = {〈〈p,M〉, w〉 | 〈H, p,M,w〉 ∈ D} (4)
Relating M to an Instance of H. We also consider an extension in which, rather than requiring the subject to be the class label H, we require the subject to be an instance of H.
I2(MH) = {〈〈p,M〉, w〉 | 〈e,H〉 ∈ O ∧〈e, p,M,w〉 ∈ D}
(5)
Modifier Expansion. In practice, when building property profiles, we do not require that the object of the fact tuple match the modifier exactly, as suggested in Eq. 4 and 5. Instead, we follow Pasca (2015) and take advantage of facts involving distributionally similar modifiers. Specifically, rather than looking only at tuples in D in which the object matches M , we consider all tuples, but discount the weight proportionally to the similarity between M and the object of the tuple. Thus, I1 is computed as below:
I1(MH) = {〈〈p,M〉, w × sim(M,N)〉 | 〈H, p,N,w〉 ∈ D} (6)
where sim(M,N) is the cosine similarity between M and N . I2 is computed analogously. We compute sim using a vector space built from Web documents following Lin and Wu (2009); Pantel et al. (2009). We retain the 100 most similar phrases for each of∼10M phrases, and consider all other similarities to be 0.
4.4 Analysis of Property Profiles
Table 1 provides examples of good and bad property profiles for several MHs. In general, frequent relations between M and H capture relevant properties of MH, but it is not always
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Class label Property profile American company * based in America American composer * born in America American novel * written in America jazz album * features jazz jazz composer * writes jazz jazz venue jazz performed at *
Table 2: Head-specific property profiles learned by relating instances of H to the modifier M (I2). Results are similar using I1.
the case. To illustrate, the most frequently discussed relation between “child” and “actor” is that actors have children, but this property is not indicative of the meaning of “child actor”. Qualitatively, the top-ranked interpretations learned by using the head noun directly (I1, Eq. 4) are very similar to those learned using instances of the head (I2, Eq. 5). However, I2 returns many more properties (10 on average per MH) than I1 (just over 1 on average). Anecdotally, we see that I2 captures more specific relations than does I1. For example, for “jazz composers”, both methods return “* write jazz” and “* compose jazz”, but I2 additionally returns properties like “* be major creative influence in jazz”. We compare I1 and I2 quantitatively in Section 6. Importantly, we do see that both I1 and I2 are capable of learning head-specific property profiles for a modifier. Table 2 provides examples.
5 Class-Instance Identification
Instance finding. After finding properties that relate a modifier to a head, we turn to the task of identifying instances of fine-grained classes. That is, for a given modifier M , we want to instantiate the function φM from Eq. 3. In practice, rather than being a binary function which decides whether or not e is in class MH, our instantiation, φ̂M , will return a realvalued score expressing the confidence that e is a member of MH. For notational convenience, let D(〈s, p, o〉) = w, if 〈s, p, o, w〉 ∈ D and 0 otherwise. We define φ̂M as follows:
φ̂M (H, e) = ∑
〈〈p,o〉,w〉∈I(MH)
w ×D(〈e, p, o〉)
(7) Applying M to H, then, is as in Eq. 3 except that instead of a discrete set, it returns a scored list of candidate instances:
JMK(H) = {〈e, φ̂M (H, e)〉 | 〈e,H〉 ∈ O} (8)
Ultimately, we need to identify instances of arbitrary class labels, which may contain multiple modifiers. Given a class label C = M1 . . .MkH which contains a head H preceded by modifiers M1 . . .Mk, we generate a list of candidate instances by finding all instances of H which have some property to support every modifier:
k⋂ i=1 {〈e, s(e)〉 | 〈e, w〉 ∈ JMiK(H) ∧w > 0} (9)
where s(e) is the mean6 of the scores assigned by each separate φ̂Mi . From here on, we use Mods to refer to our method which generates lists of instances for a class using Eq. 8 and 9. When φ̂M (Eq. 7) is implemented using I1, we use the name ModsH (for “heads”). When it is implemented using I2, we use the name ModsI (for “instances”).
Weakly Supervised Reranking. Eq. 8 uses a naive ranking in which the weight for e ∈MH is the product of how often e has been observed with some property and the weight of that property for the class MH. Thus, instances of H with overall higher counts in D receive high weights for every MH. We therefore train a simple logistic regression model to predict the likelihood that e belongs to MH. We use a small set of features7, including the raw weight as computed in Eq. 7. For training, we sample 〈e, C〉 pairs from our IsA repository O as positive examples and random pairs that were not extracted by any Hearst pattern as negative examples. We frame the task as a binary prediction of whether e ∈ C, and use the model’s confidence as the value of φ̂M in place of the function in Eq. 7.
6 Evaluation
6.1 Experimental Setup
Evaluation Sets. We evaluate our models on their ability to return correct instances for arbitrary class labels. As a source of evaluation data, we use Wikipedia category pages8. These are pages in which the title is the name of the category (e.g., “pakistani film actresses”) and the body is a manually
6Also tried minimum, but mean gave better results. 7Feature templates in supplementary material. 8http://en.wikipedia.org/wiki/Help:Category
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
2008 california wildfires · australian army chaplains · australian boy bands · canadian business journalists · canadian military nurses · canberra urban places · cellular automaton rules · chinese rice dishes · coldplay concert tours · daniel libeskind designs · economic stimulus programs · german film critics · invasive amphibian species · log flume rides · malayalam short stories · pakistani film actresses · puerto rican sculptors · string theory books · tampa bay devil rays scouts
Table 3: Examples of labels from UniformSet.
curated list of links to other pages which fall under the category. We measure the precision and recall of each method for discovering the instances listed on these pages given the page title (henceforth “class label”).
We collect the titles of all Wikipedia category pages, removing those in which the last word is capitalized or which contain fewer than three words. These heuristics are intended to retain compositional titles in which the head is a single common noun. We also remove any titles which contain links to sub-categories. This is to favor fine-grained classes (“pakistani film actresses”) over coarse-grained ones (“film actresses”). We perform heuristic modifier chunking in order to group together multiword modifiers (e.g., “puerto rican”); for details, see supplementary material. From the resulting list of class labels, we draw two samples of 100 labels each, enforcing that no H appear as the head of more than three class labels per sample. The first sample is chosen uniformly at random (denoted UniformSet). The second (WeightedSet) is weighted so that the probability of drawing M1 . . .MkH is proportional to the total number of class labels in whichH appears as the head. Available at http://anonymized, these different evaluation sets are intended to evaluate performance on the head versus the tail of class label distribution, since information retrieval methods often perform differently on different parts of the distribution. On average, there are 17 instances per category in UniformSet and 19 in WeightedSet. Table 3 gives example class labels from UniformSet.
Baselines. We implement two baselines using our IsA repository (O as defined in Section 4.1). Our simplest baseline ignores modifiers altogether, and simply assumes that any instance of H is an instance of MH, regardless of M . In this case the confidence value for 〈e,MH〉 is equivalent to that for 〈e,H〉. We
refer to this baseline simply as Baseline. Our second, stronger baseline uses the IsA repository directly to identify instances of the finegrained class C = M1 . . .MkH. That is, we consider e to be an instance of the class if 〈e, C〉 ∈ O, meaning the entire class label appeared in a source sentence matching some Hearst pattern. We refer to this baseline as Hearst. The weight used to rank the candidate instances is the confidence value assigned by the Hearst pattern extraction (Section 4.2).
Compositional Models. As a baseline compositional model, we augment the Hearst baseline via set intersection. Specifically, for a class C = M1 . . .MkH, if each of the MiH appears in O independently, we take the instances of C to be the intersection of the instances of each of the MiH. We assign the weight of an instance e to be the sum of the weights associated with each independent modifier. We refer to this method as Hearst∩. We contrast this with our proposed model which recognizes instances of a fine-grained class by 1) assigning a meaning to each modifier in the form of a property profile and 2) checking whether a candidate instance exhibits these properties. We refer to the versions of our method as ModsH and ModsI , as described in Section 5. When relevant, we use “raw” to refer to the version in which instances are ranked using raw weights and “RR” to refer to the version in which instances are ranked using logistic regression (Section 5). We also try using the proposed methods to extend rather than replace the Hearst baseline. We combine predictions by merging the ranked lists produced by each system: i.e. the score of an instance is the inverse of the sum of its ranks in each of the input lists. If an instance does not appear at all in an input list, its rank in that list is set to a large constant value. We refer to these combination systems as Hearst+ModsH and Hearst+ModsI .
6.2 Results
Precision and Coverage. We first compare the methods in terms of their coverage, the number of class labels for which the method is able to find some instance, and their precision, to what extent the method is able to correctly rank true instances of the class above
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Flemish still life painters: Clara Peeters · Willem Kalf · Jan Davidsz de Heem · Pieter Claesz · Peter Paul Rubens · Frans Snyders · Jan Brueghel the Elder · Hans Memling · Pieter Bruegel the Elder · Caravaggio · Abraham Brueghel Pakistani cricket captains: Salman Butt · Shahid Afridi · Javed Miandad · Azhar Ali · Greg Chappell · Younis Khan · Wasim Akram · Imran Khan · Mohammad Hafeez · Rameez Raja · Abdul Hafeez Kardar · Waqar Younis · Sarfraz Ahmed Thai buddhist temples: Wat Buddhapadipa · Wat Chayamangkalaram · Wat Mongkolratanaram · Angkor Wat · Preah Vihear Temple ·Wat Phra Kaew ·Wat Rong Khun ·Wat Mahathat Yuwaratrangsarit · Vat Phou · Tiger Temple · Sanctuary of Truth · Wat Chalong · Swayambhunath · Mahabodhi Temple · Tiger Cave Temple · Harmandir Sahib
Table 4: Instances extracted for several fine-grained classes from Wikipedia. Lists shown are from ModsI . Instances in italics were also returned by Hearst∩. Strikethrough denotes incorrect.
non-instances. We report total coverage, the number of labels for which the method returns any instance, and correct coverage, the number of labels for which the method returns a correct instance. For precision, we compute the average precision (AP) for each class label. AP ranges from 0 to 1, where 1 indicates that all positive instances were ranked above all negative instances. We report mean average precision (MAP), which is the mean of the APs across all the class labels. MAP is only computed over class labels for which the method returns something, meaning methods are not punished for returning empty lists.
UniformSet WeightedSet Coverage MAP Coverage MAP
Baseline 95 / 70 0.01 98 / 74 0.01 Hearst 9 / 9 0.63 8 / 8 0.80 Hearst∩ 13 / 12 0.62 9 / 9 0.80 ModsH raw 56 / 32 0.23 50 / 30 0.16 ModsH RR 56 / 32 0.29 50 / 30 0.25 ModsI raw 62 / 36 0.18 59 / 38 0.20 ModsI RR 62 / 36 0.24 59 / 38 0.23
Table 5: Coverage and precision for populating Wikipedia category pages with instances. “Coverage” is the number of class labels (out of 100) for which at least one instance was returned, followed by the number for which at least one correct instance was returned. “MAP” is mean average precision. MAP does not punish methods for returning empty lists, thus favoring the baseline (see Figure 2).
Table 4 gives examples of instances returned for several class labels and Table 5 shows the precision and coverage for each of the methods. Figure 2 illustrates how the single mean AP score (as reported in Table 5) can misrepresent the relative precision of different methods. In combination, Table 5 and Figure 2 demonstrate that the proposed methods extract instances about as well as the baseline, whenever the baseline can extract anything at all; i.e. the proposed method does not cause a precision drop on classes covered by the base-
line. In addition, there are many classes for which the baseline is not able to extract any instances, but the proposed method is.
Table 5 also reveals that the reranking model (RR) consistently increases MAP for the proposed methods. Therefore, going forward, we only report results using the reranking model (i.e. ModsH and ModsI will refer to ModsH RR and ModsI RR, respectively).
Manual Re-Annotation. It possible that true instances of a class are missing from our Wikipedia reference set, and thus that our precision scores underestimate the actual precision of the systems. We therefore manually verify the top 10 predictions of each of the systems for a random sample of 25 class labels. We choose class labels for which Hearst was able to return at least one instance, in order to ensure reliable precision estimates. For each of these labels, we manually check the top 10 instances proposed by each method to determine whether each belongs to the class. Table 6 shows the precision scores for each method computed against the
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
(a) Uniform random sample (UniformSet). (b) Weighted random sample (WeightedSet).
Figure 3: ROC curves for selected methods (Hearst in blue, proposed in red). Given a ranked list of instances, ROC curves plot true positives vs. false positives retained by setting various cutoffs. The curve becomes linear once all remaining instances have the same score (e.g., 0), as this makes it impossible to add true positives without also including all remaining false positives.
original Wikipedia list of instances and against our manually-augmented list of gold instances. The overall ordering of the systems does not change, but the precision scores increase notably after re-annotation. We continue to evaluate against the Wikipedia lists, but acknowledge that reported precision is likely an underestimate of true precision.
Wikipedia Gold Hearst 0.56 0.79 Hearst∩ 0.53 0.78 ModsH 0.23 0.39 ModsI 0.24 0.42 Hearst+ModsH 0.43 0.63 Hearst+ModsI 0.43 0.63
Table 6: P@10 before/after re-annotation; Wikipedia underestimates true precision.
Precision-Recall Analysis. We next look at the precision-recall tradeoff in terms of the area under the curve (AUC) achieved when each method attempts to rank the complete list of candidate instances. We take the union of all of the instances proposed by all of the methods (including the Baseline method which, given a class label M0 . . .MkH, proposes every instance of the head H as a candidate). Then, for each method, we rank this full set of candidates such that any instance returned by the method is given the score the method assigns, and every other instance is scored as 0. Table 7 reports the AUC and recall and Figure 3 plots the full ROC curves. The requirement by Hearst that class labels appear in full in a single sentence results in very low recall, which translates into very low AUC when considering the full set of candi-
date instances. By comparison, the proposed compositional methods make use of a larger set of sentences, and provide non-zero scores for many more candidates, resulting in a >10 point increase in AUC on both UniformSet and WeightedSet (Table 7).
UniformSet WeightedSet AUC Recall AUC Recall
Baseline 0.55 0.23 0.53 0.28 Hearst 0.56 0.03 0.52 0.02 Hearst∩ 0.57 0.04 0.53 0.02 ModsH 0.68 0.08 0.60 0.06 ModsI 0.71 0.09 0.65 0.09 Hearst∩+ModsH 0.70 0.09 0.61 0.08 Hearst∩+ModsI 0.73 0.10 0.66 0.10
Table 7: Recall of instances on Wikipedia category pages, measured against the full set of instances from all pages in sample. AUC captures tradeoff between true and false positives.
7 Conclusion
We have presented an approach to IsA extraction which takes advantage of the compositionality of natural language. Existing approaches often treat class labels as atomic units which must be observed in full in order to be populated with instances. As a result, current methods are not able to handle the infinite number of classes describable in natural language, most of which never appear in text. Our method reasons about each modifier in the label individually, in terms of the properties that it implies about the instances. This approach allows us to harness information that is spread across multiple sentences, significantly increasing in the number of finegrained classes which we are able to populate.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:  - the model if theoretically solid and motivated by formal semantics.   - Weaknesses:   - The paper is about is-a relation extraction but the majority of literature about taxonomization is not referenced in the paper, inter alia:  Flati Tiziano, Vannella Daniele, Pasini Tommaso, Navigli Roberto. 2016. MultiWiBi: The multilingual Wikipedia bitaxonomy project.  Soren Auer, Christian Bizer, Georgi Kobilarov, Jens  u00a8 Lehmann, Richard Cyganiak, and Zachary Ive. 2007. DBpedia: A nucleus for a web of open data.  Gerard de Melo and Gerhard Weikum. 2010. MENTA: Inducing Multilingual Taxonomies from Wikipedia.  Zornitsa Kozareva and Eduard H. Hovy. 2010. A Semi-Supervised Method to Learn and Construct Taxonomies Using the Web.   Vivi Nastase, Michael Strube, Benjamin Boerschinger, Caecilia Zirn, and Anas Elghafari. 2010. WikiNet: A Very Large Scale Multi-Lingual Concept Network.  Simone Paolo Ponzetto and Michael Strube. 2007. Deriving a large scale taxonomy from Wikipedia.  Simone Paolo Ponzetto and Michael Strube. 2011. Taxonomy induction based on a collaboratively built knowledge repository.   Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2008. YAGO: A large ontology from Wikipedia and WordNet.   Paola Velardi, Stefano Faralli, and Roberto Navigli. 2013. OntoLearn Reloaded: A graph-based algorithm for taxonomy induction.    - Experiments are poor, they only compare against  Hearst patterns  without taking into account the works previously cited.  - General Discussion:  The paper is easy to follow and the supplementary material is also well written and useful, however the paper lack of references of is a relation extraction and taxonomization literature. The same apply for the experiments. In fact no meaningful comparison is performed and the authors not even take into account the existence of other systems (more recent than hearst patterns).  I read authors answers but still i'm not convinced that they couldn't perform more evaluations. I understand that they have a solid theoretical motivation but still, i think that comparison are very important to asses if the theoretical intuitions of the authors are confirmed also in practice. While it's true that all the works i suggested as comparison build taxonomies, is also true that a comparison is possible considering the edges of a taxonomy.  Anyway, considering the detailed author answer and the discussion with the other reviewer i can rise my score to 3 even if i still think that this paper is poor of experiments and does not frame correctly in the is-a relation extraction   taxonomy building literature.


Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection
The state-of-the-art named entity recognition (NER) systems are supervised machine learning models that require large amounts of human-annotated data to achieve high accuracy. However, annotating NER data by human is expensive and time-consuming, and can be quite difficult for a new language. In this paper, we present two approaches for weakly supervised cross-lingual NER with no human annotation in a target language. The first approach is to create weakly labeled NER data for a target language via annotation projection on comparable corpora, where we develop a heuristic scheme that effectively selects good-quality weakly labeled data from noisy data. The second approach is to project distributed representations of words (word embeddings) from a target language to a source language, so that the source-language NER system can be applied to the target language without re-training. We also design two co-decoding schemes that combine the outputs of the two projection-based approaches in a smart way. We evaluate the performance of the proposed approaches on both in-house and open NER data sets for a number of target languages. The results show that the combined systems outperform two state-of-the-art cross-lingual NER approaches on the CoNLL test data.
1 000
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Named entity recognition (NER) is a fundamental information extraction task that automatically detects named entities in text and classifies them into pre-defined entity types such as PERSON, ORGANIZATION, GPE (GeoPolitical Entities), EVENT,
LOCATION, DATE, etc. NER provides essential inputs for many information extraction applications, including relation extraction, entity linking, question answering and text mining. Building fast and accurate NER systems is a crucial step towards enabling large-scale automated information extraction and knowledge discovery on the huge volumes of electronic documents existing today.
The state-of-the-art NER systems are statistical machine learning models including maximum entropy Markov models (MEMMs) (McCallum et al., 2000), conditional random fields (CRFs) (Lafferty et al., 2001) and neural networks (Collobert et al., 2011; Lample et al., 2016). To achieve high accuracy, an NER system needs to be trained with a large amount of human-annotated data, and is often supplied with language-specific resources (gazetteers, word clusters). Annotating NER data by human is rather expensive and time-consuming, and can be quite difficult for a new language. This places a big challenge in building NER systems of multiple languages for supporting multilingual information extraction applications.
The difficulty of acquiring human-annotated multilingual NER data raises the following question: given a well-trained NER system in a source language (e.g., English), how can one go about extending it to a new language with decent performance and no human annotation in the target language? There are mainly two types of approaches for building weakly supervised cross-lingual NER systems.
The first type of approaches are based on annotation projection, e.g., (Zitouni and Florian, 2008; Ehrmann et al., 2011). These approaches require parallel corpora between a source language (usually English) and a target language with alignment information. The source-language sentences are labeled with NER tags, either by human as in (Ehrmann et al., 2011), or by a source-language NER system as in (Zitouni and Florian, 2008). The
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
NER tags are then projected to the target-language sentences via the alignment information. Note that the quality of the projection-labeled data heavily depends on the alignment accuracy of the parallel sentences, while this factor was not considered in the previous work when determining which sentence pairs should be selected for projection.
The second type of approaches are based on direct model transfer, e.g., (Täckström et al., 2012; Tsai et al., 2016). The basic idea is to train a single NER system in the source language with language-independent features, so the system can be applied to other languages using those universal features. Täckström et al. (2012) uses monolingual data in source/target languages and aligned parallel data between source and target languages to build cross-lingual word clusters. The crosslingual word clusters (together with part-of-speech tags) are then used to generate universal features. Tsai et al. (2016) applies the cross-lingual wikifier developed in (Tsai and Roth, 2016) and multilingual Wikipedia dump to generate languageindependent labels (FreeBase types and Wikipedia categories) for n-grams in a document, and those labels are used as universal features.
In this paper, we make the following contributions to weakly supervised cross-lingual NER with no human annotation in the target languages. First, for the annotation projection approach, we develop a heuristic data selection scheme that seeks to select good-quality projection-labeled NER data from comparable corpora. Experimental results show that the data selection scheme can significantly improve the accuracy of the targetlanguage NER system (compared with no data selection) when the alignment accuracy is low and the projection-labeled data are noisy.
Second, we propose a new approach for direct NER model transfer based on representation projection. It projects word representations in vector space (word embeddings) from a target language to a source language, to create a universal representation of the words in different languages. Under this approach, the NER system trained for the source language can be directly applied to the target language without the need for re-training.
Finally, we design two co-decoding schemes that combine the outputs (views) of the two projection-based systems to produce an output that is more accurate than the outputs of individual systems. We evaluate the performance of the pro-
posed approaches on both in-house and open NER data sets for a number of target languages. The results show that the combined systems outperform the state-of-the-art cross-lingual NER approaches proposed in Täckström et al. (2012) and Tsai et al. (2016) on the CoNLL NER test data (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003).
We organize the paper as follows. In Section 2 we introduce three NER models that are used in the paper. In Section 3 we present an annotation projection approach with effective data selection. In Section 4 we propose a representation projection approach for direct NER model transfer. In Section 5 we describe two co-decoding schemes that combine the outputs of two projection-based approaches in a smart way. We evaluate the performance of the proposed approaches in Section 6 and conclude the paper in Section 7.
2 NER Models
The NER task can be formulated as a sequence labeling problem: given a sequence of words x1, ..., xn, we want to infer the NER label li for each word xi, 1 ≤ i ≤ n. In this section we introduce three NER models that are used in the paper.
2.1 CRFs and MEMMs
Conditional random fields (CRFs) are a class of discriminative probabilistic graphical models that provide powerful tools for labeling sequential data (Lafferty et al., 2001). CRFs learn a conditional probability model pλ(l|x) from a set of labeled training data, where x = (x1, ...,xn) is a random sequence of input words, l = (l1, ..., ln) is the sequence of label variables (NER tags) for x, and l has certain Markov properties conditioned on x. Specifically, a general-order CRF with order o assumes that label variable li is dependent on a fixed number o of previous label variables li−1, ..., li−o, with the following conditional distribution:
pλ(l|x) = e ∑n i=1 ∑K k=1 λkfk(li,li−1,...,li−o,x)
Z(x) (1)
where fk’s are the feature functions, λk’s are the weights (parameters) of the feature functions, and Z(x) is the normalization constant. The exponential parametric form can be derived using the maximum entropy principle (Berger et al., 1996). When o = 1, we have a first-order CRF which is also known as a linear-chain CRF.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Given a set of labeled training data D = (x(j), l(j))j=1,...,N , the training of a CRF model is to find an optimal set of parameters λ∗ that maximize the conditional log-likelihood of the data:
λ∗ = arg max λ ΣNj=1 log pλ(l (j)|x(j)) (2)
Once we obtain λ∗, we can use the trained model pλ∗(l|x) to decode the most likely label sequence l∗ for any new input sequence of words x (via the Viterbi algorithm for example):
l∗ = arg max l pλ∗(l|x) (3)
A related conditional probability model, called maximum entropy Markov model (MEMM) (McCallum et al., 2000), assumes that l is a Markov chain conditioned on x:
pλ(l|x) = n∏ i=1 pλ(li|li−1, ..., li−o,x)
= n∏ i=1 e ∑K k=1 λkfk(li,li−1,...,li−o,x) Z(li−1, ..., li−o,x) (4)
The main difference between CRFs and MEMMs is that CRFs normalize the conditional distribution over the whole sequence as in (1), while MEMMs normalize the conditional distribution per token as in (4). As a result, CRFs can better handle the label bias problem (Lafferty et al., 2001). This benefit, however, comes at a price. The training time of order-o CRFs grows exponentially (O(Mo+1)) with the number of output labels M , which is typically slow even for moderate-size training data ifM is large. In contrast, the training time of order-o MEMMs is linear (O(M)) with respect to M independent of o, so it can handle larger training data with higher order of dependency. We have implemented a linear-chain CRF model and a general-order MEMM model.
2.2 Neural Networks
With the increasing popularity of distributed (vector) representations of words, neural network models have been recently applied to tackle many NLP tasks including NER (Collobert et al., 2011).
We have implemented a feedforward neural network model which maximizes the log-likelihood of the tag sequence similar to that of (Collobert et al., 2011). We use a locally normalized model (the conditional distribution is normalized per token as in MEMMs) and introduce context dependency by conditioning on the previously assigned
tags. We use the target word and surrounding context as features. We do not employ other common features such as gazetteers or character-level representations as such features might not be readily available or might not transfer to other languages.
We have deployed two neural network architectures. The first one (called NN1) uses the word embedding of a word as the input. We also devise a second architecture (called NN2) which adds a smoothing layer that computes the cosine similarity between a word embedding and a fixed set of prototype vectors (learned during training) and returns a weighted average of these prototype vectors as the input. In our experiments we find that with the smoothing layer, NN2 tends to have a more balanced precision and recall than NN1. Both networks have one hidden layer, with sigmoid and softmax activation functions on the hidden and output layers respectively. The two neural network models are depicted in Figure 1.
3 Annotation Projection Approach
Traditional annotation projection approaches require parallel corpora between a source language and a target language with alignment information. In this paper, we develop a heuristic scheme that can effectively select good-quality projectionlabeled NER data from noisy comparable corpora (not necessarily parallel corpora). We use English as the source language.
Suppose we have sentence pairs (X,Y) between English and a target language, where X includes N English sentences x(1), ...,x(N), Y includes N target-language sentences y(1), ...,y(N), and y(i) is aligned to x(i) via an alignment model, 1 ≤ i ≤ N . We use a sentence pair (x,y) as an example to illustrate how the annotation projection procedure works, where x = (x1, x2, ..., xs) is an English sentence, and y = (y1, y2, ..., yt) is a target-language sentence that is aligned to x.
Annotation Projection Procedure
1. Apply the English NER system on the En-
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
glish sentence x to generate the NER tags l = (l1, l2, ..., ls) for x.
2. Project the NER tags to the target-language sentence y using the alignment information. Specifically, if a sequence of English words (xi, ..., xi+p) is aligned to a sequence of target-language words (yj , ..., yj+q), and (xi, ..., xi+p) is recognized (by the English NER system) as an entity with NER tag l, then (yj , ..., yj+q) is labeled with l1. Let l′ = (l′1, l ′ 2, ..., l ′ t) be the projected NER
tags for the target-language sentence y.
We can apply the annotation projection procedure on all the sentence pairs (X,Y), to generate projected NER tags L′ for the target-language sentences Y. (Y, L′) are automatically labeled NER data with no human annotation in the target language. One can use those projection-labeled data to train an NER system in the target language. The quality of such weakly labeled NER data, and consequently the accuracy of the target-language NER system, depend on both 1) the accuracy of the English NER system, and 2) the alignment accuracy of the comparable corpora.
If some of the projection-labeled data have bad quality and we use all the data for weakly supervised learning, the accuracy of the target-language NER system will be adversely affected by those bad-quality data. We would like to design effective data selection schemes that can select goodquality projection-labeled data from noisy data, to improve the accuracy of the annotation projection approach for cross-lingual NER.
3.1 Data Selection Scheme
We first design a metric to measure the annotation quality of a projection-labeled sentence in the target language. We construct a frequency table T which includes all the entities in the projectionlabeled target-language sentences. For each entity e, T also includes the projected NER tags for e and the frequency T(e, l) that entity e is labeled with tag l. Table 1 shows a snapshot of the frequency table where the target language is Portuguese.
We use the frequency T(e, l) to measure the reliability of labeling entity e with tag l in the target language. The intuition is that if an entity e is labeled by a tag l with higher frequency than other
1If the IOB (Inside, Outside, Beginning) tagging format is used, then (yj , yj+1, ..., yj+q) is labeled with (B-l, I-l,...,I-l).
Entity Name NER Tag Frequency Estados Unidos GPE 0.853 Estados Unidos ORGANIZATION 0.143 Estados Unidos PEOPLE 0.001 Estados Unidos PRODUCT 0.001 Estados Unidos TITLEWORK 0.001 Estados Unidos EVENT 0.001
Table 1: A snapshot of the frequency table where the target language is Portuguese. Estados Unidos means United States. The correct NER tag for Estados Unidos is GPE which has the highest frequency in the weakly labeled data.
tags in the projection-labeled data, it is more likely that the annotation is correct, because the source (English) NER system and the alignment system are independent statistical models and it is less likely that both systems make consistent errors simultaneously generating the same wrong tag.
Based on the frequency scores, we calculate the quality score of a projection-labeled targetlanguage sentence y by averaging the frequency scores of the projected entities in the sentence:
q(y) = Σe∈yT(e, l
′(e))
n(y) (5)
where l′(e) is the projected NER tag for e, and n(y) is the total number of entities in sentence y.
We use q(y) to measure the annotation quality of sentence y. In addition, we use n(y) to measure the amount of annotation information contained in sentence y. Based on these two metrics, we design a heuristic data selection scheme which selects projection-labeled sentences in the target language that satisfy the following condition:
q(y) ≥ q; n(y) ≥ n (6)
where q is a quality score threshold and n is an entity number threshold. We can tune the two parameters to make tradeoffs among the annotation quality of the selected sentences, the annotation information contained in the selected sentences, and the total number of sentence selected.
3.2 Improvements
We evaluate the effectiveness of the data selection scheme via experiments for 4 target languages: Japanese, Korean, German and Portuguese. We have comparable corpora between English and each target language with alignment information (ranging from 2.0M to 5.5M tokens). For each
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Language (q, n) Training Size F1 Score Japanese (0, 0) 5.10M 41.1
(0.8, 4) 691K 55.0 Korean (0, 0) 4.72M 18.7 (0.9, 3) 246K 38.7 German (0, 0) 5.47M 65.7
(0.5, 3) 3.37M 66.4 Portuguese (0, 0) 2.17M 61.7
(0.5, 4) 1.46M 62.9
Table 2: Performance comparison of NER systems trained without data selection ((q, n) = (0, 0)) and with data selection (F1 score shown in bold).
target language, we also have a set of humanannotated NER data (ranging from 30K to 45K tokens) which are served as the test data for evaluating the target-language NER system.
The source (English) NER system is a linearchain CRF model which achieves an accuracy of 88.9 F1 score on an independent NER test set. The alignment systems between English and the target languages are maximum entropy models (Ittycheriah and Roukos, 2005), with an accuracy of 69.4 F1 score for Japanese, 62.0 F1 score for Korean, 76.1 F1 score for German and 88.0 F1 score for Portuguese, respectively, on independent alignment test sets.
For each target language, we compare an NER system trained with all the projection-labeled data with no data selection (i.e., q = 0 and n = 0 in the data selection scheme) and an NER system trained with projection-labeled data selected by the data selection scheme. For the data selection scheme, we have tried different threshold parameters q and n and reported the best numbers in Table 2. Both NER systems are 2nd-order MEMM models2 which use the same template of features.
The results are shown in Table 2. For different target languages, we use the same source (English) NER system for annotation projection, so the differences in the improvements are mainly due to the alignment accuracy of the comparable corpora between English and different target languages. When the alignment accuracy is low (e.g., as for Japanese and Korean) and hence the projection-labeled NER data are quite noisy, the proposed data selection scheme is very effective in selecting good-quality projection-labeled data and the improvement is significant: +13.9 F1 score for
2CRFs cannot handle training data with a few millions of words, since our NER system has over 50 entity types, and the training time of CRFs grows at least quadratically in the number of entity types.
Japanese and +20.0 F1 score for Korean.
4 Representation Projection Approach
In this section, we present a new approach for direct NER model transfer based on representation projection. Under this approach, we train a single English NER system that uses only word embeddings as input representations. We create mapping functions which can map words in any language into English and we simply use the English NER system to decode. In particular, by mapping all languages into English, we are using one universal NER system and we do not need to re-train when a new language is added.
4.1 Monolingual Word Embeddings
We first build vector representations of words (word embeddings) for a language from monolingual data. We have developed a variant of the CBOW word2vec model (Mikolov et al., 2013a), which concatenates the context words surrounding a target word x using a weight ( 1dist(x,xc) ) that decays with the distance of a context word xc to x.
We train 300-dimensional word embeddings for English. Following (Mikolov et al., 2013b), we use larger dimensional embeddings for the target languages, namely 800. We train word2vec for 1 epoch for English/Spanish and 5 epochs for the rest of the languages for which we have less data.
4.2 Cross-Lingual Embedding Mappings
We learn cross-lingual word embedding mappings similarly to (Mikolov et al., 2013b). For a target language f , we first extract a small training dictionary from a phrase table that includes word-to-word alignments between English and the target language. The dictionary contains English and target-language word pairs with weights: (xi, yi, wi)i=1,...,n, where xi is an English word, yi is a target-language word, and the weight wi = P (xi|yi) is the empirical probability of xi given yi as extracted from the phrase table.
Given monolingual word embeddings for English and the target language f , where ui ∈ Rd1 is the vector representation for English word xi, vi ∈ Rd2 is the vector representation for target word yi, we find a linear mapping M f e by solving the following weighted least squares problem using the dictionary as the training data:
Mfe = arg min M Σni=1wi||ui −Mvi||2 (7)
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
where we generalize the formulation in (Mikolov et al., 2013b) by adding weights to the word pairs, so that more frequent translation pairs are of higher importance. Using Mfe , for any new word in the target language with vector representation v, we can project it into English as the vector Mfev.
The training dictionary plays a key role in finding an effective cross-lingual embedding mapping. To control the size of the dictionary, we only include word pairs with a minimum frequency threshold. We set the threshold to obtain approximately 5K to 6K unique word pairs for a target language, as experiments show that larger dictionaries might harm the performance of representation projection for direct NER model transfer.
4.3 Direct NER Model Transfer
The source (English) NER system is a neural network model (with architecture NN1 or NN2) that uses only word embedding features (embeddings of a word and its surrounding context) in the English vector space. Model transfer is achieved simply by projecting the target language word embeddings into the English vector space and decoding these using the English NER system.
More specifically, given the word embeddings of a sequence of words in a target language f , (v1, ...,vt), we project them into the English vector space by applying the cross-lingual linear mapping Mfe : (M f ev1, ...,M f evt). The English NER system is then applied on the projected input to produce the NER tags. Words which are not in the target vocabulary are projected into their English embeddings if found in the English vocabulary, or into an NER-trained unk vector otherwise.
4.4 Related Work
Many ways of obtaining cross-lingual embeddings have been proposed in the literature, following one of two main approaches. On approach builds monolingual representations separately and then brings them to the same space typically using a seed dictionary (Mikolov et al., 2013b; Faruqui and Dyer, 2014). Another line of work builds inter-lingual representations simultaneously, often by generating mixed language corpora using the supervision at hand (aligned sentences, documents, etc.) (Vulić and Moens, 2015; Gouws et al., 2015). We opt for the first solution in this paper because of its flexibility: we can map all languages to English rather than requiring separate embeddings for each language pair. Additionally we are
able to easily add a new language without any constraints on the type of data needed. Note that although we do not specifically create inter-lingual representations, by training mappings to the common language, English, we are able to map words in different languages to a common space.
Similar approaches for cross-lingual model transfer have been applied to other tasks such as document classification, syntactic parsing and part-of-speech tagging (Klementiev et al., 2012; Guo et al., 2015; Gouws and Søgaard, 2015). We expect these ideas to carry over to the NER task, in particular due to the semantic nature of the task which may allow for more natural embeddingbased cross-lingual transfer.
5 Co-Decoding
We have developed two weakly supervised systems for cross-lingual NER, which are trained with different data using different models (MEMM model for annotation projection and neural network model for representation projection). We would like to design a co-decoding scheme that can combine the outputs (views) of the two systems to produce an output that is more accurate than the outputs of individual systems.
Since both systems are statistical models and can produce confidence scores (probabilities), a natural co-decoding scheme is to compare the confidence scores of the NER tags generated by the two systems and select the tags with higher confidences scores. However, confidence scores of two weakly trained systems may not be directly comparable, especially when comparing O tags with non-O tags. We consider an exclude-O confidence-based co-decoding scheme which we find to be more effective empirically. It is similar to the pure confidence-based scheme, with the only difference that it prefers a non-O tag to an O tag independent of the confidence scores.
In experiments we find that the annotation projection system tends to have a high precision and low recall, i.e., it detects few entities, but for the detected entities the accuracy is high. The representation projection system tends to have a more balanced precision and recall. Based on this observation, we develop a rank-based co-decoding scheme that gives higher priority to the annotation projection system. The scheme will first include all the entities detected by the annotation projection system. Then, it will add all the entities de-
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Japanese Approach P R F1
Annotation-Projection (AP) 71.3 44.7 55.0 Representation-Projection (NN1) 71.5 36.6 48.4 Representation-Projection (NN2) 59.9 42.4 49.7 Co-Decoding (Conf): AP+NN1 66.2 49.7 56.7 Co-Decoding (Rank): AP+NN1 69.4 52.1 59.6 Co-Decoding (Conf): AP+NN2 60.5 53.8 56.9 Co-Decoding (Rank): AP+NN2 63.2 55.9 59.3
Supervised (272K) 84.5 80.9 82.7 Korean
Approach P R F1 Annotation-Projection (AP) 69.3 26.9 38.7 Representation-Projection (NN1) 61.1 19.3 29.3 Representation-Projection (NN2) 59.3 37.1 45.6 Co-Decoding (Conf): AP+NN1 62.0 32.7 42.8 Co-Decoding (Rank): AP+NN1 66.9 35.1 46.0 Co-Decoding (Conf): AP+NN2 59.1 44.6 50.8 Co-Decoding (Rank): AP+NN2 60.7 45.2 51.8
Supervised (90K) 72.5 44.0 54.8 German
Approach P R F1 Annotation-Projection (AP) 77.6 58.0 66.4 Representation-Projection (NN1) 69.0 48.8 57.2 Representation-Projection (NN2) 63.7 66.1 64.9 Co-Decoding (Conf): AP+NN1 69.1 60.3 64.4 Co-Decoding (Rank): AP+NN1 73.3 63.6 68.1 Co-Decoding (Conf): AP+NN2 64.9 71.1 67.8 Co-Decoding (Rank): AP+NN2 66.9 72.0 69.3
Supervised (125K) 77.8 68.1 72.6 Portuguese
Approach P R F1 Annotation-Projection (AP) 84.4 50.1 62.9 Representation-Projection (NN1) 70.5 47.6 56.8 Representation-Projection (NN2) 66.0 63.4 64.7 Co-Decoding (Conf): AP+NN1 72.0 55.8 62.8 Co-Decoding (Rank): AP+NN1 77.7 60.0 67.7 Co-Decoding (Conf): AP+NN2 68.5 67.4 68.0 Co-Decoding (Rank): AP+NN2 71.3 68.6 69.9
Supervised (173K) 79.8 71.9 75.6
Table 3: In-house NER test data: Precision, recall, F1 score on exact phrasal matches. The highest F1 score among all weakly supervised approaches is shown in bold. Same for Tables 4 and 5.
tected by the representation projection system that do not conflict3 with entities detected by the annotation projection system (to improve recall).
6 Experiments
In this section, we evaluate the performance of the proposed approaches for cross-lingual NER, including the 2 projection-based approaches and the 2 co-decoding schemes for combining them: (1) The annotation projection approach (AP) with heuristic data selection;
3Two entities detected by two different systems conflict with each other if either 1) the two entities have different boundaries but overlap with each other; or 2) the two entities have the same boundary but different NER tags.
(2) The representation projection approach (with two neural network architectures NN1 and NN2); (3) The exclude-O confidence-based co-decoding scheme; (4) The rank-based co-decoding scheme.
6.1 NER Data Sets
We have used various NER data sets for evaluation. The first group includes in-house humanannotated NER data for four languages: Japanese, Korean, German and Portuguese. The data have over 50 entity types. The main motivation of deploying such a fine-grained entity type set is to build cognitive question answering applications on top of the NER systems. The sizes of test data sets are ranging from 30K to 45K tokens.
The second group includes open humanannotated NER data for Spanish, Dutch and German from the CoNLL 02/03 development/test data sets (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003). The CoNLL data have 4 entity types: PER, ORG, LOC and MISC. The sizes of the development/test data sets are ranging from 35K to 70K tokens. The development data are used for tuning the parameters of the models.
6.2 Evaluation for In-House NER Data
In Table 3 we show the results of different approaches for the in-house NER data. For the annotation projection approach, the source (English) NER system is a linear-chain CRF model trained with 320K tokens of human-annotated English newswire data, and the target-language NER system is a 2nd-order MEMM model trained with 691K, 246K, 3.37M and 1.46M tokens of projection-labeled data for Japanese, Korean, German and Portuguese, respectively. For representation projection, the source (English) NER systems are neural network models with architectures NN1 and NN2 (Figure 1), both trained with 320K tokens of human-annotated English newswire data.
The results show that the annotation projection approach (AP) has a relatively high precision and low recall. For representation projection, neural network model NN2 (with a smoothing layer) is better than NN1, and NN2 tends to have a more balanced precision and recall. The rank-based codecoding scheme is more effective for combining the two projection-based approaches. In particular, the rank-based co-decoding scheme that combines AP and NN2 achieves the highest F1 score among all the weakly supervised approaches for
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Spanish Approach P R F1
Annotation-Projection (AP) 64.4 55.9 59.8 Representation-Projection (NN1) 63.9 52.2 57.4 Representation-Projection (NN2) 55.3 51.8 53.5 Co-Decoding (Conf): AP+NN1 62.6 64.1 63.4 Co-Decoding (Rank): AP+NN1 61.9 62.9 62.4 Co-Decoding (Conf): AP+NN2 56.8 61.4 59.0 Co-Decoding (Rank): AP+NN2 58.7 61.6 60.2
Supervised (264K) 81.3 79.8 80.6 Dutch
Approach P R F1 Annotation-Projection (AP) 71.1 60.6 65.4 Representation-Projection (NN1) 82.6 47.4 60.3 Representation-Projection (NN2) 66.3 43.5 52.5 Co-Decoding (Conf): AP+NN1 69.9 63.8 66.7 Co-Decoding (Rank): AP+NN1 70.7 62.8 66.5 Co-Decoding (Conf): AP+NN2 63.3 61.8 62.5 Co-Decoding (Rank): AP+NN2 67.7 63.3 65.4
Supervised (199K) 82.9 81.7 82.3 German
Approach P R F1 Annotation-Projection (AP) 73.7 52.9 61.6 Representation-Projection (NN1) 79.4 41.4 54.4 Representation-Projection (NN2) 64.6 42.7 51.4 Co-Decoding (Conf): AP+NN1 71.1 57.6 63.7 Co-Decoding (Rank): AP+NN1 72.4 57.7 64.2 Co-Decoding (Conf): AP+NN2 64.6 58.1 61.2 Co-Decoding (Rank): AP+NN2 68.0 59.1 63.2
Supervised (206K) 81.2 64.3 71.8
Table 4: CoNLL NER development data.
Korean, German and Portuguese (second highest F1 score for Japanese), and it improves over the best of the two projection-based systems by 2.9 to 6.2 F1 score.
We also provide the performance of supervised learning where the NER system is trained with human-annotated NER data in the target language (with size shown in the bracket). While in general the performance of the weakly supervised approaches is not as good as supervised learning, the best combined system achieves an F1 score close to the supervised Korean/German system (trained with 90K/125K tokens of human-annotated data).
6.3 Evaluation for CoNLL NER Data
For the CoNLL data, the source (English) NER system for annotation projection is a linear-chain CRF model trained with the CoNLL English training data (∼200K tokens), and the targetlanguage NER system is a 2nd-order MEMM model trained with 2.26M, 3.02M and 1.17M tokens of projection-labeled data for Spanish, Dutch and German, respectively. For representation projection, the source (English) NER systems are neural network models (NN1 and NN2) both
trained with the CoNLL English training data. In Table 4 we show the results for the CoNLL development data. For representation projection, neural network model NN1 is better than NN2. Both the annotation projection approach and NN1 tend to have a high precision. In this case, the confidence-based co-decoding scheme that combines AP and NN1 achieves the highest F1 score for Spanish and Dutch (second highest F1 score for German), and improves over the best of the two projection-labeled systems by 1.3 to 3.6 F1 score.
In Table 5 we compare our top systems (confidence or rank-based co-decoding of AP and NN1, determined by the development data) with the best results of the cross-lingual NER approaches proposed in Täckström et al. (2012) and Tsai et al. (2016) on the CoNLL test data. Our systems outperform the previous two approaches, especially for German, and are closer to supervised learning.
7 Conclusion
In this paper we developed two weakly supervised approaches for cross-lingual NER based on effective annotation and representation projection. We also designed two co-decoding schemes that combine the two projection-based systems in a smart way. Experimental results show that the combined system achieves higher accuracy than both systems and outperforms two state-of-the-art crosslingual NER approaches, providing a strong baseline for building cross-lingual NER systems with no human annotation in the target languages.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper describes a model for cross-lingual named entity recognition (NER). The authors employ conditional random fields, maximum entropy Markov, and neural network-based NER methods. In addition, authors propose two methods to combine the output of those methods (probability-based and ranking-based), and a method to select the best training instances from cross-lingual comparable corpora. The cross-lingual projection is done using a variant of Mikolov  proposal. In general, the paper is easy to follow, well-structured, and the English quality is also correct. The results of the combined annotations are interesting.  Detailed comments:  I was wondering which is the motivation behind proposing a Continuous Bag-of-word (CBOW) model variation. You don t give much details about this (or the parameters employed). Was the original model (or the Continuous Skip-gram model) offering low results? I suggest to include also the results with the CBOW model, so readers can analyse the improvements of your approach. Since you use a decay factor for the surrounding embeddings, I suggest to take a look to the exponential decay used in [1].  Similarly to the previous comment, I would like to look at the differences between the original Mikolov  cross-lingual projections and your frequency weighted projections. These contributions are more valuable if readers can see that your method is really superior.   the proposed data selection scheme is very effective in selecting good-quality projection-labeled data and the improvement is significant d  u2190 Have you conducted a test of statistical significance? I would like to know if the differences between result in this work are significant.   I suggest to integrate the text of Section 4.4 at the beginning of Section 4.2. It would look cleaner. I also recommend to move the evaluation of Table 2 to the evaluation section.  I miss a related work section. Your introduction includes part of that information. I suggest to divide the introduction in two sections.  The evaluation is quite short (1.5 pages with conclusion section there). You obtain state-of-the-art results, and I would appreciate more discussion and analysis of the results.  Suggested references:  [1] Iacobacci, I., Pilehvar, M. T., & Navigli, R. (2016). Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol. 1, pp. 897-907).