The ambitious goal of this work is to develop a cross-lingual name tagging and linking framework for 282 languages that exist in Wikipedia. Given a document in any of these languages, our framework is able to identify name mentions, assign a coarse-grained or fine-grained type to each mention, and link it to an English Knowledge Base (KB) if it is linkable. We achieve this goal by performing a series of new KB mining methods: generating “silver-standard” annotations by transferring annotations from English to other languages through crosslingual links and KB properties, refining annotations through self-training and topic selection, deriving language-specific morphology features from anchor links, and mining word translation pairs from crosslingual links. Both name tagging and linking results for 282 languages are promising on Wikipedia data. The results on non-Wikipedia data (news articles and discussion forum posts) are also comparable to the supervised models trained from manually annotated documents including thousands of names. All the data sets, resources and systems for 282 languages will be made publicly available as a new benchmark.
1 Introduction
Information provided in languages which people can understand saves lives in crises. For example, language barrier was one of the main difficulties faced by humanitarian workers responding to the Ebola crisis in 2014. We propose to break language barriers by extracting information (e.g.,
entities) from a massive variety of languages and ground the information into an existing knowledge base which is accessible to a user in his/her own language (e.g., a reporter from the World Health Organization who speaks English only).
✤Wikipedia Article: Mao Zedong (d. 26 Aralık 1893 - ö. 9 Eylül 1976), Çinli devrimci ve siyasetçi. Çin Komünist Partisinin (ÇKP) ve Çin Halk Cumhuriyetinin kurucusu.
✤Wikipedia Markup: [[Mao Zedong]] (d. [[26 Aralık]] [[1893]] - ö. [[9 Eylül]] [[1976]]), Çinli devrimci ve siyasetçi. [[Çin Komünist Partisi]]nin (ÇKP) ve [[Çin Halk Cumhuriyeti]]nin kurucusu.
KB Properties (e.g., DBpedia, YAGO)
formationDate headquarter
ideology …
(Mao Zedong (December 26, 1893 - September 9, 1976) is a Chinese revolutionary and politician. The founder of the Chinese Communist Party (CCP) and the People's Republic of China.)
tr/Çin_Komünist_Partisi Anchor Link en/Communist_Party_of_China Cross-lingual Link
e.g., [[Çin Komünist Partisi]]nin
nin
Wikipedia Topic Categories
Ruling Communist parties Chinese Civil War
Parties of one-party systems …
Affix
It is probably too ambitious and challenging to perform this task for all the languages in the world due to the scarcity of resources, but perhaps we could give it a try for hundreds of them. For instance, Wikipedia is a massively multi-lingual resource that currently hosts 295 languages and contains naturally annotated markups 1 and rich informational structures through crowd-sourcing for 35 million articles in 3 billion words. Name mentions in Wikipedia are often labeled as anchor links to their corresponding referent pages. Each entry in Wikipedia is also mapped to external knowl-
1https://en.wikipedia.org/wiki/Help:Wiki markup
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
edge bases such as DBpedia2, YAGO (Mahdisoltani et al., 2015) and Freebase (Bollacker et al., 2008) that contain rich properties. Figure 1 shows an example of Wikipedia markups and KB properties. We will leverage these markups for developing a cross-lingual name tagging and linking system for 282 languages. The major challenges and our proposed new solutions are summarized as follows.
Creating “Silver-standard” through crosslingual entity transfer. The first step is to classify English Wikipedia entries into certain entity types and then propagate these labels to other languages. Most of the previous work (Nothman et al., 2008; Dakka and Cucerzan, 2008; Ringland et al., 2009; Alotaibi and Lee, 2012; Nothman et al., 2012; Althobaiti et al., 2014) manually classified many English Wikipedia entries into pre-defined ACE 3 or CoNLL entity types4. In contrast, we exploit the English Abstract Meaning Representation (AMR) corpus (Banarescu et al., 2013) which includes both name tagging and linking annotations for fine-grained entity types to train an automatic classifier. We also map each AMR type to YAGO type hierarchy. Therefore, the generated typing schema has multiple levels of granularity. Furthermore, we exploit each entry’s properties in DBpedia as features and thus eliminate the need of language-specific features and resources such as part-of-speech tagging as in previous work. Then we project annotations to all entity mentions in Wikipedia articles in other languages through cross-lingual links (Section 2.2).
Refine annotations through self-training. However, such initial annotations are too incomplete and inconsistent. Previous work used name string match to propagate labels. In contrast, we apply self-training to label other mentions without links in Wikipedia articles even if they have different surface forms from the linked mentions (Section 2.3).
Customize annotations through cross-lingual topic transfer. For some populous languages, we obtain millions of labeled sentences from selftraining. For the first time, we propose to customize the annotations for specific down-stream applications. Again, we use a cross-lingual knowledge transfer strategy to leverage the widely avail-
2http://wiki.dbpedia.org 3http://www.itl.nist.gov/iad/mig/tests/ace/ 4http://www.cnts.ua.ac.be/conll2003/ner/
able English corpora to choose entities with specific Wikipedia topic categories (Section 2.4).
Derive morphology analysis from Wikipedia markups. Another unique challenge for many morphologically rich languages is to segment each token into its stemming form and affixes. Previous methods relied on either high-cost supervised learning (Roth et al., 2008; Mahmoudi et al., 2013; Ahlberg et al., 2015), or low-quality unsupervised learning (Gronroos et al., 2014; Ruokolainen et al., 2016). We exploit Wikipedia markups to automatically learn affixes as language-specific features (Section 2.5).
Mine word translations from cross-lingual links. Name translation is a crucial step to generate candidate entities in cross-lingual entity linking. Only a small percentage of names can be directly translated by matching against cross-lingual Wikipedia title pairs. Based on the observation that Wikipedia titles within any language tend to follow a consistent style and format, we propose an effective method to derive word translation pairs from these titles based on automatic alignment (Section 3.2).
2 Name Tagging
2.1 Overview
Our first step is to generate “silver-standard” name annotations from Wikipedia markups and train a universal name tagger. Figure 2 shows our overall procedure and the following subsections will elaborate each component.
2.2 Annotation Generation
We start by assigning an entity type or “other” to each English Wikipedia entry. We utilize Abstract Meaning Representation (AMR) corpus (Banarescu et al., 2013) where each entity name mention is manually labeled as one of 139 types and linked to Wikipedia if it’s linkable. In total we obtain 2,756 entity mentions, along with their AMR entity types, Wikipedia titles and DBPedia properties.
For each pair of AMR entity type ta and YAGO entity type ty, we compute the Pointwise Mutual Information (Church and Hanks, 1990) of mapping ta to ty across all mentions in the AMR corpus. In this way, our framework produces three levels of entity typing schemas with different granularity: 4 regular types (Person (PER), Organization (ORG), Geo-political Entity (GPE), Location
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
[[Мітт Ромні]]Politician|PER народився в [[Детройт]]City|GPE, [[Мічиган]]State|GPE. Закінчив [[Гарвардський університет]]University|ORG.
❖Classify English KB pages using KB properties as features, trained from AMR annotations
en/Mitt_Romney Politician| PER birthPlace, governor, party, successor, ……
en/Detroit City|GPEareaCode, areaTotal, postalCode, elevation, ……
en/Michigan State|GPE demonym, largestCity, language, country, ……
en/Harvard_ University
University| ORG numberOfStudents, motto location, campus, ……
❖Propagate classification results using cross-lingual links and project classification results to anchor links
en/Michigan State|GPE
Ukrainian: uk/Мічиган Amharic: am/ሚሺጋን Tibetan: bo/མི་ཅི་གྷན། Tamil: ta/!c#க% Thai: th/รัฐมิชิแกน
……
Cross-lingual Links
❖Apply self-training for unlabeled data
Training Data
Name Tagger
Unlabeled Data
Train Tag
Add High Confident Instances
❖Select seeds to train an initial name tagger
Training Data Seeds
SelectGenerate (Sec. 2.2)
✤Annotation Generation (Section 2.2)
✤Self Training (Section 2.3)
Train
✤Training Data Selection (Section 2.4)
Wikipedia Articles
Training Data
Entity Commonness Topic Relatedness
Based Ranking Selected Data
(Mitt Romney was born in Detroit, Michigan. He graduated from Harvard University.)
Propagate
Project
Figure 2: Name Tagging Annotation Generation and Training
(LOC)), 139 types in AMR, and 9,154 types in YAGO.
We also propose a new idea to leverage an entity’s properties in DBpedia as features for assigning types. For example, an entity with a birth date is likely to be a person, while an entity with a population property is likely to be a geo-political entity. Using entity properties as features, we train Maximum Entropy models to assign types with three levels of granularity to all English Wikipedia pages. In total we obtained 10 million English pages labeled as entities of interest.
Nothman et al. (2012) manually annotated 4,853 English Wikipedia pages with 6 coarsegrained types (Person, Organization, Location, Other, Non-Entity, Disambiguation Page). Using
this data set for training and testing, we achieved 96.0% F-score on this initial step, slightly better than their results (94.6% F-score).
Next, we propagate the label of each English Wikipedia page to all entity mentions in all languages in the entire Wikipedia through monolingual redirect links and cross-lingual links.
2.3 Self-Training
The name annotations acquired from the above procedure are far from complete to compete with manually labeled gold-standard data. For example, if a name mention appears multiple times in a Wikipedia article, only the first mention is labeled with an anchor link. We apply self-training to propagate and refine the labels.
We use a typical neural network architecture that consists of Bi-directional Long Short-Term Memory and Conditional Random Fields (CRFs) network (Lample et al., 2016) as our underlying learning model for the name tagger. We first train an initial name tagger using seeds selected from the labeled data. We adopt an idea from (Guo et al., 2014) which computes normalized pointwise mutual information (NPMI) (Bouma, 2009) between a tag and a token:
NPMI(tag, token) = ln p(tag,token)p(tag)p(token)
− ln p(tag, token) (1)
Then we select the sentences in which all annotations satisfy NPMI(tag, token) > τ as seeds 5.
For all Wikipedia articles in a language, we cluster the unlabeled sentences into n clusters 6 by collecting sentences with low cross-entropy into the same cluster. Then we apply the initial tagger to the first unlabeled cluster, select the automatically labeled sentences with high confidence, add them back into the training data, and then re-train the tagger. This procedure is repeated n times until we scan through all unlabeled data.
2.4 Training Data Selection
For some populous languages that have many millions of pages in Wikipedia, we obtain many sentences from self-training. In some emergent settings such as natural disasters it’s important to train a system rapidly. Therefore we develop the following effective methods to rank and select high-quality annotated sentences.
5τ = 0 in our experiment. 6n = 20 in our experiment.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Commonness: we prefer sentences that include common entities appearing frequently in Wikipedia. We rank names by their frequency and dynamically set the frequency threshold to select a list of common names. We first initialize the name frequency threshold S to 40. If the number of the sentences is more than a desired size D for training 7, we set the threshold S = S + 5, otherwise S = S − 5. We iteratively run the selection algorithm until the size of the training set reaches D for a certain S.
Topical Relatedness: Various criteria should be adopted for different scenarios. Using an emergent disaster setting as a use case, we prefer sentences that include entities related to disaster related topics. We run an English name tagger (Manning et al., 2014) and entity linker (Pan et al., 2015) on the Leidos corpus released by the DARPA LORELEI program 8. The Leidos corpus consists of documents related to various disaster topics. Based on the linked Wikipedia pages, we rank the frequency of Wikipedia categories and select the top 1% categories (4,035 in total) for our experiments. Some top-ranked topic labels include “International medical and health organizations”, “Human rights organizations”, “International development agencies”, “Western Asian countries”, “Southeast African countries”and “People in public health”. Then we select the annotated sentences including names (e.g., “World Health Organization”) in all languages labeled with these topic labels.
2.5 KB Derived Features
When a Wikipedia user tries to link an entity mention in a sentence to an existing page, she/he will mark the title (the entity’s canonical form, without affixes) within the mention using brackets “[[]]”, from which we can naturally derive a word’s stem and affixes for free. For example, from the Wikipedia markups of the following Turkish sentence: “Kıta Fransası, güneyde [[Akdeniz]]den kuzeyde
[[Manş Denizi]] ve [[Kuzey Denizi]]ne,
doğuda [[Ren Nehri]]nden batıda [[Atlas
Okyanusu]]na kadar yayılan topraklarda yer alır. (Metropolitan France extends from the Mediterranean Sea to the English Channel and the North Sea, and from the Rhine to the Atlantic
7D = 30,000 in our experiment. 8http://www.darpa.mil/program/low-resource-languages-
for-emergent-incidents
Ocean.)”, we can learn the following suffixes: “den”, “ne”, “nden” and “na”. We use such affix lists to perform basic word stemming, and use them as additional features to determine name boundary and type. For example, “den” is a noun suffix which indicates ablative case in Turkish. [[Akdeniz]]den means “from Mediterranean Sea”. Thus, if a token contains a suffix “den”, it is likely to be a location name. Note that this approach can only perform morphology analysis for words whose stem forms and affixes are directly concatenated.
Table 1 summarizes name tagging features.
Features Descriptions Form Lowercase forms of (w−1, w0, w+1) Case Case of w0 Syllable The first and the last character of w0 Stem Stems of (w−1, w0, w+1) Affix Affixes of (w−1, w0, w+1) Gazetteer Cross-lingual gazetteers learned from training data Embeddings Character embeddings and word embed-
dings 9learned from training data
Table 1: Name Tagging Features
3 Cross-lingual Entity Linking
3.1 Overview
After we extract names from documents in a source language, we try to translate them into English by automatically mined word translation pairs (Section 3.2), and then link translated English mentions to an external English KB (in this paper we use DBpedia) (Section 3.3). The overall linking process is illustrated in Figure 3.
3.2 Name Translation
If the source language is not English, an important step in the linking process is to translate identified name mentions into English. We can utilize crosslingual Wikipedia title pairs for exact name string matching. For example, “Pekin” and “Beijing” is a Turkish-English Wikipedia title pair. However, our experiments as well as previous work (Nothman et al., 2012) show that only a small percentage of name mentions can be covered in this way. Fortunately, these cross-lingual title pairs, generated through crowd-sourcing, generally follow a consistent style and format in each language. For example, from Table 2 we can see that the order of
9For languages that don’t have space, we consider each character as a token, and use character embeddings only.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
m1 m2 m3
m4 m5 m6
t5
t1 t4 t3
t6 t2
Translate (e.g., m1 to t1)
Construct Knowledge
Networks (KNs)
KNs in KB Salience, Similarity and Coherence Comparison
Tagged Mentions
Linking
KNs in Source
Translated and Linked Mentions e1t1m1 e1t2m2 e2t3m3
e3t4m4 t5m5 NIL t6m6 NIL
Figure 3: Cross-lingual Entity Linking Overview
Extracted Cross-lingual Wikipedia Title Pairs “Pekin”
Pekin Beijing Pekin metrosu Beijing Subway Pekin Ulusal Stadyumu Beijing National Stadium “Teknoloji” Nükleer teknoloji Nuclear technology Teknoloji transferi Technology transfer Teknoloji eğitimi Technology education “Enstitüsü” Torchwood Enstitüsü Torchwood Institute Hudson Enstitüsü Hudson Institute Smolny Enstitüsü Smolny Institute
“Pekin Teknoloji” [NONE] “Teknoloji Enstitüsü”
Kraliyet Teknoloji Enstitüsü Royal Institute of Technology Karlsruhe Teknoloji Enstitüsü Karlsruhe Institute of Technology Georgia Teknoloji Enstitüsü Georgia Institute of Technology
“Pekin Teknoloji Enstitüsü” [NONE]
Mined Word Translation Pairs Word Translation Alignment
Confidence
pekin Beijing Exact Match beijing 0.5263 peking 0.3158
teknoloji technology 0.8890 technology 0.0233
technological 0.0174
enstitüsü institute 0.2765
of 0.2028 for 0.0221
Table 2: Word Translation Mining from Crosslingual Wikipedia Title Pairs
modifier and head word keeps consistent in Turkish and English titles. Therefore, we propose a new and simple method to mine word and phrase translation pairs as follows.
For each name mention, we generate all possi-
ble combinations of continuous tokens. For example, no Wikipedia titles contain the Turkish name string “Pekin Teknoloji Enstitüsü (Beijing Institute of Technology)”. We generate the following 6 combinations: “Pekin”, “Teknoloji”, “Enstitüsü”, “Pekin Teknoloji”, “Teknoloji Enstitüsü” and “Pekin Teknoloji Enstitüsü”. We then extract all cross-lingual Wikipedia title pairs containing each combination. Finally we run GIZA++ (Och and Ney, 2003) to extract word for word translations from these title pairs. Using this exhaustive search method we can obtain many word translations as shown in Table 2.
3.3 Entity Linking
Given a set of tagged name mentions M = {m1,m2, ...,mn}, we first obtain their English translations T = {t1, t2, ..., tn} using the approach described above. Then we apply an unsupervised collective inference approach to link T to the KB, similar to some previous work (Kulkarni et al., 2009; Fernandez et al., 2010; Radford et al., 2010; Cucerzan, 2011; Han and Sun, 2011; Ratinov et al., 2011; Chen and Ji, 2011; Kozareva et al., 2011; Dalton and Dietz, 2013; Pan et al., 2015; Wang et al., 2015). We construct knowledge networks (KNs) g(ti) for T based on their cooccurrence within a context window 10. For each translated name mention ti, an initial list of candidate entities E(ti) = {e1, e2, ..., ek} is generated based on a surface form dictionary mined from KB properties (e.g., redirects, names, aliases). If no surface form can be matched then we determine the mention as unlinkable. Then we construct knowledge network g(ej) for each entity candidate ej in ti’s entity candidate list E(ti). We compute the similarity between g(ti) and g(ej) based on three measures as described in (Pan et al., 2015): salience, similarity and coherence, and select the candidate entity with the highest score as the appropriate entity for linking.
4 Experiments
4.1 Performance on Wikipedia Data
We first conduct an evaluation using Wikipedia data as “silver-standard”. For each language, we use 70% of the selected sentences for training and 30% for testing. For entity linking, we don’t have ground truth for unlinkable mentions, so we only
10In our experiments, we use the previous four and next four name mentions as a context window.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
587
588
589
590
591
592
593
594
595
596
597
598
599
compute linking accuracy for linkable name mentions. Table 3 presents the overall performance for three coarse-grained entity types: person, organization and geo-political entity/location.
Not surprisingly, name tagging performs better for languages with more mentions. The fscore is generally higher than 80% when there are more than 30K mentions, and it significantly drops when there are less than 250 mentions. The languages with low name tagging performance can be categorized into three types: (1) the number of mentions is less than 2K, such as AtlanticCongo (Wolof), Berber (Kabyle), Chadic (Hausa), Oceanic (Fijian), Hellenic (Greek), Igboid (Igbo), Mande (Bambara), Kartvelian (Georgian, Mingrelian), Timor-Babar (Tetum), Tupian (Guarani) and Iroquoian language groups; Precision is generally higher than recall for most of these languages, because the small number of linked mentions is not enough to cover a wide variety of entities. (2) there is no space between words, including Chinese, Thai and Japanese; (3) they are not written in latin script, such as the Dravidian group (Tamil, Telugu, Kannada, Malayalam).
The training instances for various entity types are quite imbalanced for many languages. For example, Cebuano data includes 1,251 person names, 787,349 geo-political entity/location names and 8,292 organization names. As a result, the performance of organization is the lowest, while geo-political entities and locations achieve higher than 75% F-scores for most languages. The linking accuracy is higher than 80% for most languages.
Also note that since we don’t have perfect annotations on Wikipedia data for any language, these results can be used to estimate how predictable our “silver-standard” data is, but they are not directly comparable to traditional name tagging results measured against gold-standard data annotated by human.
4.2 Performance on Non-Wikipedia Data
Therefore, in order to have more direct comparison with state-of-the-art name taggers trained from human annotated gold-standard data, we conduct experiments on Non-Wikipedia data in 9 languages for which we have human annotated ground truths from the DARPA LORELEI pro-
11The mapping to language names can be found at https://meta.wikimedia.org/wiki/List of Wikipedias
gram. Table 4 shows the data statistics. The documents are from news sources and discussion fora.
We can see that our approach advances stateof-the-art language-independent methods (Zhang et al., 2016a; Tsai et al., 2016) on the same data sets for most languages, and achieves 5.8% - 17.6% lower F-scores than the models trained from manually annotated gold-standard documents that include thousands of name mentions. We also measured the impact of our morphology analysis methods based on the affix lists derived from Wikipedia markups on two morphologicallyrich languages: Turkish and Uzbek. The morphology features contributed 11.1% and 7.1% absolute F-score gains to Turkish and Uzbek respectively.
4.3 Impact of Self-Training
Using Turkish as a case study, the learning curves of self-training on Wikipedia and Non-Wikipedia test sets are shown in Figure 4. We can see that self-training provides significant improvement for both Wikipedia (6% absolute gain) and NonWikipedia test data (12% absolute gain). As expected the learning curve on Wikipedia data is more smooth and converges more slowly than that of Non-Wikipedia data.
5 Related Work
Our work was mainly inspired from previous work that leveraged Wikipedia markups to train name taggers (Nothman et al., 2008; Dakka and Cucerzan, 2008; Mika et al., 2008; Ringland et al., 2009; Alotaibi and Lee, 2012; Nothman et al., 2012; Althobaiti et al., 2014). Our approach doesn’t need any manual annotations or languagespecific features, while generates both coarsegrained and fine-grained types.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
L M F A L M F A L M F A L M F A en 70M 91.8 85.3 lb 102K 81.5 88.4 nah 8.4K 89.9 90.6 m-b 1.6K 78.3 81.3 de 14M 89.0 88.4 tl 102K 92.7 90.3 b-s 8.1K 84.5 88.0 rw 1.6K 95.4 92.7 fr 12M 93.3 92.4 sq 97K 94.1 92.9 rm 8.0K 82.0 91.3 ln 1.6K 82.8 93.3 es 6.9M 93.9 93.4 sco 94K 86.8 89.6 azb 7.9K 88.4 90.6 kl 1.5K 75.0 95.5 it 6.9M 96.6 94.7 ce 90K 99.4 71.0 ay 7.9K 88.5 96.2 sn 1.5K 95.0 99.3 ja 6.5M 79.2 91.8 pnb 81K 90.8 86.2 ps 7.7K 66.9 91.8 av 1.4K 82.0 83.7 ru 6.4M 90.1 89.2 als 81K 85.0 91.2 mi 7.5K 95.9 24.5 wo 1.3K 87.7 97.3 pl 6.0M 90.0 91.3 lmo 80K 98.3 89.0 gag 7.3K 89.3 84.0 xal 1.3K 98.7 90.9 nl 5.6M 93.2 91.9 is 76K 80.2 83.2 am 7.0K 84.7 83.0 na 1.2K 87.6 88.7 sv 4.8M 93.6 89.7 fy 75K 86.6 91.4 nrm 7.0K 96.4 99.6 ltg 1.2K 74.3 92.1 pt 3.9M 90.7 92.4 an 73K 93.0 93.1 nso 6.9K 98.9 99.4 tet 1.2K 73.5 92.3 sh 2.9M 97.8 95.6 gu 70K 76.0 92.4 sc 6.8K 78.1 95.2 ksh 1.2K 56.0 83.6 uk 2.3M 91.5 89.4 qu 63K 92.5 88.2 so 6.5K 85.8 95.7 haw 1.1K 88.0 84.6 zh 2.1M 82.0 89.7 ast 60K 89.2 94.0 co 6.0K 85.4 89.9 pfl 1.1K 42.9 80.4 fi 2.0M 93.4 94.7 z-y 58K 87.3 88.4 stq 6.0K 70.0 92.6 tpi 1.1K 83.3 90.1 cs 1.9M 94.6 92.0 sw 56K 93.4 93.0 pcd 5.8K 86.1 75.8 lo 1.0K 52.8 92.0 no 1.9M 94.1 93.8 ga 55K 85.3 91.3 dsb 5.8K 84.7 82.1 ki 1.0K 97.5 90.0 hu 1.9M 95.9 92.8 jv 54K 82.6 87.8 wa 5.8K 81.6 82.0 ty 1.0K 86.7 89.8 ca 1.8M 90.3 91.6 new 51K 98.2 97.9 wuu 5.8K 79.7 90.8 ady 979 92.7 95.5 fa 1.4M 96.4 83.8 nds 44K 84.5 88.2 frr 5.7K 70.1 86.3 ig 968 74.4 45.5 sr 1.4M 95.3 93.3 ht 42K 98.9 99.8 lad 5.6K 92.3 95.6 tyv 903 91.1 98.8 ko 1.3M 90.6 91.4 pms 40K 98.0 89.5 z-c 5.4K 88.2 87.0 tn 902 76.9 90.1 tr 1.2M 90.5 87.3 bar 39K 97.1 96.3 nap 4.9K 86.9 88.6 cu 898 75.5 76.9 vi 1.2M 89.6 82.0 kn 39K 60.1 92.1 crh 4.9K 90.1 89.9 sm 888 80.0 85.3 ro 1.1M 90.6 85.0 ba 38K 93.8 68.3 km 4.6K 52.2 89.9 to 866 92.3 69.2 he 1.0M 79.0 92.5 arz 35K 77.8 89.3 c-z 4.5K 75.0 89.2 tum 831 93.8 98.5 ar 1.0M 88.3 85.3 ne 31K 81.5 92.5 lez 4.4K 84.2 82.3 r-r 750 93.0 90.5 bg 871K 65.8 88.4 cv 31K 95.7 97.4 mai 4.3K 99.7 90.0 om 709 74.2 81.1 ceb 796K 96.3 86.6 fo 30K 83.6 78.1 hak 4.3K 85.5 88.1 glk 688 59.5 80.8 sk 779K 87.3 90.3 li 28K 89.4 91.6 ang 4.2K 84.0 96.7 lbe 651 88.9 68.6 id 775K 87.8 87.3 ckb 28K 88.1 76.1 r-t 4.2K 88.1 89.0 bjn 640 64.7 89.5 da 717K 87.1 85.8 gd 28K 92.8 91.7 udm 4.2K 88.9 85.0 srn 619 76.5 90.1 z-m 680K 99.3 89.2 io 26K 87.2 95.5 csb 4.1K 87.0 94.4 mdf 617 82.2 97.3 eo 647K 88.7 81.4 mn 25K 76.4 84.4 lij 4.1K 72.3 92.0 tw 572 94.6 90.4 eu 640K 82.5 88.3 tg 25K 88.3 90.6 nov 4.0K 77.0 95.0 pih 555 87.2 90.0 ms 569K 86.8 84.1 bug 25K 99.9 90.0 ace 4.0K 81.6 90.9 rmy 551 68.5 90.0 sl 537K 89.5 90.1 scn 25K 93.2 89.2 gn 4.0K 71.2 89.3 lg 530 98.8 89.9 el 516K 84.6 88.3 ku 23K 83.2 85.1 koi 4.0K 89.6 93.5 chr 530 70.6 90.9 hr 490K 82.8 88.5 pa 21K 74.8 84.3 mhr 3.9K 86.7 94.7 ha 517 75.0 80.0 et 456K 86.8 89.9 yi 20K 76.9 87.2 min 3.8K 85.8 88.5 got 506 91.7 92.3 lt 423K 86.3 87.2 sa 20K 73.9 93.0 ext 3.7K 77.8 91.6 ab 506 60.0 93.3 nn 393K 88.1 86.7 hsb 20K 91.5 74.9 kab 3.3K 75.7 84.3 bi 490 88.5 93.3 th 391K 56.2 87.7 vls 19K 78.2 89.1 szl 3.0K 82.7 92.7 st 455 84.4 89.8 gl 382K 87.4 88.2 ilo 18K 90.3 93.1 tk 2.9K 86.3 90.1 chy 450 85.1 89.9 war 369K 94.9 87.4 vec 15K 87.9 93.8 kv 2.9K 89.7 97.2 iu 450 66.7 83.3 sim 368K 85.7 91.6 bpy 15K 98.3 98.0 f-v 2.9K 65.4 88.8 zu 449 82.3 89.9 lv 349K 92.1 89.8 my 13K 51.5 92.4 gan 2.9K 84.9 90.9 pnt 445 61.5 87.5 ur 309K 96.4 83.4 mzn 13K 86.4 86.9 fur 2.8K 84.5 89.2 ik 436 94.1 54.3 ka 290K 79.8 89.5 os 13K 87.4 89.4 mwl 2.7K 76.1 89.4 lrc 416 65.2 90.0 hy 280K 90.4 88.9 ky 13K 71.8 88.4 nv 2.7K 90.9 94.1 bm 386 77.3 87.5 vo 269K 98.5 97.5 xmf 13K 73.4 94.4 sd 2.7K 65.8 90.9 za 382 57.1 90.3 uz 266K 98.3 94.8 or 13K 86.4 73.5 bxr 2.6K 75.0 91.9 mo 373 69.6 90.2 la 260K 90.8 89.4 pam 13K 87.2 92.9 bo 2.6K 70.4 88.9 ss 362 69.2 71.4 mk 236K 93.4 83.3 si 12K 87.7 94.1 frp 2.5K 86.2 98.5 ee 297 63.2 90.0 bs 227K 84.8 89.9 sah 12K 91.2 78.8 myv 2.5K 88.6 94.4 dz 262 50.0 80.0 kk 224K 88.3 81.8 mt 12K 82.3 90.3 cdo 2.5K 91.0 96.2 ak 258 86.8 92.6 be 205K 84.1 88.3 mrj 12K 97.0 55.3 gom 2.4K 88.8 90.0 sg 245 100.0 89.9 ta 193K 77.9 88.2 n-n 11K 92.6 92.9 bh 2.2K 92.6 92.8 ts 236 93.3 88.9 az 176K 85.1 86.0 as 11K 89.6 83.0 ug 2.1K 79.7 92.4 rn 185 40.0 75.0 hi 170K 86.9 88.0 vep 11K 85.8 89.8 kaa 2.1K 55.2 89.5 ve 183 100.0 83.3 cy 169K 90.7 89.3 diq 10K 79.3 80.9 krc 2.1K 84.9 88.9 ny 169 56.0 75.0 bn 168K 93.8 87.2 zea 10K 86.8 90.3 dv 2.0K 76.2 94.3 ff 168 76.9 88.9 af 142K 85.7 91.1 hif 10K 81.1 96.7 rue 1.9K 82.7 72.1 ch 159 70.6 90.0 br 138K 87.0 85.5 se 9.7K 90.3 83.5 eml 1.8K 83.5 88.5 xh 141 35.3 75.0 mg 134K 98.7 92.9 ie 9.5K 88.8 95.5 arc 1.8K 68.5 82.0 fj 126 75.0 80.0 te 128K 80.5 86.1 su 9.4K 72.7 89.2 pdc 1.8K 78.1 91.1 ks 124 75.0 83.3 b-x 126K 85.1 87.7 bcl 9.3K 82.3 94.0 kbd 1.7K 74.9 80.6 ti 52 94.2 90.0 ml 125K 82.4 88.8 yo 9.2K 94.0 93.0 pap 1.7K 88.8 59.1 cr 49 91.8 89.8 tt 119K 87.7 91.4 ia 8.9K 75.4 90.5 jbo 1.7K 92.4 96.5 pi 41 83.3 90.2 oc 118K 92.5 90.0 kw 8.7K 94.0 96.1 pag 1.7K 91.2 89.5 mr 109K 82.4 91.0 gv 8.4K 84.8 89.1 kg 1.6K 82.1 90.1
Table 3: Performance on Wikipedia Data (L: language ID 11; M: the number of name mentions in selected annotated data; F: name tagging F-score (%); A: linking accuracy (%))
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Language Gold Training Silver Training Test Bengali 8,760 22,093 3,495 Hungarian 3,414 34,022 1,320 Russian 2,751 35,764 1,213 Tamil 7,033 25,521 4,632 Tagalog 4,648 15,839 3,351 Turkish 2,172 37,058 1,353 Uzbek 3,137 64,242 2,056 Vietnamese 2,261 63,971 987 Yoruba 4,061 9,274 3,395
Table 4: # of Names in Non-Wikipedia Data
Language Training from Gold Training from Silver
(Zhang et al., 2016a)
(Tsai et al., 2016)
Bengali 61.6 44.0 34.8 43.3 Hungarian 63.9 47.9 - - Russian 61.8 49.4 - - Tamil 42.2 35.7 33.8 29.6 Tagalog 70.7 58.3 51.3 65.4 Turkish 57.3 51.5 43.6 47.1 Uzbek 56.0 44.2 - - Vietnamese 54.3 44.5 - - Yoruba 55.1 37.6 36.0 36.7
Table 5: Name Tagging F-score (%) on NonWikipedia Test Data
Many fine-grained entity typing approaches (Fleischman and Hovy, 2002; Giuliano, 2009; Ekbal et al., 2010; Xiao and Weld, 2012; Yosef et al., 2012; Nakashole et al., 2013; Gillick et al., 2014; Yogatama et al., 2015; Corro et al., 2015) also created annotations based on Wikipeida anchor links. Our framework performs both name identification and typing and takes advantage of richer structures in the KBs. Previous work on Arabic name tagging (Althobaiti et al., 2014) extracted entity titles as a gazetteer for stemming, and thus it cannot handle unknown names. We developed a new method to derive generalizable affixes for morphologically rich language based on Wikipedia markups.
Wikipedia has also been used as additional features to improve various Information Extraction (IE) tasks, including name tagging (Kazama and Torisawa, 2007), coreference resolution (Ponzetto and Strube, 2006), relation extraction (Chan and Roth, 2010) and event extraction (Hogue and Nothman, 2014). Other automatic name annotation generation methods have been proposed, including KB driven distant supervision (An et al., 2003; Mintz et al., 2009; Ren et al., 2015) and cross-lingual projection (Li et al., 2012; Kim et al., 2012; Che et al., 2013; Wang et al., 2013; Wang
and Manning, 2014; Zhang et al., 2016b). Some recent research (Zhang et al., 2016a; Littell et al., 2016; Tsai et al., 2016) under the DARPA LORELEI program focused on developing name tagging techniques for low-resource languages. These approaches require English annotations for projection (Tsai et al., 2016), some input from a native speaker, either through manual annotations (Littell et al., 2016), or a linguistic survey (Zhang et al., 2016a). Without using any manual annotations, our name taggers outperform previous methods on the same data sets for many languages. NIST TAC-KBP Tri-lingual entity linking (Ji et al., 2016) focused on three languages: English, Chinese and Spanish. (McNamee et al., 2011) extended it to 21 languages. But their methods required labeled data and name transliteration. We share the same goal as (Sil and Florian, 2016) to extend cross-lingual entity linking to all languages in Wikipedia. They exploited Wikipedia links to train a supervised linker. We mine reliable word translations from cross-lingual Wikipedia titles, which enables us to adopt unsupervised English entity linking techniques.
6 Conclusions and Future Work
We developed a simple yet effective framework that can extract names from 282 languages and link them to an English KB. This framework follows a fully automatic training and testing pipeline, without the needs of any manual annotations or knowledge from native speakers. We evaluated our framework on both Wikipedia articles and external formal and informal texts and obtained promising results. To the best of our knowledge, our multilingual name tagging and linking framework is applied to the largest number of languages. We release the following resources for each of these 282 languages: “silver-standard” name tagging and linking annotations with multiple levels of granularity, morphology analyzer if it’s a morphologically-rich language, and an endto-end name tagging and linking system. In this work, we treat all languages independently when training their corresponding name taggers. In the future, we will explore the topological structure of related languages and exploit cross-lingual knowledge transfer to enhance the quality of extraction and linking. The general idea of deriving noisy annotations from KB properties can also be extended to other IE tasks such as relation extraction.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
- Strengths:    - The paper states clearly the contributions from the beginning     - Authors provide system and dataset    - Figures help in illustrating the approach    - Detailed description of the approach    - The authors test their approach performance on other datasets and compare to other published work  - Weaknesses:    -The explanation of methods in some paragraphs is too detailed and there is no mention of other work and it is repeated in the corresponding method sections, the authors committed to address this issue in the final version.    -README file for the dataset [Authors committed to add README file]  - General Discussion:    - Section 2.2 mentions examples of DBpedia properties that were used as features. Do the authors mean that all the properties have been used or there is a subset? If the latter please list them. In the authors' response, the authors explain in more details this point and I strongly believe that it is crucial to list all the features in details in the final version for clarity and replicability of the paper.     - In section 2.3 the authors use Lample et al. Bi-LSTM-CRF model, it might be beneficial to add that the input is word embeddings (similarly to Lample et al.)    - Figure 3, KNs in source language or in English? (since the mentions have been translated to English). In the authors' response, the authors stated that they will correct the figure.    - Based on section 2.4 it seems that topical relatedness implies that some features are domain dependent. It would be helpful to see how much domain dependent features affect the performance. In the final version, the authors will add the performance results for the above mentioned features, as mentioned in their response.     - In related work, the authors make a strong connection to Sil and Florian work where they emphasize the supervised vs. unsupervised difference. The proposed approach is still supervised in the sense of training, however the generation of training data doesn t involve human interference