Interpreting Neural Networks to Understand Written Justifications in Values-Affirmation Essays
We report efforts to interpret a Long-Short Term Memory neural network trained to recognize gender and writing instructions on a set of essays from a psychological educational intervention known as a values affirmation. Adjusting the model at test time to output sequential probabilities as each new token is encountered, rather than predicting the class holistically, we query the model with carefully constructed sentences designed to test a theoretically informed hypothesis: male versus female students write in a way that reflects a greater emphasis on independence versus interdependence, respectively. The LSTM model outperforms the baseline, and the model’s predictions to our constructed test sentences suggest modest support for these hypotheses.
1 Introduction
The difficulty of interpreting neural networks is a major barrier to their wider adoption among applied quantitative scientists. Although these models have achieved a high level of performance across a number of NLP and vision processing tasks (Sutskever et al., 2014; Bahdanau et al., 2014; Socher et al., 2011; Chen and Manning, 2014), attempts to understand why a model performs as it does have been less frequent. Some recent work in this area has been devoted to understanding long-term dependencies such as bracketing (Hermans and Schrauwen, 2013; Karpathy et al., 2015), studying learned sparse word vectors (Faruqui et al., 2015) and to generating rationalization of predictions without manual annotations (Lei et al., 2016). These efforts are important for understanding where and why a model ex-
periences shortcomings. Additionally, as we address here, finding straightforward ways to interpret these models will increase their applicability across the sciences more generally.
In this paper, we take a somewhat different approach to understanding properties learned by a Long-Short Term Memory (LSTM) network (Hochreiter and Schmidhuber, 1997). Rather than exploring the components of the model itself, we take a hypothesis-driven approach in which we allow the model to respond (i.e., make predictions) to stimuli we devise that is of theoretical interest to the investigators. This approach is imported directly from experimental psychology in which the objective is to mechanistically understand cognition. This is a flexible approach that allows the investigator to directly devise stimuli in an intuitive manner which address questions about what the model has learned from the information it was trained on.
To demonstrate the utility of this approach, we explore a data set of essays taken from a series of field studies of a written educational intervention conducted in a North American middle school (Cohen et al., 2009, 2006; Cook et al., 2012). This intervention is designed to mitigate the detrimental effects of stress on student performance. Researchers have demonstrated its effectiveness at boosting the performance of students who are under excessive amounts of stress due to having an identity that is stigmatized in academic settings (e.g. African American, or women in STEM classes). The typical format of the intervention has students select from a list of values (e.g. religion, sports, relationships) and write about why the value they selected is important to them. This values affirmation intervention has reduced the achievement gap between African American and White students in middle school (Cohen et al., 2009), Latino students and White students in mid-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
dle school (Sherman et al., 2013), and between male and female students in a college physics class (Miyake et al., 2010), among other examples (reviewed in Cohen and Sherman, 2014).
However, there has been little work exploring the text that students produce while they are writing these essays. We study the justifications that students provide when writing about their values. That is, because the writing prompts instruct students to select a value, and then justify that selection by writing about why it is important, we sought to understand whether these justifications differ between categories of students. In particular, we examine whether female students are more likely than male students to justify their value selection by making reference to others. We also examine whether male students are more likely than female students to justify their value selection by making reference to the self - in particular, types of self-action (Section 3.3). Both of these hypotheses are motivated by previous psychological research. Specifically, women are thought to generally tend toward an interdependent selfconstrual (Gabriel and Gardner, 1999), which implies a greater emphasis on social relationships (Kuebli and Fivush, 1992; Broderick and Beltz, 1996). Similarly, males are believed to tend toward an independent self-construal, which implies greater emphasis on the self and would suggest that they would be more likely to justify their value selection by referring directly to the self or with a self-action (Cross et al., 2000).
To be able to carry out these hypothesis-driven studies, we proposed an approach that aims to understanding LSTM predictions at a higher level of abstraction (i.e., from the output distribution directly rather than by analyzing hidden states and/or gate activations) (Section 2). In short, we specify a set of inputs that vary along some dimension of theoretical interest, and observe the outputs (probability distributions of classes) that correspond to these inputs. To achieve this, we train the proposed neural network architecture in a supervised way with a dense softmax layer as the last layer at the end of the last time step. During test time, we use this last layer at each time step and record the outputs to understand the sequential class probabilities. We show that our methods lead to more interpretable results in the task of understanding the gender-specific characteristics of text from values affirmation essays.
1.1 Related work
Linguistic Analysis of Values Affirmation To our knowledge, there are just three other papers directly addressing linguistic data in valuesaffirmation interventions. Shnabel et al., (2013) hand coded a small sample of values-affirmation essays, looking for whether they contained themes of social belonging. Their results found that not only were essays written by writers in the treatment condition more likely to contain such themes, but those writers in the treatment condition who wrote about these topics were also more likely to perform well academically than writers in the treatment condition who did not write about social belonging. This effect was unexpectedly reversed for students who were not under threat - White students in these data. Though this reversal observed for white students could reflect some deeper underlying theoretical insight, it is worth noting that this result is based on a relatively small sample of white students (n = 186, approximately half of whom were in the focal condition), and as such could be explained by a lack of power to estimate this type of effect. Taking a similar approach, with a similarly sized dataset, Tibbetts et al. (2016) found that first-generation college students (another group who typically experiences threat in an academic environment) who wrote about independence in the treatment condition performed better academically than those who did not write about this theme. Finally, Riddle et al. (2015) take a more descriptive approach to these data, using topic models to look for topics that distinguished writers of different ethnicities and genders.
Interpreting Neural Networks Due to their high-level performance on a number of tasks, there has been some recent work attempting to understand neural networks beyond simple performance metrics. Hermans and Schrauwen (2013) worked on understanding long term dependencies such as detecting parenthesis using recurrent networks by analyzing state activations. Relatedly, Karpathy et al. (2015) used the linux kernel source code dataset to study similar long term dependencies such as in bracketing and line-lengths, incorporating analysis of gate activations during prediction to understand their models. Faruqui et al. (2015) learn sparse word vectors, and show that these vectors can be linked to semantic lexicons and word properties. Additionally, Lei et al. (2016)
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
uses an encoder-generator framework to automatically generate rationalizations of predictions without the need for manual annotations.
To our knowledge there has been no work that attempts to understand LSTM predictions at a higher level of abstraction (i.e., from the output distributions directly rather than analysis of hidden states and/or gate activations).
This approach borrows directly from standard experimental psychology methods. These methods are designed to test hypotheses about why a decision maker behaves in as observed. Applying these same approaches to understanding LSTM can enable quantitative scientists to explore specific hypotheses in their data, as we demonstrate below.
2 Methods
Our aim in this work is to develop a system that allows the applied quantitative scientist to query the network with a specific set of test input (stimuli) of his or her choosing that can highlight a question of theoretical interest. In our case, we are interested in understanding textual features of writing that make a values affirmation essay more likely to have been written by a male versus female student in the treatment or control conditions.
2.1 Data
Our data come from a series of field studies conducted at a middle school in the northeastern United States with 8 cohorts of students. These studies, described in part elsewhere (Cohen et al., 2009, 2006) were designed to test the effectiveness of a written psychological intervention, called a values-affirmation on reducing the academic achievement gap. At the beginning of their time in middle school, and at several other times throughout their middle school years, students completed a short essay in response to either a treatment or control writing prompt. In the treatment condition, students were typically asked to look at list of values (athletic ability, art, being smart and getting good grades, creativity, independence, social groups, music, politics, relationships, religion, sense of humor, or living in the moment), and pick one to three that were most important to them and write about why that/those values were important. The control condition was similar, but students were instead asked to pick one to three values that might be important to someone else
and write about why they might be important to someone else. 1 An example of of a treatment and control essay are included below:
Treatment: They are important because I like listening to music and playing my flute. I like having friends and I also enjoy being funny sometimes.
Control: Art might be important to someone else because they might like art, want to go to school to study art to one day become an artist.
In total, the dataset consists of 6224 essays written by 1233 students. Female students wrote 3315 of these essays (of which 1851 were control essays), while male students wrote 2909 (of which 1622 were control essays). For an unrelated project, research assistants manually fixed all gross spelling and grammatical mistakes. The resulting cleaned essays have an average length of 42.8 words (SD = 23.9), and are composed of 5816 distinct tokens.
2.2 Neural Network Architecture
Our classification task is to categorize an essay into one of four classes consisting of the four combinations of gender (male versus female) and experimental condition (treatment versus control).
Our NN architectures are depicted in Figures 1 and 2. Figure 1 is the training architecture while figure 2 is the testing architecture. We initialize our network by using the 840B 300d Stanford GloVe word embedding vectors (Pennington et al., 2014) as a bias for words that exist in our data.
Input Layer: . All essays are first tokenized, and each word in the essay is mapped to an index in a dictionary. The essays are fed as input as a list of constantly-sized integers to the neural network - shorter essays were padded to create inputs of the same length.
1The nature of the treatment and control prompts changed somewhat across interventions in order to keep students engaged and to examine the effects of small tweaks of the intervention (e.g. writing about one versus three values).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 2: Neural network architecture (testing)
Figure 1: Neural network architecture (training)
Embedding Layer: . The input vector is fed to the embedding layer which converts each word into a distributional vector of 300 dimensions. If these vectors are found in the GloVe dataset , they are derived from the corresponding GloVe vector. If not, they are initialized to 0. This layer is trainable. The input token index is mapped to one of the 300 dimensional vectors and the set of indices mapped to the 300 dimensional vectors is sent as the output of this layer.
The output from the embedding layer is of dimension (Ti, 300), where T is the number of tokens in essay i. This output is fed to a dropout layer for the purpose of regularization, which drops units with probability .5 during training to avoid overfitting. During test time, no units are dropped.
LSTM layer: The output from the dropout layer is fed to an LSTM layer to capture long term and short term dependencies. The state at the last time step of the LSTM layer is sent to a dropout layer
to regularize and then a fully connected layer of dimension 4 (one for each of the output classes) with a softmax activation function to classify to one of the four outputs in terms of a probability distributions.
The loss function used is categorical cross entropy:
L(X,Y ) = − 1 n n∑ i=1 y(i)lna(x(i))
+ (1− y(i))ln(1− a((x(i)))) (1)
where X = {x(1), ..., x(n)} is the set of input examples in the training dataset and Y = {y(1), ..., y(n)} is the corresponding set of labels for those input examples and a(x) represents the output of the neural network, given input x.
For optimization, we employ rmsprop - Dividing the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight (Tieleman and Hinton, 2012). The metrics used while training are precision, recall, macro f1, and categorical accuracy. The minibatch size used is 256, and the dimension of the hidden state in the LSTM is 50.
At test time, the architecture is changed such that the output at each timestep is fed to the softmax layer to predict the probability of the observed class at that time instant (Figure 2). This modification helps us to understand how specified sequences of text contribute to the classification decision.
The model was constructed and fit using the Keras interface (Chollet, 2015) in Python 2.7 with a Theano backend (Theano Development Team, 2016).
3 Results
3.1 Model Performance
For all model comparisons, we used a 85/15 train/test split to evaluate performance. Our baseline model - a linear support vector classifier was trained to identify the four gender and condition categories (male & control, male & treatment, female & control, female & treatment) using features imported from previous work (Riddle et al., 2015). Specifically, this model included a tf-idf language model with standard preprocessing steps (lemmatized, stop words removed, words with document frequency ¡ 4 removed), and a 50-topic
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Figure 3: Correlations between SVM coefficients and LSTM probability changes
lda topic model. The linear support vector classifier was trained using 10-fold cross-validation, with each fold using 15% of the training data as a validation set. This model achieved a macro f1 score of .53 on the final test set. In contrast, the LSTM model, which was validated during training on a held-out 15% performed better, achieving a macro f1 score of .60 on the final test set.
3.2 Validating the Interpretations
As described above, our model emits class probabilities at each word. In order to establish that these probabilities are valid ways to interpret the model, we compare them with coefficients from our baseline classifier. Because the probability change for the LSTM is context dependent, each token is possibly associated with multiple changes in probability. Thus, for each token, we obtained the average probability change for each class. As seen in figure 3, there are reasonably strong correlations (pearson’s correlation coefficients between .32 and .43) between a token’s coefficient from the SVM model and that token’s average probability change from the LSTM. This provides some evidence that the probability shifts for a token measure the same underlying construct as SVM coefficients.
Example Interpretation: In Figure 4, we show an illustrative example of model interpretation. In this example, we use an example essay written by a female student in the treatment condition:
They are important because I like listening to music and playing my flute. I
like having friends and I also enjoy being funny sometimes.
There are a number of noteworthy points in the example essay where the probabilities shift in a way that would be expected based on the nature of the writing prompt and the interests of male and female American middle schoolers. For instance, after the first five words, the model is beginning to predict that the essay is from the treatment condition. The first five words, They are important because I are more typical for writing in response to the treatment prompt (write about why the values are important to you) as opposed to the control prompt (write about why the values are important to someone else). Another noteworthy shift here is when the model encounters the word flute, an instrument that is relatively gendered in American culture (i.e. it is more strongly associated with women than with men). At this point, the model is positive that this is an essay from the treatment condition, written by a female student.
3.3 Making Psychological Queries
Having validated the approach to interpreting the model and shown an example case, we next turn to asking richer questions about our data. In particular, we present evidence that suggests that male writers and female writers use slightly different justifications for selecting values. Given that the writing prompt for the treatment condition asks students to ”write about why [their chosen value] is important to you”, these justifications can tell us something about the underlying needs and motivations for some group of writers - in this case, male versus female students.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
To explore this question, we refit the LSTM model on the entire dataset. After adjusting to the model to obtain class probabilities through the essay as described above, we adapted methods from experimental psychology to query what the model has learned. In psychological experiments, human participants are often given linguistic items that differ along some key dimension of interest while the experimenter records the responses to these items. We took this approach to investigating the model, with the responses consisting of the sequential probabilities.
Figure 5: Sequential probabilities for the last five words in the other-justified test sentences. The final word (justification) is marked as j. Each line is an individual sentence fragment’s sequential probabilities.
The dimension of interest that was systematically varied was the type of justification used in a stereotypical treatment essay. Specifically, we fed the model a series of sentence segments taking the form of ”V is important to me because J”, where the V is one of the values that students could have selected (e.g. athletic ability, getting good grades, etc), and J was a series of bigrams that was one of two types - either selffocused justifications, or other-focused justifications. As described in the introduction, our expectation is that female students will be more likely to use other-focused justifications in which they describe their value in relation to other people because this reflects a greater degree of interdependent self-construal (e.g. a female student would be more likely to write ”Art is important to me because my mom...”). For male students, we expect them to be more likely to use self-focused justi-
fications, reflecting a greater degree of independent self-construal. Such a justification would take them form of their explaining a value by making some reference directly to the self doing, thinking, or performing something (e.g. a male student might be more likely to write ”Art is important to me because I would...”).
Table 1 displays the values and justifications that were used for the experiments. Using these combinations, we generated 195 other-focused sentence segments and 195 self-focused sentence segments for testing. This methodological approach can tell us whether one of these classes is more or less likely to use other-focused versus self-focused justifications.
Figure 6: Sequential probabilities for the last five words in the self-justified test sentences. The final word (justification) is marked as j. Each line is an individual sentence fragment’s sequential probabilities.
Figures 5 and 6 displays the sequence of probabilities for the last five words in the generated sentence fragments. As expected, the probabilities for a given value are identical for a given class until the justification word is encountered, at which points the probabilities diverge. Because these probabilities are compositional in nature (i.e. there are D alternative probabilities in the vector of any time point and these alternatives must sum to 1), we first transformed the probabilities for each sentence fragment at each time step according to the centered log-ratio transformation (Aitchison, 1986). This transformation is of the
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Value other-focused self-focused Athletic ability my friends I want Art my family I need Being smart my grandparents I will Getting good grades my parents I have Creativity my teacher I can Independence my friend I feel Social groups my dad I should Music my mom I would Politics my sister I hope Relationships my brother I am Religion my mother I get Sense of Humor my father I might Living in the moment my cousin I use - my aunt I like - my uncle I take
Table 1: Combinations of values and justifications (other-focused versus self-focused) used in the test sentence segments
form:
log( x1 g(x) , x2 g(x) , ... xD g(x) ) (2)
Where xD are the class probabilities, and g(x) is the geometric mean of the probabilities.
To quantify the effect of these two different justifications, we fit two multilevel bayesian model to the centered log-ratio transformed LSTM output (Baayen et al., 2008) - one model for otherfocused sentences and one model for self-focused sentences. In mixed-effects terminology, these model had fixed effects terms for class, time, and their interaction, and a three fully crossed random effects - one for sentence fragment, one for the value being justified, and one for the different justifications. Each of these crossed random effects has its own vector of intercept and time coefficients. These types of models are a preferred approach for the analysis of this type of psycholinguistic data, or any other type of data in which the experimenter is recording responses from a participant (or in our case, an LSTM network) to stimuli that are sampled from some larger class (Baayen et al., 2008; Westfall et al., 2016; Judd et al., 2017). All priors were set to a standard normal distribution. The bayesian model was fit using Stan (Carpenter et al., 2016) and the Rstanarm interface (Gabry and Goodrich, 2016).
Taking draws from the posterior distribution, we obtained the difference in probability distribution
for word J and word J − 1 for each class. Because our hypotheses were primarily about how male and female students would justify their selection in values, we then subtracted these two difference distribution, yielding a posterior distribution that quantifies the degree to which the lstm model believes that female students are more likely to use a given type of justification. More formally:
δm = Xmj −Xmp (3a) δf = Xfj −Xfp (3b) Fpref = δf − δm (3c)
where X designates the posterior distribution estimated directly in the bayesian model, the subscripts m and f reference male and female students, respectively, and the subscripts j and p reference the justification word and the word preceding the justification word, respectively.Xmj is the posterior distribution for the justification word J for males, X The intuition is that we are interested in if the justification word J differentially increased probability for female students over male students or vice versa.
For other-focused justification, the analysis indicates a weak tendency for these types of justifications to be used more often by women (mean of Fpref = .06, 95% highest density interval (HDI) = [−.11, .22]), with the posterior distribution indicating that 74% of the posterior dis-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
tribution is consistent with an effect of this direction. In real terms, the overall probability of the treatment-female class increased from a mean of .24(SD = .09) at the penultimate word to a mean of .26(SD = .13) after seeing the justification word, while the overall probability of the male-treatment class increased a very small amount from .60(SD = .14) to .61(SD = .17) for the justification word. The analysis of selffocused justifications is in the opposite direction, with the effect suggesting stronger support for the hypothesis. Namely, the LSTM model seems to have learned that male students may be more likely to justify their values with language that reflects a self-focus (mean of Fpref = −.23, 95% HDI = [−.39, .07], with the posterior distribution indicating that 99.8% of the posterior distribution is consistent with an effect of this direction. Again, in real terms, the overall probability of the female-treatment class decreased from a mean of .21, (SD = .08) at the penultimate word to a mean of .19, (SD = .08) at the justification word, while the overall probability of the male-treatment class increased from a probability of .63, (SD = .15) to a mean of .69, (SD = .14).
4 Conclusion
We have described a novel way to explore neural networks. By treating the model as a input-output device and carefully crafting stimuli that systematically varies along a dimension of interest, we can map systematic differences in the output to the stimulus input. Doing so allows us to understand whether the model has learned about this dimension, and consequently, whether there is information of this type in the training data, even if the same language does not appear verbatim.
Our work also extends literature in psychology, as this is the first instance in which the justifications of values-affirmation essays were explored. We find evidence largely consistent with expected gender differences as a function of theories of gendered self-construal (Gabriel and Gardner, 1999; Kuebli and Fivush, 1992; Broderick and Beltz, 1996). Future work should disentangle the degree to which these justifications influence the effectiveness of the intervention on academic outcomes.

- Strengths: this paper addresses (in part) the problem of interpreting Long Short-Term Memory (LSTM) neural network models trained to categorize written justifications in values-affirmation essays. This is definitely an interesting research question. To do so, the authors want to rely on approaches that have are standard in experimental psychology. Furthermore, the authors also aim at validating sociological assumptions via this study.  - Weaknesses: one of the main weaknesses of the paper lies in the fact that the goals are not clear enough. One overall, ambitious goal put forward by the authors is to use approaches from experimental psychology to interpret LSTMs. However, no clear methodology to do so is presented in the paper. On the other hand, if the goal is to validate sociological assumptions, then one should do so by studying the relationships between gender markers and the written justifications, independently on any model. The claim that  expected gender differences (are) a function of theories of gendered self-construal  is not proven in the study.  - General Discussion: if the study is interesting, it suffers from several weak arguments. First of all, the fact that the probability shift of a token in the LSTM network are correlated with the corresponding SVM coefficients is no proof that  these probabilities are valid ways to interpret the model . Indeed, (a) SVM coefficients only reveal part of what is happening in the decision function of an SVM classifie and (b) it is not because one coefficient provides an interpretation in one model that a correlated coefficient provides an explanation in another model. Furthermore, the correlation coefficients are not that high, so that the point put forward is not really backed up.  As mentioned before, another problem lies in the fact that the authors seem to hesitate between two goals. It would be better to clearly state one goal and develop it. Concerning the relation to experimental psychology, which is a priori an important part of the paper, it would be interesting to develop and better explain the multilevel bayesian models used to quantify the gender-based self-construal assumptions. It is very difficult to assess whether the methodology used here is really appropriate without more details. As this is an important aspect of the method, it should be further detailed.


The ambitious goal of this work is to develop a cross-lingual name tagging and linking framework for 282 languages that exist in Wikipedia. Given a document in any of these languages, our framework is able to identify name mentions, assign a coarse-grained or fine-grained type to each mention, and link it to an English Knowledge Base (KB) if it is linkable. We achieve this goal by performing a series of new KB mining methods: generating “silver-standard” annotations by transferring annotations from English to other languages through crosslingual links and KB properties, refining annotations through self-training and topic selection, deriving language-specific morphology features from anchor links, and mining word translation pairs from crosslingual links. Both name tagging and linking results for 282 languages are promising on Wikipedia data. The results on non-Wikipedia data (news articles and discussion forum posts) are also comparable to the supervised models trained from manually annotated documents including thousands of names. All the data sets, resources and systems for 282 languages will be made publicly available as a new benchmark.
1 Introduction
Information provided in languages which people can understand saves lives in crises. For example, language barrier was one of the main difficulties faced by humanitarian workers responding to the Ebola crisis in 2014. We propose to break language barriers by extracting information (e.g.,
entities) from a massive variety of languages and ground the information into an existing knowledge base which is accessible to a user in his/her own language (e.g., a reporter from the World Health Organization who speaks English only).
✤Wikipedia Article: Mao Zedong (d. 26 Aralık 1893 - ö. 9 Eylül 1976), Çinli devrimci ve siyasetçi. Çin Komünist Partisinin (ÇKP) ve Çin Halk Cumhuriyetinin kurucusu.
✤Wikipedia Markup: [[Mao Zedong]] (d. [[26 Aralık]] [[1893]] - ö. [[9 Eylül]] [[1976]]), Çinli devrimci ve siyasetçi. [[Çin Komünist Partisi]]nin (ÇKP) ve [[Çin Halk Cumhuriyeti]]nin kurucusu.
KB Properties (e.g., DBpedia, YAGO)
formationDate headquarter
ideology …
(Mao Zedong (December 26, 1893 - September 9, 1976) is a Chinese revolutionary and politician. The founder of the Chinese Communist Party (CCP) and the People's Republic of China.)
tr/Çin_Komünist_Partisi Anchor Link en/Communist_Party_of_China Cross-lingual Link
e.g., [[Çin Komünist Partisi]]nin
nin
Wikipedia Topic Categories
Ruling Communist parties Chinese Civil War
Parties of one-party systems …
Affix
It is probably too ambitious and challenging to perform this task for all the languages in the world due to the scarcity of resources, but perhaps we could give it a try for hundreds of them. For instance, Wikipedia is a massively multi-lingual resource that currently hosts 295 languages and contains naturally annotated markups 1 and rich informational structures through crowd-sourcing for 35 million articles in 3 billion words. Name mentions in Wikipedia are often labeled as anchor links to their corresponding referent pages. Each entry in Wikipedia is also mapped to external knowl-
1https://en.wikipedia.org/wiki/Help:Wiki markup
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
edge bases such as DBpedia2, YAGO (Mahdisoltani et al., 2015) and Freebase (Bollacker et al., 2008) that contain rich properties. Figure 1 shows an example of Wikipedia markups and KB properties. We will leverage these markups for developing a cross-lingual name tagging and linking system for 282 languages. The major challenges and our proposed new solutions are summarized as follows.
Creating “Silver-standard” through crosslingual entity transfer. The first step is to classify English Wikipedia entries into certain entity types and then propagate these labels to other languages. Most of the previous work (Nothman et al., 2008; Dakka and Cucerzan, 2008; Ringland et al., 2009; Alotaibi and Lee, 2012; Nothman et al., 2012; Althobaiti et al., 2014) manually classified many English Wikipedia entries into pre-defined ACE 3 or CoNLL entity types4. In contrast, we exploit the English Abstract Meaning Representation (AMR) corpus (Banarescu et al., 2013) which includes both name tagging and linking annotations for fine-grained entity types to train an automatic classifier. We also map each AMR type to YAGO type hierarchy. Therefore, the generated typing schema has multiple levels of granularity. Furthermore, we exploit each entry’s properties in DBpedia as features and thus eliminate the need of language-specific features and resources such as part-of-speech tagging as in previous work. Then we project annotations to all entity mentions in Wikipedia articles in other languages through cross-lingual links (Section 2.2).
Refine annotations through self-training. However, such initial annotations are too incomplete and inconsistent. Previous work used name string match to propagate labels. In contrast, we apply self-training to label other mentions without links in Wikipedia articles even if they have different surface forms from the linked mentions (Section 2.3).
Customize annotations through cross-lingual topic transfer. For some populous languages, we obtain millions of labeled sentences from selftraining. For the first time, we propose to customize the annotations for specific down-stream applications. Again, we use a cross-lingual knowledge transfer strategy to leverage the widely avail-
2http://wiki.dbpedia.org 3http://www.itl.nist.gov/iad/mig/tests/ace/ 4http://www.cnts.ua.ac.be/conll2003/ner/
able English corpora to choose entities with specific Wikipedia topic categories (Section 2.4).
Derive morphology analysis from Wikipedia markups. Another unique challenge for many morphologically rich languages is to segment each token into its stemming form and affixes. Previous methods relied on either high-cost supervised learning (Roth et al., 2008; Mahmoudi et al., 2013; Ahlberg et al., 2015), or low-quality unsupervised learning (Gronroos et al., 2014; Ruokolainen et al., 2016). We exploit Wikipedia markups to automatically learn affixes as language-specific features (Section 2.5).
Mine word translations from cross-lingual links. Name translation is a crucial step to generate candidate entities in cross-lingual entity linking. Only a small percentage of names can be directly translated by matching against cross-lingual Wikipedia title pairs. Based on the observation that Wikipedia titles within any language tend to follow a consistent style and format, we propose an effective method to derive word translation pairs from these titles based on automatic alignment (Section 3.2).
2 Name Tagging
2.1 Overview
Our first step is to generate “silver-standard” name annotations from Wikipedia markups and train a universal name tagger. Figure 2 shows our overall procedure and the following subsections will elaborate each component.
2.2 Annotation Generation
We start by assigning an entity type or “other” to each English Wikipedia entry. We utilize Abstract Meaning Representation (AMR) corpus (Banarescu et al., 2013) where each entity name mention is manually labeled as one of 139 types and linked to Wikipedia if it’s linkable. In total we obtain 2,756 entity mentions, along with their AMR entity types, Wikipedia titles and DBPedia properties.
For each pair of AMR entity type ta and YAGO entity type ty, we compute the Pointwise Mutual Information (Church and Hanks, 1990) of mapping ta to ty across all mentions in the AMR corpus. In this way, our framework produces three levels of entity typing schemas with different granularity: 4 regular types (Person (PER), Organization (ORG), Geo-political Entity (GPE), Location
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
[[Мітт Ромні]]Politician|PER народився в [[Детройт]]City|GPE, [[Мічиган]]State|GPE. Закінчив [[Гарвардський університет]]University|ORG.
❖Classify English KB pages using KB properties as features, trained from AMR annotations
en/Mitt_Romney Politician| PER birthPlace, governor, party, successor, ……
en/Detroit City|GPEareaCode, areaTotal, postalCode, elevation, ……
en/Michigan State|GPE demonym, largestCity, language, country, ……
en/Harvard_ University
University| ORG numberOfStudents, motto location, campus, ……
❖Propagate classification results using cross-lingual links and project classification results to anchor links
en/Michigan State|GPE
Ukrainian: uk/Мічиган Amharic: am/ሚሺጋን Tibetan: bo/མི་ཅི་གྷན། Tamil: ta/!c#க% Thai: th/รัฐมิชิแกน
……
Cross-lingual Links
❖Apply self-training for unlabeled data
Training Data
Name Tagger
Unlabeled Data
Train Tag
Add High Confident Instances
❖Select seeds to train an initial name tagger
Training Data Seeds
SelectGenerate (Sec. 2.2)
✤Annotation Generation (Section 2.2)
✤Self Training (Section 2.3)
Train
✤Training Data Selection (Section 2.4)
Wikipedia Articles
Training Data
Entity Commonness Topic Relatedness
Based Ranking Selected Data
(Mitt Romney was born in Detroit, Michigan. He graduated from Harvard University.)
Propagate
Project
Figure 2: Name Tagging Annotation Generation and Training
(LOC)), 139 types in AMR, and 9,154 types in YAGO.
We also propose a new idea to leverage an entity’s properties in DBpedia as features for assigning types. For example, an entity with a birth date is likely to be a person, while an entity with a population property is likely to be a geo-political entity. Using entity properties as features, we train Maximum Entropy models to assign types with three levels of granularity to all English Wikipedia pages. In total we obtained 10 million English pages labeled as entities of interest.
Nothman et al. (2012) manually annotated 4,853 English Wikipedia pages with 6 coarsegrained types (Person, Organization, Location, Other, Non-Entity, Disambiguation Page). Using
this data set for training and testing, we achieved 96.0% F-score on this initial step, slightly better than their results (94.6% F-score).
Next, we propagate the label of each English Wikipedia page to all entity mentions in all languages in the entire Wikipedia through monolingual redirect links and cross-lingual links.
2.3 Self-Training
The name annotations acquired from the above procedure are far from complete to compete with manually labeled gold-standard data. For example, if a name mention appears multiple times in a Wikipedia article, only the first mention is labeled with an anchor link. We apply self-training to propagate and refine the labels.
We use a typical neural network architecture that consists of Bi-directional Long Short-Term Memory and Conditional Random Fields (CRFs) network (Lample et al., 2016) as our underlying learning model for the name tagger. We first train an initial name tagger using seeds selected from the labeled data. We adopt an idea from (Guo et al., 2014) which computes normalized pointwise mutual information (NPMI) (Bouma, 2009) between a tag and a token:
NPMI(tag, token) = ln p(tag,token)p(tag)p(token)
− ln p(tag, token) (1)
Then we select the sentences in which all annotations satisfy NPMI(tag, token) > τ as seeds 5.
For all Wikipedia articles in a language, we cluster the unlabeled sentences into n clusters 6 by collecting sentences with low cross-entropy into the same cluster. Then we apply the initial tagger to the first unlabeled cluster, select the automatically labeled sentences with high confidence, add them back into the training data, and then re-train the tagger. This procedure is repeated n times until we scan through all unlabeled data.
2.4 Training Data Selection
For some populous languages that have many millions of pages in Wikipedia, we obtain many sentences from self-training. In some emergent settings such as natural disasters it’s important to train a system rapidly. Therefore we develop the following effective methods to rank and select high-quality annotated sentences.
5τ = 0 in our experiment. 6n = 20 in our experiment.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Commonness: we prefer sentences that include common entities appearing frequently in Wikipedia. We rank names by their frequency and dynamically set the frequency threshold to select a list of common names. We first initialize the name frequency threshold S to 40. If the number of the sentences is more than a desired size D for training 7, we set the threshold S = S + 5, otherwise S = S − 5. We iteratively run the selection algorithm until the size of the training set reaches D for a certain S.
Topical Relatedness: Various criteria should be adopted for different scenarios. Using an emergent disaster setting as a use case, we prefer sentences that include entities related to disaster related topics. We run an English name tagger (Manning et al., 2014) and entity linker (Pan et al., 2015) on the Leidos corpus released by the DARPA LORELEI program 8. The Leidos corpus consists of documents related to various disaster topics. Based on the linked Wikipedia pages, we rank the frequency of Wikipedia categories and select the top 1% categories (4,035 in total) for our experiments. Some top-ranked topic labels include “International medical and health organizations”, “Human rights organizations”, “International development agencies”, “Western Asian countries”, “Southeast African countries”and “People in public health”. Then we select the annotated sentences including names (e.g., “World Health Organization”) in all languages labeled with these topic labels.
2.5 KB Derived Features
When a Wikipedia user tries to link an entity mention in a sentence to an existing page, she/he will mark the title (the entity’s canonical form, without affixes) within the mention using brackets “[[]]”, from which we can naturally derive a word’s stem and affixes for free. For example, from the Wikipedia markups of the following Turkish sentence: “Kıta Fransası, güneyde [[Akdeniz]]den kuzeyde
[[Manş Denizi]] ve [[Kuzey Denizi]]ne,
doğuda [[Ren Nehri]]nden batıda [[Atlas
Okyanusu]]na kadar yayılan topraklarda yer alır. (Metropolitan France extends from the Mediterranean Sea to the English Channel and the North Sea, and from the Rhine to the Atlantic
7D = 30,000 in our experiment. 8http://www.darpa.mil/program/low-resource-languages-
for-emergent-incidents
Ocean.)”, we can learn the following suffixes: “den”, “ne”, “nden” and “na”. We use such affix lists to perform basic word stemming, and use them as additional features to determine name boundary and type. For example, “den” is a noun suffix which indicates ablative case in Turkish. [[Akdeniz]]den means “from Mediterranean Sea”. Thus, if a token contains a suffix “den”, it is likely to be a location name. Note that this approach can only perform morphology analysis for words whose stem forms and affixes are directly concatenated.
Table 1 summarizes name tagging features.
Features Descriptions Form Lowercase forms of (w−1, w0, w+1) Case Case of w0 Syllable The first and the last character of w0 Stem Stems of (w−1, w0, w+1) Affix Affixes of (w−1, w0, w+1) Gazetteer Cross-lingual gazetteers learned from training data Embeddings Character embeddings and word embed-
dings 9learned from training data
Table 1: Name Tagging Features
3 Cross-lingual Entity Linking
3.1 Overview
After we extract names from documents in a source language, we try to translate them into English by automatically mined word translation pairs (Section 3.2), and then link translated English mentions to an external English KB (in this paper we use DBpedia) (Section 3.3). The overall linking process is illustrated in Figure 3.
3.2 Name Translation
If the source language is not English, an important step in the linking process is to translate identified name mentions into English. We can utilize crosslingual Wikipedia title pairs for exact name string matching. For example, “Pekin” and “Beijing” is a Turkish-English Wikipedia title pair. However, our experiments as well as previous work (Nothman et al., 2012) show that only a small percentage of name mentions can be covered in this way. Fortunately, these cross-lingual title pairs, generated through crowd-sourcing, generally follow a consistent style and format in each language. For example, from Table 2 we can see that the order of
9For languages that don’t have space, we consider each character as a token, and use character embeddings only.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
m1 m2 m3
m4 m5 m6
t5
t1 t4 t3
t6 t2
Translate (e.g., m1 to t1)
Construct Knowledge
Networks (KNs)
KNs in KB Salience, Similarity and Coherence Comparison
Tagged Mentions
Linking
KNs in Source
Translated and Linked Mentions e1t1m1 e1t2m2 e2t3m3
e3t4m4 t5m5 NIL t6m6 NIL
Figure 3: Cross-lingual Entity Linking Overview
Extracted Cross-lingual Wikipedia Title Pairs “Pekin”
Pekin Beijing Pekin metrosu Beijing Subway Pekin Ulusal Stadyumu Beijing National Stadium “Teknoloji” Nükleer teknoloji Nuclear technology Teknoloji transferi Technology transfer Teknoloji eğitimi Technology education “Enstitüsü” Torchwood Enstitüsü Torchwood Institute Hudson Enstitüsü Hudson Institute Smolny Enstitüsü Smolny Institute
“Pekin Teknoloji” [NONE] “Teknoloji Enstitüsü”
Kraliyet Teknoloji Enstitüsü Royal Institute of Technology Karlsruhe Teknoloji Enstitüsü Karlsruhe Institute of Technology Georgia Teknoloji Enstitüsü Georgia Institute of Technology
“Pekin Teknoloji Enstitüsü” [NONE]
Mined Word Translation Pairs Word Translation Alignment
Confidence
pekin Beijing Exact Match beijing 0.5263 peking 0.3158
teknoloji technology 0.8890 technology 0.0233
technological 0.0174
enstitüsü institute 0.2765
of 0.2028 for 0.0221
Table 2: Word Translation Mining from Crosslingual Wikipedia Title Pairs
modifier and head word keeps consistent in Turkish and English titles. Therefore, we propose a new and simple method to mine word and phrase translation pairs as follows.
For each name mention, we generate all possi-
ble combinations of continuous tokens. For example, no Wikipedia titles contain the Turkish name string “Pekin Teknoloji Enstitüsü (Beijing Institute of Technology)”. We generate the following 6 combinations: “Pekin”, “Teknoloji”, “Enstitüsü”, “Pekin Teknoloji”, “Teknoloji Enstitüsü” and “Pekin Teknoloji Enstitüsü”. We then extract all cross-lingual Wikipedia title pairs containing each combination. Finally we run GIZA++ (Och and Ney, 2003) to extract word for word translations from these title pairs. Using this exhaustive search method we can obtain many word translations as shown in Table 2.
3.3 Entity Linking
Given a set of tagged name mentions M = {m1,m2, ...,mn}, we first obtain their English translations T = {t1, t2, ..., tn} using the approach described above. Then we apply an unsupervised collective inference approach to link T to the KB, similar to some previous work (Kulkarni et al., 2009; Fernandez et al., 2010; Radford et al., 2010; Cucerzan, 2011; Han and Sun, 2011; Ratinov et al., 2011; Chen and Ji, 2011; Kozareva et al., 2011; Dalton and Dietz, 2013; Pan et al., 2015; Wang et al., 2015). We construct knowledge networks (KNs) g(ti) for T based on their cooccurrence within a context window 10. For each translated name mention ti, an initial list of candidate entities E(ti) = {e1, e2, ..., ek} is generated based on a surface form dictionary mined from KB properties (e.g., redirects, names, aliases). If no surface form can be matched then we determine the mention as unlinkable. Then we construct knowledge network g(ej) for each entity candidate ej in ti’s entity candidate list E(ti). We compute the similarity between g(ti) and g(ej) based on three measures as described in (Pan et al., 2015): salience, similarity and coherence, and select the candidate entity with the highest score as the appropriate entity for linking.
4 Experiments
4.1 Performance on Wikipedia Data
We first conduct an evaluation using Wikipedia data as “silver-standard”. For each language, we use 70% of the selected sentences for training and 30% for testing. For entity linking, we don’t have ground truth for unlinkable mentions, so we only
10In our experiments, we use the previous four and next four name mentions as a context window.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
587
588
589
590
591
592
593
594
595
596
597
598
599
compute linking accuracy for linkable name mentions. Table 3 presents the overall performance for three coarse-grained entity types: person, organization and geo-political entity/location.
Not surprisingly, name tagging performs better for languages with more mentions. The fscore is generally higher than 80% when there are more than 30K mentions, and it significantly drops when there are less than 250 mentions. The languages with low name tagging performance can be categorized into three types: (1) the number of mentions is less than 2K, such as AtlanticCongo (Wolof), Berber (Kabyle), Chadic (Hausa), Oceanic (Fijian), Hellenic (Greek), Igboid (Igbo), Mande (Bambara), Kartvelian (Georgian, Mingrelian), Timor-Babar (Tetum), Tupian (Guarani) and Iroquoian language groups; Precision is generally higher than recall for most of these languages, because the small number of linked mentions is not enough to cover a wide variety of entities. (2) there is no space between words, including Chinese, Thai and Japanese; (3) they are not written in latin script, such as the Dravidian group (Tamil, Telugu, Kannada, Malayalam).
The training instances for various entity types are quite imbalanced for many languages. For example, Cebuano data includes 1,251 person names, 787,349 geo-political entity/location names and 8,292 organization names. As a result, the performance of organization is the lowest, while geo-political entities and locations achieve higher than 75% F-scores for most languages. The linking accuracy is higher than 80% for most languages.
Also note that since we don’t have perfect annotations on Wikipedia data for any language, these results can be used to estimate how predictable our “silver-standard” data is, but they are not directly comparable to traditional name tagging results measured against gold-standard data annotated by human.
4.2 Performance on Non-Wikipedia Data
Therefore, in order to have more direct comparison with state-of-the-art name taggers trained from human annotated gold-standard data, we conduct experiments on Non-Wikipedia data in 9 languages for which we have human annotated ground truths from the DARPA LORELEI pro-
11The mapping to language names can be found at https://meta.wikimedia.org/wiki/List of Wikipedias
gram. Table 4 shows the data statistics. The documents are from news sources and discussion fora.
We can see that our approach advances stateof-the-art language-independent methods (Zhang et al., 2016a; Tsai et al., 2016) on the same data sets for most languages, and achieves 5.8% - 17.6% lower F-scores than the models trained from manually annotated gold-standard documents that include thousands of name mentions. We also measured the impact of our morphology analysis methods based on the affix lists derived from Wikipedia markups on two morphologicallyrich languages: Turkish and Uzbek. The morphology features contributed 11.1% and 7.1% absolute F-score gains to Turkish and Uzbek respectively.
4.3 Impact of Self-Training
Using Turkish as a case study, the learning curves of self-training on Wikipedia and Non-Wikipedia test sets are shown in Figure 4. We can see that self-training provides significant improvement for both Wikipedia (6% absolute gain) and NonWikipedia test data (12% absolute gain). As expected the learning curve on Wikipedia data is more smooth and converges more slowly than that of Non-Wikipedia data.
5 Related Work
Our work was mainly inspired from previous work that leveraged Wikipedia markups to train name taggers (Nothman et al., 2008; Dakka and Cucerzan, 2008; Mika et al., 2008; Ringland et al., 2009; Alotaibi and Lee, 2012; Nothman et al., 2012; Althobaiti et al., 2014). Our approach doesn’t need any manual annotations or languagespecific features, while generates both coarsegrained and fine-grained types.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
L M F A L M F A L M F A L M F A en 70M 91.8 85.3 lb 102K 81.5 88.4 nah 8.4K 89.9 90.6 m-b 1.6K 78.3 81.3 de 14M 89.0 88.4 tl 102K 92.7 90.3 b-s 8.1K 84.5 88.0 rw 1.6K 95.4 92.7 fr 12M 93.3 92.4 sq 97K 94.1 92.9 rm 8.0K 82.0 91.3 ln 1.6K 82.8 93.3 es 6.9M 93.9 93.4 sco 94K 86.8 89.6 azb 7.9K 88.4 90.6 kl 1.5K 75.0 95.5 it 6.9M 96.6 94.7 ce 90K 99.4 71.0 ay 7.9K 88.5 96.2 sn 1.5K 95.0 99.3 ja 6.5M 79.2 91.8 pnb 81K 90.8 86.2 ps 7.7K 66.9 91.8 av 1.4K 82.0 83.7 ru 6.4M 90.1 89.2 als 81K 85.0 91.2 mi 7.5K 95.9 24.5 wo 1.3K 87.7 97.3 pl 6.0M 90.0 91.3 lmo 80K 98.3 89.0 gag 7.3K 89.3 84.0 xal 1.3K 98.7 90.9 nl 5.6M 93.2 91.9 is 76K 80.2 83.2 am 7.0K 84.7 83.0 na 1.2K 87.6 88.7 sv 4.8M 93.6 89.7 fy 75K 86.6 91.4 nrm 7.0K 96.4 99.6 ltg 1.2K 74.3 92.1 pt 3.9M 90.7 92.4 an 73K 93.0 93.1 nso 6.9K 98.9 99.4 tet 1.2K 73.5 92.3 sh 2.9M 97.8 95.6 gu 70K 76.0 92.4 sc 6.8K 78.1 95.2 ksh 1.2K 56.0 83.6 uk 2.3M 91.5 89.4 qu 63K 92.5 88.2 so 6.5K 85.8 95.7 haw 1.1K 88.0 84.6 zh 2.1M 82.0 89.7 ast 60K 89.2 94.0 co 6.0K 85.4 89.9 pfl 1.1K 42.9 80.4 fi 2.0M 93.4 94.7 z-y 58K 87.3 88.4 stq 6.0K 70.0 92.6 tpi 1.1K 83.3 90.1 cs 1.9M 94.6 92.0 sw 56K 93.4 93.0 pcd 5.8K 86.1 75.8 lo 1.0K 52.8 92.0 no 1.9M 94.1 93.8 ga 55K 85.3 91.3 dsb 5.8K 84.7 82.1 ki 1.0K 97.5 90.0 hu 1.9M 95.9 92.8 jv 54K 82.6 87.8 wa 5.8K 81.6 82.0 ty 1.0K 86.7 89.8 ca 1.8M 90.3 91.6 new 51K 98.2 97.9 wuu 5.8K 79.7 90.8 ady 979 92.7 95.5 fa 1.4M 96.4 83.8 nds 44K 84.5 88.2 frr 5.7K 70.1 86.3 ig 968 74.4 45.5 sr 1.4M 95.3 93.3 ht 42K 98.9 99.8 lad 5.6K 92.3 95.6 tyv 903 91.1 98.8 ko 1.3M 90.6 91.4 pms 40K 98.0 89.5 z-c 5.4K 88.2 87.0 tn 902 76.9 90.1 tr 1.2M 90.5 87.3 bar 39K 97.1 96.3 nap 4.9K 86.9 88.6 cu 898 75.5 76.9 vi 1.2M 89.6 82.0 kn 39K 60.1 92.1 crh 4.9K 90.1 89.9 sm 888 80.0 85.3 ro 1.1M 90.6 85.0 ba 38K 93.8 68.3 km 4.6K 52.2 89.9 to 866 92.3 69.2 he 1.0M 79.0 92.5 arz 35K 77.8 89.3 c-z 4.5K 75.0 89.2 tum 831 93.8 98.5 ar 1.0M 88.3 85.3 ne 31K 81.5 92.5 lez 4.4K 84.2 82.3 r-r 750 93.0 90.5 bg 871K 65.8 88.4 cv 31K 95.7 97.4 mai 4.3K 99.7 90.0 om 709 74.2 81.1 ceb 796K 96.3 86.6 fo 30K 83.6 78.1 hak 4.3K 85.5 88.1 glk 688 59.5 80.8 sk 779K 87.3 90.3 li 28K 89.4 91.6 ang 4.2K 84.0 96.7 lbe 651 88.9 68.6 id 775K 87.8 87.3 ckb 28K 88.1 76.1 r-t 4.2K 88.1 89.0 bjn 640 64.7 89.5 da 717K 87.1 85.8 gd 28K 92.8 91.7 udm 4.2K 88.9 85.0 srn 619 76.5 90.1 z-m 680K 99.3 89.2 io 26K 87.2 95.5 csb 4.1K 87.0 94.4 mdf 617 82.2 97.3 eo 647K 88.7 81.4 mn 25K 76.4 84.4 lij 4.1K 72.3 92.0 tw 572 94.6 90.4 eu 640K 82.5 88.3 tg 25K 88.3 90.6 nov 4.0K 77.0 95.0 pih 555 87.2 90.0 ms 569K 86.8 84.1 bug 25K 99.9 90.0 ace 4.0K 81.6 90.9 rmy 551 68.5 90.0 sl 537K 89.5 90.1 scn 25K 93.2 89.2 gn 4.0K 71.2 89.3 lg 530 98.8 89.9 el 516K 84.6 88.3 ku 23K 83.2 85.1 koi 4.0K 89.6 93.5 chr 530 70.6 90.9 hr 490K 82.8 88.5 pa 21K 74.8 84.3 mhr 3.9K 86.7 94.7 ha 517 75.0 80.0 et 456K 86.8 89.9 yi 20K 76.9 87.2 min 3.8K 85.8 88.5 got 506 91.7 92.3 lt 423K 86.3 87.2 sa 20K 73.9 93.0 ext 3.7K 77.8 91.6 ab 506 60.0 93.3 nn 393K 88.1 86.7 hsb 20K 91.5 74.9 kab 3.3K 75.7 84.3 bi 490 88.5 93.3 th 391K 56.2 87.7 vls 19K 78.2 89.1 szl 3.0K 82.7 92.7 st 455 84.4 89.8 gl 382K 87.4 88.2 ilo 18K 90.3 93.1 tk 2.9K 86.3 90.1 chy 450 85.1 89.9 war 369K 94.9 87.4 vec 15K 87.9 93.8 kv 2.9K 89.7 97.2 iu 450 66.7 83.3 sim 368K 85.7 91.6 bpy 15K 98.3 98.0 f-v 2.9K 65.4 88.8 zu 449 82.3 89.9 lv 349K 92.1 89.8 my 13K 51.5 92.4 gan 2.9K 84.9 90.9 pnt 445 61.5 87.5 ur 309K 96.4 83.4 mzn 13K 86.4 86.9 fur 2.8K 84.5 89.2 ik 436 94.1 54.3 ka 290K 79.8 89.5 os 13K 87.4 89.4 mwl 2.7K 76.1 89.4 lrc 416 65.2 90.0 hy 280K 90.4 88.9 ky 13K 71.8 88.4 nv 2.7K 90.9 94.1 bm 386 77.3 87.5 vo 269K 98.5 97.5 xmf 13K 73.4 94.4 sd 2.7K 65.8 90.9 za 382 57.1 90.3 uz 266K 98.3 94.8 or 13K 86.4 73.5 bxr 2.6K 75.0 91.9 mo 373 69.6 90.2 la 260K 90.8 89.4 pam 13K 87.2 92.9 bo 2.6K 70.4 88.9 ss 362 69.2 71.4 mk 236K 93.4 83.3 si 12K 87.7 94.1 frp 2.5K 86.2 98.5 ee 297 63.2 90.0 bs 227K 84.8 89.9 sah 12K 91.2 78.8 myv 2.5K 88.6 94.4 dz 262 50.0 80.0 kk 224K 88.3 81.8 mt 12K 82.3 90.3 cdo 2.5K 91.0 96.2 ak 258 86.8 92.6 be 205K 84.1 88.3 mrj 12K 97.0 55.3 gom 2.4K 88.8 90.0 sg 245 100.0 89.9 ta 193K 77.9 88.2 n-n 11K 92.6 92.9 bh 2.2K 92.6 92.8 ts 236 93.3 88.9 az 176K 85.1 86.0 as 11K 89.6 83.0 ug 2.1K 79.7 92.4 rn 185 40.0 75.0 hi 170K 86.9 88.0 vep 11K 85.8 89.8 kaa 2.1K 55.2 89.5 ve 183 100.0 83.3 cy 169K 90.7 89.3 diq 10K 79.3 80.9 krc 2.1K 84.9 88.9 ny 169 56.0 75.0 bn 168K 93.8 87.2 zea 10K 86.8 90.3 dv 2.0K 76.2 94.3 ff 168 76.9 88.9 af 142K 85.7 91.1 hif 10K 81.1 96.7 rue 1.9K 82.7 72.1 ch 159 70.6 90.0 br 138K 87.0 85.5 se 9.7K 90.3 83.5 eml 1.8K 83.5 88.5 xh 141 35.3 75.0 mg 134K 98.7 92.9 ie 9.5K 88.8 95.5 arc 1.8K 68.5 82.0 fj 126 75.0 80.0 te 128K 80.5 86.1 su 9.4K 72.7 89.2 pdc 1.8K 78.1 91.1 ks 124 75.0 83.3 b-x 126K 85.1 87.7 bcl 9.3K 82.3 94.0 kbd 1.7K 74.9 80.6 ti 52 94.2 90.0 ml 125K 82.4 88.8 yo 9.2K 94.0 93.0 pap 1.7K 88.8 59.1 cr 49 91.8 89.8 tt 119K 87.7 91.4 ia 8.9K 75.4 90.5 jbo 1.7K 92.4 96.5 pi 41 83.3 90.2 oc 118K 92.5 90.0 kw 8.7K 94.0 96.1 pag 1.7K 91.2 89.5 mr 109K 82.4 91.0 gv 8.4K 84.8 89.1 kg 1.6K 82.1 90.1
Table 3: Performance on Wikipedia Data (L: language ID 11; M: the number of name mentions in selected annotated data; F: name tagging F-score (%); A: linking accuracy (%))
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Language Gold Training Silver Training Test Bengali 8,760 22,093 3,495 Hungarian 3,414 34,022 1,320 Russian 2,751 35,764 1,213 Tamil 7,033 25,521 4,632 Tagalog 4,648 15,839 3,351 Turkish 2,172 37,058 1,353 Uzbek 3,137 64,242 2,056 Vietnamese 2,261 63,971 987 Yoruba 4,061 9,274 3,395
Table 4: # of Names in Non-Wikipedia Data
Language Training from Gold Training from Silver
(Zhang et al., 2016a)
(Tsai et al., 2016)
Bengali 61.6 44.0 34.8 43.3 Hungarian 63.9 47.9 - - Russian 61.8 49.4 - - Tamil 42.2 35.7 33.8 29.6 Tagalog 70.7 58.3 51.3 65.4 Turkish 57.3 51.5 43.6 47.1 Uzbek 56.0 44.2 - - Vietnamese 54.3 44.5 - - Yoruba 55.1 37.6 36.0 36.7
Table 5: Name Tagging F-score (%) on NonWikipedia Test Data
Many fine-grained entity typing approaches (Fleischman and Hovy, 2002; Giuliano, 2009; Ekbal et al., 2010; Xiao and Weld, 2012; Yosef et al., 2012; Nakashole et al., 2013; Gillick et al., 2014; Yogatama et al., 2015; Corro et al., 2015) also created annotations based on Wikipeida anchor links. Our framework performs both name identification and typing and takes advantage of richer structures in the KBs. Previous work on Arabic name tagging (Althobaiti et al., 2014) extracted entity titles as a gazetteer for stemming, and thus it cannot handle unknown names. We developed a new method to derive generalizable affixes for morphologically rich language based on Wikipedia markups.
Wikipedia has also been used as additional features to improve various Information Extraction (IE) tasks, including name tagging (Kazama and Torisawa, 2007), coreference resolution (Ponzetto and Strube, 2006), relation extraction (Chan and Roth, 2010) and event extraction (Hogue and Nothman, 2014). Other automatic name annotation generation methods have been proposed, including KB driven distant supervision (An et al., 2003; Mintz et al., 2009; Ren et al., 2015) and cross-lingual projection (Li et al., 2012; Kim et al., 2012; Che et al., 2013; Wang et al., 2013; Wang
and Manning, 2014; Zhang et al., 2016b). Some recent research (Zhang et al., 2016a; Littell et al., 2016; Tsai et al., 2016) under the DARPA LORELEI program focused on developing name tagging techniques for low-resource languages. These approaches require English annotations for projection (Tsai et al., 2016), some input from a native speaker, either through manual annotations (Littell et al., 2016), or a linguistic survey (Zhang et al., 2016a). Without using any manual annotations, our name taggers outperform previous methods on the same data sets for many languages. NIST TAC-KBP Tri-lingual entity linking (Ji et al., 2016) focused on three languages: English, Chinese and Spanish. (McNamee et al., 2011) extended it to 21 languages. But their methods required labeled data and name transliteration. We share the same goal as (Sil and Florian, 2016) to extend cross-lingual entity linking to all languages in Wikipedia. They exploited Wikipedia links to train a supervised linker. We mine reliable word translations from cross-lingual Wikipedia titles, which enables us to adopt unsupervised English entity linking techniques.
6 Conclusions and Future Work
We developed a simple yet effective framework that can extract names from 282 languages and link them to an English KB. This framework follows a fully automatic training and testing pipeline, without the needs of any manual annotations or knowledge from native speakers. We evaluated our framework on both Wikipedia articles and external formal and informal texts and obtained promising results. To the best of our knowledge, our multilingual name tagging and linking framework is applied to the largest number of languages. We release the following resources for each of these 282 languages: “silver-standard” name tagging and linking annotations with multiple levels of granularity, morphology analyzer if it’s a morphologically-rich language, and an endto-end name tagging and linking system. In this work, we treat all languages independently when training their corresponding name taggers. In the future, we will explore the topological structure of related languages and exploit cross-lingual knowledge transfer to enhance the quality of extraction and linking. The general idea of deriving noisy annotations from KB properties can also be extended to other IE tasks such as relation extraction.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths:    - The paper states clearly the contributions from the beginning     - Authors provide system and dataset    - Figures help in illustrating the approach    - Detailed description of the approach    - The authors test their approach performance on other datasets and compare to other published work  - Weaknesses:    -The explanation of methods in some paragraphs is too detailed and there is no mention of other work and it is repeated in the corresponding method sections, the authors committed to address this issue in the final version.    -README file for the dataset [Authors committed to add README file]  - General Discussion:    - Section 2.2 mentions examples of DBpedia properties that were used as features. Do the authors mean that all the properties have been used or there is a subset? If the latter please list them. In the authors' response, the authors explain in more details this point and I strongly believe that it is crucial to list all the features in details in the final version for clarity and replicability of the paper.     - In section 2.3 the authors use Lample et al. Bi-LSTM-CRF model, it might be beneficial to add that the input is word embeddings (similarly to Lample et al.)    - Figure 3, KNs in source language or in English? (since the mentions have been translated to English). In the authors' response, the authors stated that they will correct the figure.    - Based on section 2.4 it seems that topical relatedness implies that some features are domain dependent. It would be helpful to see how much domain dependent features affect the performance. In the final version, the authors will add the performance results for the above mentioned features, as mentioned in their response.     - In related work, the authors make a strong connection to Sil and Florian work where they emphasize the supervised vs. unsupervised difference. The proposed approach is still supervised in the sense of training, however the generation of training data doesn t involve human interference