Integrating text and knowledge into a unified semantic space has attracted significant research interests recently. However, the ambiguity in the common space remains a challenge, namely that the same mention phrase usually refer to various entities. In this paper, to deal with the ambiguity of entity mentions, we propose a novel Multi-Prototype Mention Embedding model, which learns multiple sense embeddings for each mention by jointly modeling words from textual contexts and entities derived from a knowledge base. In addition, we further design an efficient language model based approach to disambiguate each mention to a specific sense. In experiments, both qualitative and quantitative analysis demonstrate the high quality of the word, entity and multi-prototype mention embeddings. Using entity linking as a study case, we apply our disambiguation method as well as the multi-prototype mention embeddings on the benchmark dataset, and achieve the state-of-the-art.
1 000
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
061
062
063
064
065
066
067
068
069
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Jointly learning text and knowledge representations in a unified vector space greatly benefits many Natural Language Processing (NLP) tasks, such as knowledge graph completion (Han et al., 2016; Wang and Li, 2016), relation extraction (Weston et al., 2013), word sense disambiguation (Mancini et al., 2016) and entity linking (Huang et al., 2015).
Existing work can be roughly divided into two categories. One is encoding words and entities into a unified vector space using Deep Neural Networks (DNN). These methods suffer from the
problem of expensive training and have great limitations on the size of word and entity vocabulary (Han et al., 2016; Toutanova et al., 2015; Wu et al., 2016). The other is to learn word and entity embeddings separately, and then align similar words and entities into a common space with the help of Wikipedia hyperlinks, so that they share similar representations (Wang et al., 2014; Yamada et al., 2016).
However, there are two major problems arising from directly integrating word and entity embeddings into a unified semantic space. First, mention phrases are of great ambiguity and can refer to multiple entities in the common space. As shown in Figure 1, the same mention independence day (m1) can either refer to a holiday: Independence Day (US) or a film: Independence Day (film). Second, an entity often has various aliases when mentioned in various contexts, which implies a much larger size of mention vocabulary compared with entities. For example, in Figure 1, the document d2 and d3 describes the same entity Independence Day (US) (e2) with distinct mentions: independence day and July 4th. We observe a tens of millions of mention vocabulary on 5 millions of entities in Wikipedia.
To address these issues, we propose to learn multiple embeddings for each mention. Each embedding denotes a meaning, namely mention sense. Similar as the Word Sense Disambiguation
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
(WSD) problem (Reisinger and Mooney, 2010; Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Li and Jurafsky, 2015), each mention sense can be disambiguated by textual context information as well as its reference entity. For example, in Figure 1, the Independence Day in d2 shares similar contexts with the July 4th in d3 and refers to the same entity. In contract, it may also refer to another entity when occur in distinct contexts, such as d1. Further, to reduce the size of mention vocabulary, we assume that mentions share the same sense representation if they refer to the same entity, e.g., the Independence Day in d2 and July 4th in d3 will have the same mention representation as they refer to the same holiday. Thus, mention senses can serve as a repository and each mention can be mapped to them by a pre-defined dictionary.
In this paper, we propose a novel MultiPrototype Mention Embedding (MPME) model, which jointly learns the representations of words, entities, and mentions at sense level. The basic idea behind it is to use both textual context information and knowledge of reference entity to distinguish different mention senses. Following the frameworks in (Wang et al., 2014; Yamada et al., 2016), we use separate models to learn the representations for words, entities and mentions, and further align them by a unified optimization objective. Extending from skip-gram model and CBOW model, our model can be trained efficiently (Mikolov et al., 2013a,b) even on a large scale corpus. In addition, we also design an language model based approach to determine the sense for each mention in a document based on multi-prototype mention embeddings.
For evaluation, we first provide qualitative analysis to verify the effectiveness of MPME to bridge text and knowledge representations at the sense level. Then, separate tasks for words and entities show improvements by incorporating our word, entity and mention representations. Finally, using entity linking as a study case, experimental results on the benchmark dataset demonstrate the effectiveness of our embedding model as well as the disambiguation method.
2 Preliminaries
In this section, we formally define the input and output of multi-prototype mention embedding.
A knowledge baseKB contains a set of entities
E = {ej}, and their relations. Following (Yamada et al., 2016), we use Wikipedia as the given knowledge base, and organize it as a directed network, namely knowledge network: nodes denote entities, and edges are outlinks from wikipedia pages. We define the neighbors of ej on the network as N (ej).
A text corpus D is a set of sequential words D = {w1, · · · , wi, · · · , w|D|}, where wi is the ith word and |D| is the length of the word sequence. We use ml to denote an entity mention (perhaps consisting of multiple words). We define an annotated text corpus asD′ = {x1, · · · , xi, · · · , x|D′|}, where xi corresponds to a word wi or a mention ml. We define the words around xi within a predefined window as its context words C(·).
An Anchor indicates the Wikipedia hyperlinks from mention ml linking to entity ej , and is represented as a pair< mh, ej >∈ A. The anchors provide mention boundaries as well as their reference entities from Wikipedia articles. These Wikipedia articles are used as annotated text corpusD′ in this paper.
Multi-Prototype Mention Embedding . Given a KB, an annotated text corpus D′ and a set of anchorsA, multi-prototype mention embedding is to learn multiple sense embeddings sjl ∈ Rk for each mention ml as well as word embeddings w and entity embeddings e. We useM∗l = {slj} to denote the sense set of mention ml, where each slj refers to an entity ej . Thus, the vocabulary size is reduced to a fixed number |{s∗j}| = |E|. We use s∗j to denote the shared sense of mentions referring to entity ej . Example As shown in Figure 1, Independence Day (m1) is learned with two mention senses s11, s 1 2, and July 4th (m2) has one mention sense s22. Based on the assumption in Section 1, we have s∗2 = s 1 2 = s 2 2 referring to entity Independence Day (US) (e2).
3 An Overview of Our Method
Given knowledge base KB, annotated text corpus D′ and a set of anchors A, we aim to jointly learn word, entity and mention sense representations: w, e, s.
As shown in Figure 2, our framework contains two key components:
Mention Sense Mapping To reduce the mention vocabulary, each mention is mapped to a set of shared mention senses according to a predefined
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ACL 2017 Submission 104. Confidential Review Copy. DO NOT DISTRIBUTE.
Representation Learning
Entity Representation Learning
Text Representation Learning
bands played it during public events,
such as [[Independence Day
(US)|July 4th]] celebrations
… In the 1996 action film [[Independence Day (film)|Independence Day]], the United States
military uses alien technology captured …
4
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336
337
338
339
340
341
342
343
344
345
346
347
348
349
350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386
387
388
389
390
391
392
393
394
395
396
397
398
399
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. computing similarity between word and mention embeddings referring to that entity. 3 Method In this section, we present three main components in MPME: text model, knowledge model and joint model, and then introduce the detailed information on training process. Finally, we briefly introduce the framework for entity linking. 3.1 Skip-gram model capable of iterative learning; capable of learning more mention names; capable of tuning mention sense via text model; capable of NIL sense; 1. take pre-trained word and entity embeddings as input; 2. collect mention name to entity title mapping; use anchor to annotate each mention. each mention corresponds multiple sense; each sense relates to one entity title. 3. given the context and the mention’s sense, predict the entity; got entity title embedding. 4. each title has multiple vector, each corresponds to a different entity. maintain the context cluster; the cluster role. 5. text model again, use context to predict mention sense, to predict the context; also can predict a new sense, called NIL in EL tasks, future work. 3.2 Text model Lw = TX t=1 log P (wt+j |wmt , si)P (si|wcontext) + TX t=1 X cjc,j 6=0 log P (wt+j |wt) (1) DX CX P (wt+j |wmt , si)P (si|wmt , wcontext)
3.3 Knowledge model KBX NX
P (eneighbor|ei)
3.4 Joint model AX
P (ej |wmt , si) + P (ej |wcontext)
eIndependence Day (film)
eIndependence Day (US)
wm1Independence Day
wm2Independence Day 3.5 Training 3.6 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align mention with entity using single prototype model. 4.3 Parameter Setting 4.4 Text Evaluation 4.5 Entity Evaluation 4.6 EL evaluation 5 Related Work 6 Conclusion References Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125. Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. CoRR, abs/1504.07678. Massimiliano Mancini, José Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2016. Embedding words and senses together via joint knowledgeenhanced training. CoRR, abs/1612.02703.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111–3119.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. ACL Association for Computational Linguistics.
4
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336
337
338
339
340
341
342
343
344
345
346
347
348
349
350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 73 374 375 376 377 378 379 380 381 382 383 384 385 386
387
388
389
390
391
392
393
394
395
396
397
398
399
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. computing similarity between word and mention embeddings referring to that entity. 3 Method In this section, we present three main components in MPME: text model, knowledge model and joint model, and then introduce the detailed information on training process. Finally, we briefly introduce the framework for entity linking. 3.1 Skip-gram model capable of iterative learning; capable of learning more mention names; capable of tuning mention sense via text model; capable of NIL sense; 1. take pre-trained word and entity embeddings as input; 2. collect mention name to entity title mapping; use anchor to annotate each mention. each mention corresponds multiple sense; each sense relates to one entity title. 3. giv n the context and the mentio ’s sense, predict the entity; got entity title embedding. 4. each title has multiple vector, each corre p nds to a diff ren entity. maintain the context clust r; the clust r role. 5. text model again, us context to predict mention sense, to predict the context; lso can predict a n w sense, called NIL in EL tasks, future work. 3.2 Text model Lw = TX t=1 log P (wt+j |wmt , si)P (si|wcontext) + TX t=1 X cjc,j 6=0 log P (wt+j |wt) (1) DX CX P (wt+j |wmt , si)P (si|wmt , wcontext) 3.3 Knowledge model KBX NX
P (eneighbor|ei)
3.4 Joint model AX
P (ej |wmt , si) + P (ej |wcontext)
eIndependence Day (film)
eIndependence Day (US)
wm1Independence Day
wm2Independence Day 3.5 Training 3.6 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align mention with entity using single prototype model. 4.3 Parameter Setting 4.4 Text Evaluation 4.5 Entity Evaluation 4.6 EL ev luation 5 Related Work 6 Conclusion Refer nces Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowl d e graph completion. CoRR, abs/161 .04125. Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. CoRR, abs/1504.07678. Massimiliano Mancini, José Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2016. Embedding words and senses together via joint knowledgeenhanced training. CoRR, abs/1612.02703. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pages 3111–3119.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Ch dhury, and Micha l Gamon. 2015. Representing text for joint embedding of text and knowledge bases. ACL Associatio for Computational Lin uistics.
Anchor
Text 4
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. tities by modeling semantic network constructed from the given knowledge base. Joint model learns multiple mention embeddings by maximizing the probability of the mention in the context referring to target entity. Text model . Kg model . Joint model . As shown in Figure 3, for each anchor ai = (mj , ek), we firstly replace the mention name with entity title m⇤t via pre-defined mapping rules. Given KB, D and the mapped anchors A, we iteratively train the three models until convergence using a joint optimization objective, which will be introduced later.
Though following the basic components of three models in (Wang et al., 2014; Yamada et al., 2016), MPME designs different structure in text model and joint model to combine text and knowledge in phrase level via multi-prototype mention embedding, rather than aligning between singleprototype word embeddings and entity embeddings. Actually, MPME is flexible to utilize pretrained entity embeddings from arbitrary knowledge representation model, and enjoys their advantages of different aspects in knowledge bases2. This is reasonable because we output two separately semantic vector spaces for text and knowledge respectively, while we can still obtain the relatedness between word and entity indirectly by computing similarity between word and mention embeddings referring to that entity.
3 Method
In this section, we present three main components in MPME: text model, knowledge model and joint model, and then introduce the detailed information on training process. Finally, we briefly introduce the framework for entity linking.
3.1 Skip-gram model capable of iterative learning; capable of learning more mention names; capable of tuning mention sense via text model; capable of NIL sense; 1. take pre-trained word and entity embeddings as input; 2. collect mention name to entity title mapping; use anchor to annotate each mention. each mention corresponds multiple sense; each sense relates
2Thus, MPME only trains text model and joint model.
to one entity title. 3. given the context and the mention’s sense, predict the entity; got entity title embedding. 4. each title has mult ple vector, each corresponds to a differe t e tity. m intain the co - text cluster; the cluster role. 5. text model again, use context to predict menti n se s , to predict the context; also can predict a new sense, called NIL in EL tasks, future work. 3.2 Text model Lw = TX t=1 log P (wt+j |wmt , si)P (si|wcontext) + TX t=1 X cjc,j 6=0 log P (wt+j |wt) (1) DX CX P (wt+j |wmt , si)P (si|wmt , wcontext) .3 Knowledge model KBX N P (eneighbor|ei)
3.4 Joint model
AX P (ej |wmt , si) + P (ej |wcontext)
eIndependence Day (film)
eIndependence Day (US)
wm1Independence Day
wm2Independence Day
wfilm
wcelebrations
wmMemorial Day
4
300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350 351 3 2 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. tities by modeling semantic network constructed from the given knowledge base. Joint model learns multiple mention embeddings by maximizing the probability of the mention in the context referring to target entity. Text model . Kg model . Joint model . As shown in Figure 3, for each anchor ai = (mj , ek), we firstly replace the mention name with entity title m⇤t via pre-defined mapping rules. Given KB, D and the mapped anchors A, we iteratively train the three models until convergence using a joint optimization objective, which will be introduced later.
Though following the basic compon n s of three models in (Wang et al., 2014; Yamada et al., 2016), MPME designs different structure i text model and joint model to combine text a d knowledge in phrase level via multi-prototype mention embedding, rather than aligning between singleprototype word embeddings and entity embeddings. Actually, MPME is flexible to utilize pretrained entity embeddings from arbitrary knowledge representation model, and enjoys their advantages of different aspects in knowledge bases2. This is reasonable because we output two separately semantic vector spaces for text and knowledge respectively, while we can still obtain the relatedness between ord and entity indirectly by computing similarity b tween word and mention embeddings referring to that entity.
3 Method
In this section, we present three main components in MPME: text model, knowledge model and joint model, and then introduce the detailed information on training process. Finally, we briefly introduce the framework for entity linking.
3.1 Skip-gram model capable of iterative learning; capable of learning more mention names; capable of tuning mention sense via text model; capable of NIL sense; 1. take pre-trained word and entity embeddings as input; 2. collect me tion name to entity title mapping; use anchor t nnotate each mention. each mention c rre ponds multiple sense; each sense relates
2Thus, MPME only trains text model and joint model.
to one entity title. 3. given the context and the mention’s sense, predict the ntity; g t ntity title emb dding. 4. each title has multiple v ctor, each corresponds to a different ntity. maint in the context cluster; the cluster role. 5. text model again, use context to predict mention sense, to predict the context; also can predict a new sense, called NIL in EL tasks, future work. 3. Text model Lw = TX t=1 log P (wt+j |wmt , si)P (si|wcontext) + TX t=1 X cjc,j 6=0 log P (wt+j |wt) (1) DX CX P (wt+j |wmt , si)P (si|wmt , wcontext) 3. Knowledge model
KBX NX P ( neighb r| i)
3.4 Joint model
AX P (ej |wmt , si) + P (ej |wcontext)
eIndependence Day (film)
eIndependence Day (US)
wm1Independence Day
wm2Independence Day
wfilm
wcelebrations
wmMemorial Day
5
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE.
3.2 Skip-gram model
g(Independence Day, )
P (N (ej)|ej) P (ej |C(mh), tsl ) P (C(wi)|wi)P (C(mh)|tsl , mh)
3.3 Text model
Lw = TX
t=1
log P (wt+j |wmt , si)P (si|wcontext)
+ TX
t=1
X
cjc,j 6=0 log P (wt+j |wt)
(1)
DX CX P (wt+j |wmt , si)P (si|wmt , wcontext)
3.4 Knowledge model KBX NX
P (eneighbor|ei)
3.5 Joint model AX
P (ej |wmt , si) + P (ej |wcontext)
3.6 Training
3.7 Integrating into GBDT for EL
4 Experiment
4.1 Data Preparation
4.2 Baseline Methods
1. directly align words with entity. 2. align mention with entity using single prototype model.
4.3 Parameter Setting
4.4 Qualitative Analysis
4.5 Entity Relatedness
4.6 Word Analogy
4.7 EL evaluation
5 Related Work
6 Conclusion
References Antoine Bordes, Nicolas Usunier, Alberto Garcı́a-
Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multirelational data. In Burges et al. (Burges et al., 2013), pages 2787–2795.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Proc ss Systems 26: 27th Annual Conference on eural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Ta e, Ne da, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. CoRR, abs/1504.07678.
Massimiliano Mancini, José Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2016. Embedding words and senses together via joint knowledgeenhanced training. CoRR, abs/1612.02703.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Burges et al. (Burges et al., 2013), pages 3111–3119.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. ACL Association for Computational Linguistics.
Zhigang Wang and Juan-Zi Li. 2016. Text-enhanced representation learning for knowledge graph. In Subbarao Kambhampati, editor, Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1293–1299. IJCAI/AAAI Press.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph and text jointly embedding. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 15 1–1601. ACL.
Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for relation extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1366–1371. ACL.
Independence Day (US)
United States
F reworks
Independence Day (film)
Memorial Day
Celebrations
Ob se
rv ed
b y
Public holidays in the United States
cat ego
ry
Will Smith
st ar
rin g
Philadelphia
bo rn country
inlink
out link inlink
Knowledge Base
5
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE.
3.2 Skip-gram model
g(Independence Day, )
P (N (ej)|ej) P (ej |C(mh), tsl ) e1 e2
P (C(wi)|wi) · P (C(mh)|tsl , mh)
(1)
t1I dependence Day t2Indep nden e Day t1Memorial Day
g(Independence Day,
Independence Day (US)) (2)
g(Independence Day) g(July 4th) (3)
3.3 Text model
Lw = TX
t=1
log P (wt+j |wmt , si)P (si|wcontext)
+ TX
t=1
X
cjc,j 6=0 log P (wt+j |wt)
(4)
DX CX P (wt+j |wmt , si)P (si|wmt , wcontext)
3.4 Knowledge model KBX NX
P (eneighbor|ei)
3.5 Joint model AX
P (ej |wmt , si) + P (ej |wcontext)
3.6 Training
3.7 Integrating into GBDT for EL
4 Experiment
4.1 Data Preparation
4.2 Baseline Methods
1. directly align words with entity. 2. align mention with entity using single prototype model.
4.3 Parameter Setting 4.4 Qualitative Analysis 4.5 Entity Relatedness 4.6 Word Analogy 4.7 EL evaluation
5 Related Work
6 Conclusion
References Antoine Bordes, Nicolas Usunier, Alberto Garcı́a-
Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multirelati nal data. In Burges et al. (Burges et al., 2013), pages 2787–2795.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Proc ssing Systems 2013. Proceedings of a eeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. CoRR, abs/1504.07678.
Massimiliano Mancini, José Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2016. Embedding words and senses together via joint knowledgeenhanced training. CoRR, abs/1612.02703.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Burges et al. (Burges et al., 2013), pages 3111–3119.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. ACL Association for Computational Linguistics.
Zhigang W ng and Juan-Zi Li. 2016. Text-enhanced representation learning for knowledge graph. In Subbarao Kambhampati, editor, Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1293–1299. IJCAI/AAAI Press.
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478
479
480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499
ACL 2016 Submission ***. Confidential revi w copy. DO NOT DISTRIBUTE.
3.2 Skip-gram model
g(Independence Day, )
P (N (ej)|ej) P (ej |C(mh), tsl ) e1 e2
P (C(wi)|wi) · P (C(mh)|tsl , mh)
(1)
t1Independence Day t2Independence Day t1Memorial Day
g(Independence Day,
Independenc Day (US)) (2)
g(Independence Day) g(July 4th) (3)
3.3 Text model
Lw = TX
t=1
log P (wt+j |wmt , si)P (si|wcontext)
+ TX
t=1
X
cjc,j 6=0 log P (wt+j |wt)
(4)
DX CX P (wt+j |wmt , si)P (si|wmt , wcontext)
3.4 Knowledge model KBX NX
P (eneighbor|ei)
3.5 Joint model AX
P (ej |wmt , si) + P (ej |wco text)
3.6 Training
3.7 Integrating into GBDT for EL
4 Experiment
4.1 Data Preparation
4.2 Baseline Methods
1. directly align words with entity. 2. align mention with entity using single prototype model.
4.3 Parameter Setting 4.4 Qualitative Analysis 4.5 Entity Relatedness 4.6 Word Analogy 4.7 EL evaluatio
5 Relat Work
6 Conclusion
References Antoine Bordes, Nicolas Usunier, Alberto Garcı́a-
Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multirelational data. In Burges et al. (Burges et al., 2013), pages 2787–2795.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Syst ms 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
Hongzhao Huang, Larry Heck, and H ng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. CoRR, abs/1504.07678.
Massimiliano Mancini, José Camacho-Collados, Igna-
cio Iacobacci, and Roberto Navigli. 2016. Embed-
ding words and senses together via joint knowledgeenhanced training. CoRR, abs/1612.02703.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Burges et al. (Burges et al., 2013), pages 3111–3119.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. ACL Association for Computational Linguistics.
Zhigang Wang and Juan-Zi Li. 2016. Text-enhanced representation learning for knowledge graph. In Subbarao Kambhampati, editor, Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1293–1299. IJCAI/AAAI Press.
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489
490
491
492
493
494
495
496
497
498
499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each mention has multiple sense vectors including two kinds of mention senses: entity-centric sense and out-of-KB sense. Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encounter an mention of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the context information to distinguish existing mention senses, or create a new out-of-KB sense. To be concrete, each mention sense has an embedding (sense vector) tsl and a context clust r with center µ(tsl ). The representation f the context is defined as the average of the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) wj. We predict tsl , the sense of entity title tl in the m ntion < tl, C(tl) >, when observed with context C(tl) as the context cluster membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl otherwise (5) where is a hyper-parameter and tmaxl = argmaxtsl sim(µ(tsl ), C(tl)). We adopt an online non-parametric clustering procedure to learn outof-KB mention senses, which means that if the nearest distance of the context vector to sense cluster center is larger than a threshold, we create a new context cluster and a new sense vector that doesn’t belong to any entity-centric senses. The cluster center is the average of all the context vectors belonging to that cluster. For the similarity metric, we use cosine in our experiments.
Here, we extend Skip-gram model to learn word embeddings as well as mention sense embeddings by the following objective to maximize the probability of observing the context words given either a wor wi or a mention se se of entity title tsl :
Lw = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
wi/t s l , , , w, , ej , e (7)
3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words ith entity. 2. align mention with ntity using single prototype model. 4.3 Parameter Setting 4.4 Qualitative Analysis before conducting the experiments on the tasks, we first give qualit tive analysis of words, mentions an entities. firstly, we give th phrase embedding by its ne re t words and e titi s. next, we give quantitativ analysis on s veral tasks. 4.5 Entity Rela ednes 4.6 Word Similar ty 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Relate Work 6 Conclusion References Alfred V Aho and Margaret J Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340. J-I Aoe. 1989. An efficient digital search algorithm by using a double-array structure. IEEE Tra sactions on Software Engineering, 15(9):1066–1077.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th A nual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Su . 2016. Joint representation learning of text and k owledge for knowledge graph completion. CoRR, abs/1611.04125.
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 51 452 453 54 455 456 457 458 459 460 461 462 463 464 465 466
67
468 469
70
471 472 473 474 475 476 477 478 479
80
481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Giv n the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each mention has multiple sense vectors including tw kinds of mention senses: entity-centric sense and out-of-KB sense.
Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encounter an mention of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the context information to disti uish existing mention senses, or create a new out-of-KB sense. To be concrete, each mention sense has an embedding (sense vector) tsl and a context cluster with center µ(tsl ). The representation of the context is defined as the average of the word vectors in the context: C(wi) =
1 |C(wi)| P wj2C(wi) wj.
We predict tsl , the sense of entity title tl in the mention < tl, C(tl) >, when observed with context C(tl) as the context cluster membership. Formally, we have:
tsl =
⇢ ts+1l t max l <
tmaxl otherwise (5)
where is a hyper-parameter and tmaxl = argmaxtsl
sim(µ(tsl ), C(tl)). We adopt an online non-parametric clustering procedure to learn outof-KB mention senses, which means that if the nearest distance of the context vector to sense cluster center is larger than a threshold, we create a ew context cluster and a new sense vector that doesn’t belong to any entity-centric senses. The cluster center is the average of all the context vectors belonging to that cluster. For the similarity metric, we use cosine in our experiments.
Here, we extend Skip-gram model to learn word embeddings as well as mention sense embed ings by the following objective to maximize the probability of observing the context words given either a word wi or mention se s of entity tit e tsl :
L = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
C(·) (7)
3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align mention with entity using single prototype model.
4.3 Parameter Setting 4.4 Qualitative Analysis before conducting the experiments on the tasks, we first give quali ative analysis of words, mentions and entities.
firstly, we give the phrase embedding by its nearest words and entities.
next, we give quantitative analysis on several tasks.
4.5 Entity Relatedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Related Work
6 Conclusion
References Alfred V Aho and Margaret J Corasick. 1975. Effi-
cient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340.
J-I Aoe. 1989. An efficient digital search algorithm by using a double-array structure. IEEE Transactions on Software Engineering, 15(9):1066–1077.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual C nference on Neural Infor ation Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
5
0 1 2 3 4 5 6 07 08 09 0 1 2
3 4 5 6
17 18 19
0 1 2 3 4 5 6
27 28 29
0 1 2 3 4 5 6
37 38 39
0 1 2 3 4 5 6 447 448 449
0 1 2 3 4 5 6 57 58 59 0 1 2
3 4 5 6
67 68 69
0 1 2 3 4 5 6 477
78 79
0 1 2 3 4 5 6
87 88 89
0 1 2 3 4 5 6 497 498 499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the ann tated text corpus, we le rn word and me tion representations simultaneously by using a multi-pr totype embedding model. Particularly, each word has a unique vector, and each mention has multiple se se vectors including two ki ds of mention senses: entity-centric sense a d o t-of-KB sense. Based n the fixed number of entity-centric enses (Section 3.1), e further learn a vary ng
number of out-of-KB senses for each entity title. When encounter an ti of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the cont xt information to disti guish existing mention senses, or create a n w ut-of-KB sense. To be concrete, each men tion sens has an embeddi g (sense vector) tsl and a context cluster ith center µ(tsl ). The representation of the context is d fined as the average of the word vectors in the context: C(wi) =
1 |C(wi)| P wj2C(wi) wj.
We pr dict tsl , the sense of entity title tl in th mention < tl, C(tl) >, when observ d with context C(tl) s the con ext cluster membership. Formally, we have:
tsl =
⇢ ts+1l t max l <
tmaxl otherwise (5)
where is a hyper-parameter and tmaxl = argmaxtl
sim(µ( sl ), C(tl)). We adopt an on ine non-parametric clus ering procedure to lea n outof-K m n ion enses, which m ans tha if the nearest di ta ce f the co text vector to sen e cluster c nter is larger than a thre hold, we creat a new context cluster and a new sense vector that doesn’t b long t any entity-centric senses. The cluster center is the average of all the context vec-
rs belo ing to th t clus er. For the sim larity me ric, we use cosin in our experi ents.
Here, we xte d Skip-gr m model to lear word embe dings as w ll as m ntion s s embeddings by the f llowing objective to maximize the probability of observing the context words given ither a word wi or a mention sense of entity title tsl :
Lw = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
C(·) (7)
3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experi ent 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align mention with entity usi g single prototyp model.
4.3 Parameter Setting 4.4 Qualitative Analysis befor conduct ng experim nts on the tasks, we first give qualitative analysis of words, mentio s and entities.
firstly, we give the phrase embedding by its nearest words and entities.
next, we give quanti ative analysis on several tasks.
4.5 Entity Relatedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Related Work
6 Conclusion
References Alfred V Aho and Margaret J Corasick. 1975. Effi-
cie t string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340.
J-I Aoe. 1989. An efficient digital search algorit m by usi g doub e-array structure. IEEE Transactions on Software Engineering, 15(9):1066–1077.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances i Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
5
400 401 402 03 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
50 451 452 453 454 455 456 457 458 459 60 461 462
63
464 465 466 467 468 469 470 471 472
73
474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2 3 Text Representation Learning Given the annotated text corpus, we learn word and mention represen ations simultaneously by using a multi-prototype embedding model. Particularly, ach word has a unique vector, and each m nti n has multiple sense v ctors including two kinds of mention senses: entity-centric sense and out-of-KB sense.
Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encounter an mention of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the context information to distinguish existing mention senses, or create a new out-of-KB sense. To be concrete, each mention sense has an embedding (sense vector) tsl a d a context cluster with center µ(tsl ). The representation of the co t xt is defined as the average of the word vectors in the context: C(wi) =
1 |C(wi)| P wj2C(wi) wj.
We predict tsl , the sense of entity title tl in the ention < tl, C(tl) >, when observed with context C(tl) as the context cluster membership. Formally, we have:
tsl =
⇢ ts+1l t max l <
tmaxl otherwise (5)
where is a hyper-parameter and tmaxl = argmaxtsl
sim(µ(tsl ), C(tl)). We adopt an onlin non-parametric clustering procedure to learn outof-KB mention senses, which means that if the nearest distance of the context vector to sense cluster cent r is larger than a threshold, we create a new context cluster and a new sense vector that doesn’t belong to any entity-centric senses. The cl ster c nter is the average of all the context vectors belonging to that cluster. For the similarity metric, we use cosine in our experiments.
Here, we extend Skip-gram model to learn word embeddings as well as mention sense embeddings by the following objective to maximize the probability of observing the context words given either a word wi or a mention sense of entity title tsl :
Lw = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
C(·) (7)
.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating int GBDT for EL Experiment
4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity.
2. align ntion with entity using single prototype model.
4.3 Parameter Setting 4.4 Qualitative Analysis before conducting the experiments on the tasks, we first give qualitativ an lysis of words, mentions and entities.
firstly, we give the phrase embedding by its nearest words and entities.
next, we give qua titative analysis on several tasks.
4.5 tity Relatedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 uns pervised 5 Rel ted Work
6 Conclusion
References Alfred V Aho and Margaret J Corasick. 1975. Effi-
cient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340.
J-I Aoe. 1989. An efficient digital search algorithm by using a double-arr y structure. IEEE Transactions on Software Engineering, 15(9):1066–1077.
Christopher J. C. Burges, Léo Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lak Tah e, Nev da, United States.
Xu H n, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
Mention Representation Learning
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each mention has multiple sense vectors including two kinds of mention senses: entity-centric sense and out-of-KB sense. Based on the fixed number of entity-centric senses (Section 3.1), we further learn varying umber of out-of-KB senses for each entity title. When encounter an mentio f entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use th context inf rmation to distinguish existing mention senses, or create a new out-of-KB sense. To be concrete, each mention sense has an embedding (sense vector) tsl and a context cluster with center µ(tsl ). The representation of the context is defined as the average of the word vectors in the context: C(wi) = 1 |C(wi)| P j2C(wi) wj. We pr dict tsl , the sense of entity title tl in the mention < tl, C(tl) >, wh n observed with context C(tl) as the context cluster membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl otherwise (5) where is a hyper-parameter and tmaxl argmaxtsl
sim(µ(tsl ), C(tl)). We ad pt an onl ne non-parametric clustering procedure to learn outof-KB mention senses, which means that if the nearest distance of the context vector to sense cluster center is larger than a threshold, we create a new context cluster an a new s nse vector that doesn’t belong to any entity-centric senses. The cluster center is the average of all the con ext vectors belonging to that cluster. For the similarity metric, we use cosine in our experiments.
Here, we xt nd Skip-gr m model to l arn word embe di gs as well as mention sense emb ddings by he following objective o maximize the probability of observing the cont xt w r s giv eith r a word wi or a mention sense of entity title tsl :
Lw = wi, l2D P (C(wi)|wi) + P (C(tl)|tl, sl )
(6)
wi/t s l , , , w, , ej , e (7)
3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. a ign mention with entity using single protot pe model. 4.3 Parameter Setting 4.4 Qualitative Analysis before cond cting th experiments on the tasks, we first give qualitative analysis of words, mentions and entities. firstly, we give the phrase embedding by its nearest ords and entities. next, we give quantitative analysis on several tasks. 4.5 Entity Relatedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Related Work
6 Conclusion
References Alf ed V Aho and Margaret J Corasick. 1975. Effi-
ci t stri g matching: an id to bibliographic search. Communicatio s of the ACM, 18(6):333–340.
J-I Aoe. 1989. An efficient digital search algorithm by using double- rray structur . IEEE Transa tions on Software Engineering, 15(9):1066–10 7.
Christopher J. C. Burges, Léon B ttou, Zoubin Ghahramani, nd Kilian Q. Weinberge , editors. 2013. A - va c s in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Proc ssing Systems 2013. Proceedings of a meeti g held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representa ion learn ng of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 49 499 ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each mention has multiple sense v ctors including two kinds of mention senses: entity-centric sense and out-of-KB sense. Based on th fixed number of entity-c ntric senses (Section 3.1), we further learn a varying number of out- f-KB senses for each entity title. When encounter an mention of enti y ti le tl, inspired by the idea of word sense di a bigua (WSD) task, we use th context informatio to distinguish existing mention se ses, or create a new out-of-KB sense. To be concrete, each mention sense has an emb dding (sense vector) tsl and a context cluster with center µ(tsl ). The representation of the context is defined as the average of the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) wj. We predict tsl , the sense of entity title tl in the menti n < tl, C(tl) >, whe bserved with context C(tl) as the cont xt clust r membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl otherwise (5) where is a hyper-parameter and tmaxl = argmaxtsl sim(µ(tsl ), C(tl)). W adopt an online non-parametric clustering procedure to learn outof-KB mention senses, which eans t at if the nearest distance of the context vector to sense cluster center is larger tha a thr shold, we create a new context cluster and new sense vector that doesn’t belong to any entity-centric senses. The cluster center is the average of all the context vectors belonging to that cluster. F r the similarity metric, we use cosine in our experiments. Here, we extend Skip-gram model to learn word embeddings as well as mention s nse mbeddings by the following objective to maximize th pr bability of observing the context words given either a word wi or a mention sense of entity title tsl : Lw = X wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl ) (6) C(·) (7) 3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 I tegrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. dir ctly alig wor s wi h entity. 2. align m tion with entity using single prototype model. 4.3 Parameter Setting 4.4 Qualitative Analysis bef re conduc ng th xperiments on the t sks, we first giv qualit ti e analysis of words, - tions and entities. firstly, we give the phrase embedding by its nearest words and entities. next, we give quantitativ analysis on several tasks. 4.5 Entity Relatedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7 2 unsupervi ed 5 Related Work 6 Conclusion References Alfred V Aho and Margaret J Corasick. 1975. Efficient string matching: an a d to biblio raphic search. Communications of the ACM, 18(6):333–340. J-I Aoe. 1989. An efficient digital s arch algorithm by using a double-array structure. IEEE Transactions on Software Engineering, 15(9):1066–1077. Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Wei berger, editors. 2013. Advances in Neural Informatio Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeti g held Decemb r 5-8, 2013, Lake Tahoe, Nevad , United States. Xu Han, Zhiyuan Liu, and Maoso g Sun. 2016. J int e resentati n learning of text and kn wled for knowledge graph completio . CoRR, abs/1611.04125. 5 400 401 402 403 404 405 6 7 8 9 10 11 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 34 5 6 7 8 9 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 6 7 8 9 60 61 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 84 5 6 7 8 9 490 491 492 493 494 495 496 497 498 499 ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. .2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and ach mention has multiple sense vectors including two kinds of mention senses: e tity-ce tric sense and out-of-KB sense. Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encounter an m tion of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the context informatio to disting i h ex sting me tion enses, or create a n w out-of-KB sense. To be concr te, e ch mention s nse has an embedding (sense vec or) tsl and a co t xt cluster with center µ(tsl ). Th repre entation of the context is defi ed s the vera e of the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) wj. We predict tsl , the sense of ent ty title tl i the mention < tl, C(tl) >, when observed with context C(tl) as the context cluster membership. Fo - mally, we have: tsl = ⇢ ts+1l t max l < tmaxl otherw se (5) where is a hyper-parameter and tmaxl = argmaxtsl sim(µ(tsl ), C(tl)). We adopt an online non-parametric clustering procedure to learn outof-KB mention senses, which means that if the nearest distance of the context vector t sense cluster center is larger than a threshold, we create a new context cluster and a new sense vector that d esn’t belong t any entity-centric senses. The cluster ce ter is the ave age of all the context vectors belonging to that cluster. For the similarity metric, we use cosine in our experiments. Here, xtend Skip-gram mod l to l arn word embeddings as well as mention sense mbeddings by th following objectiv to m ximize th proba bility of observing the ontext words given either a word i or a mention sense of entity itl tsl : Lw = X wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, sl ) (6) C(·) (7) 3.2.4 Entity-centric Sense Representation L arning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Exp riment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align mention with entity using single prototype m del. 4.3 Para eter Set ing 4.4 Qualitativ Analysis before conducting the experiments on the tasks, we first give qualitative analysis of words, mentions and entities. firstly, we give the phrase embedding by its nearest words and entities. next, we give quantitative analysis on several tasks. 4.5 Entity R latedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervi ed 5 Relat d Work 6 Con lusion References Alfred V Aho and Margaret J Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340. J-I Aoe. 1989. An efficient digital search algorithm by using a double-array structure. EEE Transactions on Software Engine ring, 15(9):1066–1077. Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United Stat s Xu Han, Zhiyuan Liu, and Maosong Sun. 201 . Joint representation learning of text and knowldg for knowledge graph completion. CoRR, abs/1611.04125.
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426
7 8 9
30 31
432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476
7 8 9
80 81
482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each mention has multiple sense vectors including two kinds of mention ses: ent ty-centric sense and out-of-KB sense. Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encounter an mention of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the cont xt information to distinguish existing mention senses, or create a new out-of-KB sense. To be concrete, each mention sense has an embedding (sense vector) tsl and a context cluster with center µ(tsl ). The representation of th context is defined as the avera e of the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) j. We predict tsl , the s se of entity title tl in the mention < t , C(tl) >, when observed with context C(tl) as the context cluster embership. Formally, we hav :
tsl =
⇢ ts+1l t max l <
tmaxl otherwise (5)
where is a hyper-parameter and tmaxl = argmaxtsl
sim(µ(tsl ), C(tl)). We adopt an online non-parametric clustering procedure to learn outof-KB mention senses, which eans t at if the nearest di tance of he context vector to sense c uster center is larger han a threshold, we c eate a new cont x clu ter and a new s ns vec or that doesn’t belo g t any e tity-centric sense . The cluster center is the averag of all the context vectors belonging to that cluster. For the similarity metric, we use cosine in our experiments.
Here, we extend Skip-gram model to learn word embe d ngs as well as mention sense embeddings by the following objective to maximize the probability of observing the context words given either a ord wi or a mention se se of enti y title tsl :
Lw = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
C(·) (7)
3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align ention with entity using single prototype model. 4 3 Parameter Sett 4.4 Qualitative Analysis before conducting the experiments on the tasks, we first give qualitative analysis of words, mentions and entities. firstly, we give the phrase embedding by its nearest wor s and entities. next, we give quantitative analysis on several tasks.
4.5 Entity Rel tedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Related Work
6 Conclusi n
R fer nces Alfred V Aho and Margaret J Corasick. 1975. Effi-
cient string matching: an a d to bibliogr phic search. Communications of the ACM, 18(6):333–340.
J-I Aoe. 1989. An efficient digit l search algorithm by using a double-array structure. IEEE Transactions on Softw re Engine ring, 15(9):1066–1077.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27t Annual Conference on Neural Information Processing Systems 2013. Proce dings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledg graph completion. CoRR, abs/1611.04125.
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 20 21 422 423 24 425 426 427 28 29 430 431 32 33 434 435
3 3
438 439
40 41
442 443 444
45
446 447 448
49
450 451 452 4 3 454 455 4 6 457 458 459 460 461 462 463 464 465 466 4 7 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each menti n has multiple sense vectors including tw kinds of mention senses: entity-centric sense and o t-of-KB sense. Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encoun er an mention of entity title tl, inspir d by the idea of word sense disambiguation (WSD) task, we use the context information to distinguish existing mention senses, or create a new out-of-KB se se. To be concrete, each mention sense has an embedding (sense vector) tsl an a context cluster with center µ(tsl ). The epresentation of the context is defined as the average f the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) wj. We predict tsl , the sense of entity title tl in the mention < tl, C(tl) >, when observed with context C(tl) as e c ntext cluster membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl oth rwise (5) where is a hyp r-parameter and tmaxl = argmaxtsl sim(µ(tsl ), C(tl)). We dopt an online non-parame ric cl s ring procedure to learn outof-KB mention senses, which me ns that if the near st ista ce of the context vector to sense cluster cent is larger than a threshold, we create a new context cluster and a new sense vector that do sn’t bel ng to any entity-centric senses. Th clust r cent r is the average of all h context vectors belonging to that cluster. For the s milarity m tric, we use cosine in our experiments. Here, w extend Skip-gram model to learn word e beddings as well as mention se se embe dings by the f llo ing objective to maximize th probability of observing the context words given either a word wi or a mentio sen e of entity title tsl : Lw = X w ,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl ) (6) N (·) (7) 3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Join ly Training 3.3 Integrating into GBDT for EL 4 Experimen 4.1 Data Preparation 4.2 Baselin Methods 1. directly align words with entity. 2. align mention with entity using single prototype model. 4.3 Parameter Setti g 4.4 Qualitative Analysis before conducting the experim nts on the tasks, w first give qualitative analysis of words, m ntio s and entities. firstly, we give the phrase embedding by its nearest words and entities. next, we give quantitative analysis on several tasks. 4.5 Entity Relatedness 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Related Work 6 Conclusion References Alfred V Aho and Margaret J Corasick. 1975. Efficie t string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333–340. J-I Aoe. 1989. An efficie t digital search algorithm by using a doubl -array structure. IEEE Transactions on Software Engineering, 15(9):1066–1077. Christopher J. C. Bur es, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Wei berger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States. Xu H n, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledg for knowledge graph completion. CoRR, abs/1611.04125.
5
0 1 2 3 4 5 6 07 08 09 0 1 2 3 4 5 6 17 418 19 0 1 2 3 4 5 6 27 28 29 0 1 2
3
6
37 38 39
0 1 2 3 4 5 6 447 448 449
0 1 2 3 4 5 6 57 58 59 0 1 2 3 4 5 6 67 68 69 0 1 2 3 4 5 6 77 78 79 0 1 2
3 4 5 6
87 88 89
0 1 2 3 4 5 6 497 498 499
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the ann tated text cor us, we le rn word a d me tion representations simultaneously by using a multi-pr totype embedding model. Particularly, each word has a unique vector, and ea h mention has multiple se se vectors i cludi g two ki ds of mention senses: entity-centric sense a d o t-of-KB sense. Based n the fixed numb r of ntity-ce tric senses (Section 3.1), e further learn a varying number of out-of-KB senses for each entity title. Whe encount r an ti of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the cont xt information to disti guish existing mention senses, or cre te a n w out-of-KB sense. To be c crete, each mention sens has an embeddi g (sense vector) tsl and a context cluster ith center µ(tsl ). The representation of the context is d fined as the average of the word vectors in the context: C(wi) = 1 |C( i)| P wj2C(wi) wj. We pr dict sl , th se of n ity title tl in the m t on < tl, C(tl) >, when observed w th context C(tl) as the c ntext cluster membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl oth rwise (5) where is a hyp r-paramet r and tmaxl = argmaxtsl
sim(µ(tsl ), C(tl)). We adopt an onli e non-parametric clustering procedure to learn outof-KB m ntion senses, which m ans that if the nearest dista ce f the co text vector to sense cluster c nt r is larger than a threshold, we creat new context l t r and a n w sense vector t at doesn’t b long t any entity-centric senses. The clust r center is the average of all th context vectors belo ing to th t clus er. F the similarit me ric, we use cosin in our experi ents.
Here, we xte d Skip-gram model to learn word e be dings as w ll as m tion s se embeddings by the f llo ing objectiv to maximize th probability of observing the context words given ither a word wi or a mention sen e of entity title tsl :
Lw = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
N (·) (7)
3.2.4 Entity-centric Se se Representation Learni g Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experiment 4.1 Data Preparation 4.2 B seli e M thods 1. directly align words with entity. 2. align mention with entity using single prototype model. 4.3 Parameter Setting 4.4 Qualitative Analysis before conducting t experim nts on the tasks, w first give qualitative analysis of words, mentio s and entities. firstly, we give the phrase embedding by its nearest words and entities. next, we give quantitative analysis on several tasks. 4.5 Entity Relatedness 4.6 Word Similarity 4.7 EL valu tion 4.7.1 gbdt 4.7.2 unsupervised 5 Relate Work 6 Conclusion References Alfred V Aho and Margaret J Corasick. 1975. Effi-
cie t string matching: a aid to bibliographic search. Communications of th ACM, 18(6):333–340.
J-I Aoe. 1989. An efficient digital search algorit m by usi g doub e-array structure. IEEE Transactions on Software Engineering, 15(9):1066–1077.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors. 2013. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceed ngs of a me ting held December 5-8, 2013, Lake Tahoe, Nevada, United St tes.
Xu H n, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation l arning of text and knowledge for knowledge graph completion. CoRR, abs/1611.04125.
5
0 1 2 3 4 5 6 07 08 09 0 1 2 3 4 5 6 17 18 19 0 1 2 4 3 4 5 6 427 28 29
0 1 2 3
4 5
6
37 38 39
0 1 2 3 4 5 6
47
448 449
0 1 2 3 4 5 6 57 8 59 0 1 2 3 4 5 6 67 68 9 0 1 2 3 4 5 6 77 78 79
0 1 2 3 4 5 6
87 88 89
0 1 2 3 4 5 6 497 498 499
ACL 2016 Submissi n ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 T xt Representatio Lear ing Give the ann tated text corpus, we le rn word and e ti re esentations simultaneously by using a multi-pr totype embedding mod l. Particularly, each word has a unique vector, and each mention has multiple se se vectors includ g wo ki ds of mention senses: entity-centric sense a d o t-of-KB se se. B s d n he fixed number of en ity-c ntric ens s (Section 3.1), e further learn a vary ng number of out-of-KB senses for each entity title. When encounter an ti of e tity title tl, inspired by the idea of word sense dis mb uati n (WSD) task, we use th cont xt information to disti guish existing mention senses, or create a n w out-of-KB sense. To be co crete, each men tio se s h s a embeddi g (sense ector) tsl and a context cluster ith center µ(tsl ). The represe tation of the cont xt is defin d as th average of the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) wj. W pr dict tsl , th se of entity title tl in the mention < tl, C(tl) >, hen obser ed with context C(tl) as the context cluster membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl otherwise (5) here is a hyper-parameter and tmaxl = arg axtl
sim(µ( sl ), C(tl)). We adopt an on i non-param tric clus ring procedur to lea n utof-KB m n ion enses, which m ans tha if the nearest dista c of the co text vector to s n e clust r c nter is l rger than a threshold, we creat a new context clus er and a new sense vector that doesn’t b long t any entity-centric se ses. The cluster center is the average of al th context vectors belo ing to th t clus er. For the similarity m ric, we use cosin in ou experi ents.
Her e xte d Skip-gram m del to learn word e be dings as w ll as m tion s se embeddings by the following objective to maximize the robability of observing the context words give either a word wi or mention sense of ntity title tsl :
Lw = X
wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl )
(6)
N (·) (7)
3.2.4 Entity-centric Sense Representation L ar ing Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Int grating i to GBDT for EL 4 Experiment 1 D t Prepara ion 2 Baseline Methods 1. di ctly align words with entity. 2. ign mentio with e tit u ing ingle oto ype mo l. 4.3 Parameter Setting 4.4 Qualitative Analysis before conduct ng experim nts on the tasks, we first give qualitative analysis of w rds, mentio s and entities. firstly, we giv the phrase embedding by its nearest words and entities. next, we give quantitative analysis on several tasks. 4.5 Entity Relatedne s 4.6 Word Similarity 4.7 EL evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 Related Work
6 Conclusio
References Alfred V Aho a d Margar t J Corasi k. 1975. Effi
cie t string match ng: an id to bibliog phic earch. Communications of the ACM, 18(6):3 3–340.
J-I A e. 1989. An efficient digital search algorit by usi g doub e-array structure. IEEE Transactions on Software Engineering, 15(9):1066–1077.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahrama i, and Kilian Q. Weinber er, editors. 2013. Adva ces in Neur l Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. J int representation learning of text and knowledge for knowledge g aph completion. CoRR, abs/1611.04125.
played it during public events, such as [[ ]] celebrations Mention Sense Mapping
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. 3.2.3 Text Representation Learning Given the annotated text corpus, we learn word and mention representations simultaneously by using a multi-prototype embedding model. Particularly, each word has a unique vector, and each mention has multiple sense vectors including two kinds of mention senses: entity-c n ric sense and out-of-KB sense. Based on the fixed number of entity-centric senses (Section 3.1), we further learn a varying number of out-of-KB senses for each entity title. When encounter an mention of entity title tl, inspired by the idea of word sense disambiguation (WSD) task, we use the context information to distinguish existing ment on senses, or create new out-of-KB sense. To be concrete, each mention sense has an embedding (sense vector) tsl and a context cluster with center µ(tsl ). The representation of the context is defi ed as the average of the word vectors in the context: C(wi) = 1 |C(wi)| P wj2C(wi) wj. We predict tsl , the sense of entity title tl i he mention < tl, C(tl) >, when observed with context C(tl) as the context cluster membership. Formally, we have: tsl = ⇢ ts+1l t max l < tmaxl otherwise (5) where is a hyper-parameter an tmaxl = argmaxtsl sim(µ(tsl ), C(tl)). We adopt an online non-parametric clustering procedure to learn outof-KB mention senses, which means that if the nearest distance of the context vector to sense cluster center is larger than a threshold, we create a new context cluster and a new sense vector that doesn’t belong to any entity-centric senses. The cluster center is the average of all the context vectors belo ging to that cluster. For the similarity etric, we use cosine in our experiments. Here, we extend Skip-gram model to learn word embeddings as well as mention sense embeddings by the following objective to maximize the probability of observing the context words given either a word wi or a mention s nse of e tity title tsl : Lw = X wi,tl2D P (C(wi)|wi) + P (C(tl)|tl, tsl ) (6) g(July 4th, e1) (7) 3.2.4 Entity-centric Sense Representation Learning Lm = X (mh,ej)2A P (ej |C(mh), tsl ) (8) 3.2.5 Jointly Training 3.3 Integrating into GBDT for EL 4 Experi ent 4.1 Data Preparation 4.2 Baseline Methods 1. directly align words with entity. 2. align mention with entity sing sin le pro otype model. 4.3 Parameter Setting 4.4 Qualitative Analysis before conducting the xperiments on the tasks, w first give qualitative a alysis of word , m - tions and entities. firstly, we give the phrase emb dding by it nearest words and ent ties. next, e give quantitative analysis on several tasks. 4.5 Entity Relatedness 4.6 Word Simil rity 4.7 L evaluation 4.7.1 gbdt 4.7.2 unsupervised 5 R lated Work 6 C clusion R ferenc s Alfred V Aho a d Margar t J Corasick. 1975. Efficient string matching: an aid to bibliogr phic search. Communications of the ACM, 18(6):333–340. J-I Aoe. 1989. An efficient digita search algorithm by using a double-array structure. IEEE Transacti s on Software Eng neering, 15(9):1066–1077. Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editor . 2013. Advances in Neural Informati n Processing Syste s 26: 27th Annual Conference on Neural Informat o Processing Systems 2013. Proceedings of a meeti g held December 5-8, 2013, Lake Tahoe, Nevada, United States. Xu Han, Zhiyuan Liu, and Maoso Sun. 2016. Joint representation learni g of text d knowledge for knowledge graph completion. C RR, abs/1611.04125.
3
200 201 202 203 204 205 206 207 208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250 251 252 253 254 255 256 257 258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ACL 2016 Submissio ***. Confidential review copy. DO NOT DISTRIBUTE. as w ll as w rd embeddi gs w nd entit emb ddings e. Note that slj 2 m⇤l denotes that mention s nse f ml refer to entity ej , where m⇤l represents he sen e set of ml. Different mentions may share th same ention sense, deno ed as s⇤j . Example As shown i Figure 1, there ar two differe t mentio s “Indep ndence Day” m1 and “July 4th” m2 in t e docu ents. MPM is to learn two ment on senses s11, s 1 2 f r m1, and one me t on s 22 for 2. Clear y, th se t o mentions share a common s se n th last two d cum nts: the United States holid y e2, so w ve s⇤2 = 1 2 = s 2 2. Note that w,m, s ar atur lly embedded into the s m semantic space since they re basic units in texts, and e modeling the graph tructure in KB is actually in another semantic space.
3 Method
In this s cti n, w firs ly describe he framework of MPME, followed by the detail d inf m tion of ach key comp e t. Th n, we introduce a well designe me tion s se disambiguati n method, which c also be used for entity linking n unsupervis d way.
National Day
s⇤I dependence Day (film), s ⇤ Independence Day (US)
3.1 Framework Given KB, D and A, we are to jointly learn word, entity and mention representations: w, e, m. Serving as basic units in texts, W rd {wi} and entity title {tl} are naturally embedded into a unified semantic space, meanwhile entities {ej} are mapped to one of mention senses of its title: sl . Thus, text an knowledge are com-
bined via the bridge of mentions. We can eas-
ily obtain the similarity between word and entity Similarity(wi, ej) by computing the similarity between word and its corresponding mention sense: Similarity(wi, f(ej)).
As shown in Figure 2, our proposed MPME contains four key components: (1) Mention Sense Mapping: we map the anchor < mh, ej > A to the corresponding mention sense tsl to reduce the vocabulary to learn. (2) Entity Representation L ar ing given a knowledge base KB, we construct a knowledge network among entities, and
learn their embeddings so that similar entities on the graph have similar representations. (3) Text Representation Learning given text corpus D as well as the annotated anchors, we learn word and entity title embeddings by maximizing the probability f co-occurri w rds/entity titles so that si ilar words/entity titles have similar representations. (4) Mention Representation Learning given ann tated anchors tsl :< mh, ej >2 A, we learn entity title embeddings by incorporatin both contextual words embeddings and entity embeddings in order to distinguish different mention e s that has similar rep ese tations to its corresponding entity embeddings. R p esentation learning of (2), (3) and (4) ses an iterative update procedure following a unified optimization objective. The outputs of word embeddings wi and entity embeddings ej keep their own semantic space and are naturally bridged via the new learned entity title embeddings tl, which inspires u to globally optimize the robability of choosing mention s nses of all the phrases of mention names in the given document. Since each mention sense corresponds to an entity, the mentio se se disambiguation process can also be reg rded as inking entities to knowledge base in a unsupervised way, which will be detailed in Section ??.
3.2 Mention Sense Mapping
There are two kinds of mappings: from entities to mention senses, and from mention names to mention sen es. The former is pre-defined at the very beginning. Given the knowledge Base KB, we extract entity titles {tl} and initialize with multiple mention senses, where the sense number depends on how many entities share a common title. The latter is to find possible mention senses for the given mention name, which is similar to candidate mention generation in entity linking task.
Conventional candidate mention generation generally maintains a list of pairs of mention name and entity that denotes a candidate reference in knowledge base for the mention name, and recognizes the mention name in text by accurate string matching. Or it firstly recognizes possible mention names in texts using NER (Named Entity Recognition) tool, and then approximately retrieves candidate entities via an information retrieval method.
Since this c mponent is not key point in this paper, we adopt the first method to collect a3
200 201 202 203 204 205 206 207 208 209 210 211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
2 1
242
243
244
245
2 6
247
248
249
250 251 252 253 254 255 256 257 258 259 260 261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE. as w ll as word embeddings w and entity embeddings e. Note that slj 2 m⇤l denotes that mention sens of ml r fers t entity j , where m⇤l represents the sense set of ml. Different mentions may share he sam mentio sens , den ted as s⇤j . Example As s own in Figure 1, there are two differ nt me tions “Independence Day” m1 and “July 4th” 2 in the documents. MPME is to learn two mentio senses s11, s 1 2 for m1, and one
mention sens s22 f r m2. Clearly, these two mentions share common s se in the last two documents: the Unit d States holiday e2, so we have s⇤2 = s 1 2 = s 2 2. N te that w,m, s are naturally embedde into the same semantic space since they are basic units n texts, and e modeling the gr ph structur in KB is actu lly in another semantic space.
3 M t od
I his s ction, e firstly describe the framework of MPME, followed by the d tailed information of each key component. Then, we introduce a well d sign d mentio sense disambiguation method, which can also be used for entity linking in a unsupervis d way.
eNational Day
s⇤Independence Day (f lm), s ⇤ Indepe dence Day (US)
3.1 Framework Given KB, D and A, we are to joi tly lear word, entity and me tion representations: w, e, m. Serving as basic units i texts, Word {wi} and entity title {tl} are naturally e bedded into a unified semantic space, meanwhile entities {ej} are mapped to one of mention senses of its title: tsl . Thus, text and knowledge are combined via the bridge of mentions. We can easily obtain the similarity between word and entity Similarity(wi, ej) by computing the similarity between word and its corresponding mention
sense: Similarity(wi, f(ej)).
As shown in Figure 2, our proposed MPME contains four key components: (1) Mention Sense Mapping: we map the anchor < mh, ej >2 A to the corresponding mention sense tsl to reduce the vocabulary to learn. (2) Entity Representation Learning give a knowledge base KB, we construct a knowledge network among entities, and
learn their embeddings so that similar entities on the graph have similar representations. (3) Text Representation Learning given text corpus D as well as the annotated anchors, we learn word and entity itle e eddings by maximizing the probability of co-occurring words/entity titles so that similar words/entity titles have similar representations. (4) Mention Representation Learning given annotated anchors tsl :< mh, ej >2 A, we learn entity title embeddings by incorporating both contextual words embeddings and entity emb ddings in order to distinguish different mention senses that has similar representations to its corresponding entity embeddings.
Representation learn ng of (2), (3) and (4) uses an iterative update procedure following a unified optimization objective. Th outputs of word embeddi gs wi and entity mbeddings ej keep their own semantic space and are naturally bridged via th new learned entity title mbeddings tl, which inspires us to globally optimize the probability of choosing mention senses of all the phrases of mention names in the give document. Since each
enti n sense corresponds to an entity, the mention sense disambiguation process can also be regarded as linking entities to knowledge base in a unsupervised way, which will be detailed in Section ??.
3.2 Mention Sense Mapping
There are two kinds of mappings: from entities to mention senses, and from mention names to mention senses. The former is pre-defined at the very beginning. Given the knowledge Base KB, we extract entity titles {tl} and initialize with multiple mention senses, where the sense number depends on how many entities share a common title. The latter is to find possible mention senses for the given mention name, which is similar to candidate mention generation in entity linking task.
Conventional candidate mention generation generally maintains a list of pairs of mention name and entity that denotes a candidate reference in knowledge base for the mention name, and recognizes the mention name in text by accurate string matching. Or it firstly recognizes possible mention names in texts using NER (Named Entity Recognition) tool, and then approximately retrieves candidate entities via an information retrieval method.
Since this component is not key point in this paper, we adopt the first method to collect a
outlink
Observed by
category
3
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE.
KB, a xt corpu D a a set of a hors A, multiprototype e tion embedding is to learn multiple sense emb ddings sjl 2 Rk for a mention ml as well as word embeddings w and entity embeddings e. Note that slj 2 ⇤l enote that en ion s nse f ml r fers e tity ej , where m⇤l repr - sents the sense s t of ml. iff rent mentions may share the same m ntion sense, denoted as s⇤j . Example As shown in Figure 1, there are wo different mentions “Independence Day” m1 and “July 4th” m2 in the ocum ts. MPME is to learn two mention s ns s s11, s 1 2 r m1, and one mention sense s22 for m2. Clearly, these two mentions share a common sense n the last two documents: th United States holiday e2, so we have s⇤2 = s 1 2 = s 2 2. Note that w,m, s are naturally embedd d into t e sam semantic space since they are basic units in t xts, and modeling the graph structure in KB is actually in another semantic space.
3 Method
In this section, we firstly descri e the framew rk of MPME, followed by the det iled i f rmation f
each key component. Then, we introduce a well
designed mention sense disambiguation method, which can also be used for entity linking in a unsupervised way.
3.1 Framework
Giv n kn wledge bas KB, t xt corpus D and a set of anchors A, we are to jointly learn word, entity and mention representations: w, e, m. As shown in Figure 2, our proposed MPME contains four key components: (1) Mention Sense Mapping: given an anchor < ml, ej >, we map it to the corresponding mention sense to r duce the mention vocabulary t le embeddings. If only a mention is given, we map it to several m ntion senses that requires disambiguation (Section 3.4). (2) Entity Representation Learning based on outlinks in Wikipedia pages, we construct a knowledge network to represent the semantic relatedness among entities. And then learn entity embed ings so that similar entities on the graph have similar representations. (3) Mention Representation Learning given mapped anchors in contexts, we learn mention sense embeddings by incorporating both textual context embeddings and entity embeddings. (4) Text Representation Learning we extend skip-gram model to simultaneously learn
word and m tion ense emb ddings on annotated text corpu D0. Following (Y mad et al., 2016), we use wikipedia articles as text corpus, and the anchors provide annotated mentions1.
W jointly trai (2), (3) a d (4) by using a unified opti ization obj ctive. The outputs emb - di gs of word a d mention ar naturally in the sam seman ic sp ce since they are different units in annotated tex corpus D0 for text representation learning. Entity embeddings keep their own semantics in another vector space, be ause we only us them s answers to predi t in mention representation learning by extending Continuous BOW model, which w ll be further discussed in Section ??.
s⇤Memorial Day
word embed ings wi and entity mbeddings ej keep their own s mantic sp ce nd are naturally bridged via the new learned e tity title embeddings tl, which inspires us t globally optimize the probability of c oosing mention s nses f all the phrases of mention names in the given document. Since each mention se se corresponds to an
entity, the mention sense disambiguation process
can also be regarded as linking entities to knowledge base in a unsupervised way, which will be detailed in Section ??.
3.2 Mention Sense M pping
There are two kinds of mappings: from entities to mention senses, and from mention names to mention senses. The former is pre-defined at the very beginning. Given the knowledge Base KB, we extract entity titles {tl} and i itialize with multiple mention senses, where the sense number depends on how many entities share a common title. The latter is to find possible mention senses f r the given mention name, which is similar to candidate mention generation in entity linking task.
Conventional candidate mention generation generally maintains a list of pairs of mention name and entity that denotes a candidate reference in knowledge base for the mention name, and recognizes the mention name in text by accurate string
1We can also annotate text corpus by using NER tool like python nltk to recognize mentions, and disambiguating its mapped mention senses as described in Section 3.4. This is an ongoing work with the goal of learning additional out-ofKB senses by self-training. In this paper, we will focus on the effectiveness of our model and the quality of three kinds of learned embeddings.
3
200 201 202 203 204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250 251 252 253 254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
ACL 2016 Submission ***. Confidential review copy. DO NOT DISTRIBUTE.
KB, a text orpus D a d set of a chors A, mult prototype mention em edd g is to lear m ltiple sens embeddings sjl 2 Rk for each ntio ml as well a word embeddings w a d entity embeddings e. Note hat slj 2 m⇤l d o es that men ion sens f ml efers to en ity ej , where ⇤ repres nts the se s t of ml. Differ nt e ti s may share the same m ntion se s , denoted as s⇤j . Example As shown i Figur 1, ther ar two differe t mentions “Indepe dence Day” m1 and “July 4th” m2 in the do ument . MPME s o learn two mention senses s11, s 1 2 for 1, nd n m ntion s ns 22 f r m2. Clearly, th e t o m ntions share a common sense in the last two documen s: the United St es holiday e2, so we ave s⇤2 = s 1 2 = s 2 2. Not that w, , s are naturally embedded into the sa e semantic space si ce they are basic u its i texts, and e modeling the graph structure in KB is actual y in another semantic space.
3 Method
In this section, we firstly describe the framework of MPME, followed by th detailed inf mation of each key componen . Then, we introduce a well design d mention sense disambiguation method, which can also b use for entity linki g in a unsupervis d way.
3.1 Framework
Given knowledge base KB, text corpus D and set of nchors A, we are to jointly learn word, entity and mention representations: w, e, m. As shown in Figure 2, our proposed MPME contains four key components: (1) Mention Sense Mapping: given an anchor < ml, ej >, we map it to the corresponding mention sense to reduce the mention vocabulary to lear mbeddings. If only a en-
tion is given, we map it to several mention s ses
that requires disa biguation (Section 3.4). (2) Entity R presentation Learning based on outlinks in Wikipedia pages, we construct a knowledge network to represe t the s mantic relatedness among entities. And then learn entity embeddings so that similar entities on the graph have similar representations. (3) Mention Representation Learning given apped anc ors in contexts, we learn mention sense embeddings y incorporating both textual cont xt emb ddings a d entity embeddings. (4) Text epresentation Learning we extend skip-gram model to simultaneously learn
word and mention ense embeddings on annotated text corpus D0. Following (Yamada e al., 201 ), we use w kipedia articles as text corpus, and the anc ors p ovide annotated mentions1.
W joint y train (2), (3) and (4) by using a u ified optimization objective. The outputs embeddings of word and mention are naturally in the same semantic space since they are different units in an ot ted text corpus D0 f r text representation learning. E tity embeddings keep heir own se-
a tics i noth r ctor p ce, because we only use hem as answers to predict in mention represent tio l ni g by xtending Co tinuous BOW model, which will be further discussed in Section 3.3.4.
Figur 2 shows a real example of “” eMemorial Day word embeddings wi and entity embeddings ej keep their own sema tic space and are naturally bridged via the new learned entity title embeddings tl, which inspires us to globally optimize the probability o hoos ng menti n senses of all the phrases of mention names in the given document. Since e ch mention sense corresponds to an entity, the mention sense disambiguation process can also be reg rde s l king entities t knowledge base in a unsupervised way, which will be detailed in Section ??.
3.2 M ntion Se se Mapping There ar two kinds of mappi s: from ntities to mention senses, and from ention names to mention senses The former is pre-defined at t e very begin ing. Given the knowledge Base KB, we extract entity titles {tl} and initialize with multiple mention senses, wh re the sense number depends on how many ntities hare a common title. The latter is to find possible mention sens s f r the
given mention name, which is si ilar to candidate
mention generation in entity linking task.
Conventional candidate mention generation gener lly maintain a list of pairs of mention name and entity that denotes a candidate reference in knowledge base for the mentio name, and recog-
izes the m tion ame i text by accurat s ring m tchi . Or it firstly recognizes possible mention
1We can also annotate text corpus by using NER tool like python nltk to recognize mentions, and disambiguating its mapped mention senses as described in Section 3.4. This is an ongoing work with th goal of learning additional out-ofKB senses by self-training. In this paper, we will focus on the effectiveness of our model and the quality of three kinds of learned embeddings.
… holds annual [[Independence Day (US)| Ind pendence Day]] celebrations and other
festivals …
… early Confederate [[Memorial Day]] celebrations were simple, somber occasions for veterans and their families to honor the dead …
5
400 401 402 403 404 405 406 407 408 409 41 411 412 413 414 415 416 417 418 419 420 421 422 4 3 424 425 426 427 428 4 9 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 ACL 2017 ubmission ***. Confident al Review Copy. DO NOT DISTRIBUTE. predict the context word by maximizing th f llowing objective function: Lw = X wi,ml2D0 log P (C(wi)|wi) + log P (C(ml)|s⇤j ) (6) whe e s⇤j = g(< ml, ej > i obtain d from anchors in wikipedi rticles. Thus, similar words and m tio senses will be closed in text space, such as wfilm and s⇤Independence Day (film), or wc lebratio s and s⇤I dependence Day (US) b ca e y fr quently occur i the same contexts. Simila to WDS, we maintain a context cluster for each mention sense, which can be used for disa biguation given th conte s (S ction 5). r xample, i d1 f Figure 2, h context cluster of s⇤ consists of all context vectors W n ncou tering ention, th c nt xt vector we also maintain a context l ster center µ⇤j for each mention sense s⇤j , which is computed by aver ging all the ontext vectors belonging to the cluster. We define context vecto s the average sum f cont xt word embeddings 1 |C(wi)| P wj2C(wi) wj. T e cluster enter is helpful for inducing mentio sense n co texts. When encou ter a menti n, we ap it t a set of mentio senses, and then find th ear st on according to the distance from its context vector to each mention sense cluster cen er, which will be discussed in Section 5. d1, d , d3, s ⇤ j , i/s ⇤ j s⇤Independence Day (US) P (ej |C(ml), s⇤j ) P (C(wi)|wi) · P (C( l)|s⇤j ) (7) 4.5 Joint Training Considering all the above representation lear ing components, we define the overall objective function as linear combinations: L = Lw + Le + Lm (8) The training of MPME is to maximize the above function, and iteratively update thre ypes of embeddings. Also, we use negative sampli g technique for efficiency (Mikolov et al., 2013a). 5 Mention Sense Disambiguation MPME learns each mention with multiple sense e beddings, and each sense corr sponds t a context cluster. Given an nnotated document D0 including M mentions, and their sense sets accordng to Section ??: M⇤l = {slj |slj 2 g(ml), ml 2 M}. In this section, we describe how to determine the mention sense for e ch m ntion ml in the document. Based on lang age model, identifying m ntion senses in a document c n be r garded as maximizing their joint probability. However, the global optimum is xp nsive, in which each mention g ts n optimum sense, to search over the space of all men ion se ses of all menti ns in the document. Thus, we appr ximately assi n e ch mention independently: P (D0, . . . , sj , . . . , ) ⇡ Y P (D0|slj) · P ( lj) ⇡ Y P (C(ml)|slj) · P (N̂ (ml)|slj) · P (slj) (9) where P (C(ml)|slj) is prop rtion l to cosine similarity between context v ctor a d mention sense cluster center µlj to measure the ention’s local similari y, namely local prob bility. N̂ (ml) de ot s neighbor mentions of ml cooccurring in a piece f text (e. . a document), and P (N̂ (ml)|slj) is defi ed as global pr ability since it measures global coherence of neighbor m ntions. The underlying idea is to achieve consistent semantics in a pi ce of text assu ing that all entions inside it are talking about the same topic. In this paper, w regard he mention senses identified first as neighbors of the rest mentions. P (slj) denotes prior probability of a mention sense occur ing in texts proportional to the frequency of corresp ding entity in Wikipedia anchors: P (slj) = ( |Aej | |A| ) 2 [0, 1] where is a hyper-parameter to smooth the gaps between different entity frequencies, namely smoothing parameter. It controls the importance5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 41 415 416 417 418 419 420 421 422 423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2017 Submission ** . onfidential Review Copy. DO NOT DISTRIBUTE. predict the context words by maximizing the fo - low ng o jective functi n: Lw = X wi,ml2D0 log P (C(wi)|wi) + log P (C(ml)|s⇤j ) (6) where s⇤j = g(< ml, ej >) is obtained from anchors in wikip dia articles. Thu , similar words and mention senses w l be clos d n text pace, such as wfilm and ⇤Independence Day (film), or wcel brations and s⇤Independence Day (US) b ause hey frequ ntly occur in the same contexts. Similar o WDS, we intain a ont xt clust r for ac mention sense, which can be used for di - ambiguation given t e contexts (Section 5). For exampl , in d1 of Figure 2, the co text clust r of s⇤ consists of al context vectors When e co ering a ention, th context v cto
we also maintain a context clu c nt r µ⇤j for each m ntion sen e s⇤j , ich i computed by ave aging all the context vectors bel ging t the clust r. We defi e co xt ector s the average sum of context word embed ings
1 |C wi)| P wj2C(wi) wj. Th clu t r cent r s elpful f i duci g menti n sense i c nt xts. W en encoun er a mentio , we m p it o a se of e tion enses, a d then find the n arest one accordin to the distance from its con ext vect r t ach mentio s nse cluster center, hich will be discussed i Section 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j
s⇤Indep ndence Day (US)
P (ej |C(ml), s⇤j )
P (C(wi)| i) · P (C(ml)|s⇤j ) (7)
4.5 Joint Traini g
Considering al th above r pr sentation lear ing compon nts, we d fine the overall objective function as linear combinations:
L = Lw + Le + Lm (8)
The training of MPME is to maximize the above fu ctio , nd iterativ ly updat three types of embedding . Also, w use negative sampling technique for effici ncy (M k lov et al., 2013a). 5 Men ion Sense Disambiguation MPME learns each mention with multiple sense embeddings, and each sense corresponds to a context cluster. Given an annotated document D0 including M mentions, and their sense sets according to Section ??: M⇤l = {slj |slj 2 g(ml), ml 2 M}. In this section, e describe how to determine the mention sense for each m tion ml in the docum nt. Based o language model, identifying mention enses in document can be regarded as maximizing their joint probab lity. However, the global opimum s xpe sive, in which ach mention gets an optimu sense, to search over the space of ll ent o sen es of all mentions in the document. Thus, we approximately assign ach mention independently:
P (D0, . . . , slj , . . . , ) ⇡ Y
P (D0| lj) · P (slj)
⇡ Y P (C(ml)| lj) · P (N̂ (ml)|slj) · P (slj) (9)
where P (C(ml)|slj) is p oportional to cosine similarity between context vector and mention sense cluster center µlj to m asur the mention’s local imilarity, namely loc probabil ty. N̂ (ml) denotes neighbor mentions of ml co-
ccurring i a piece of text (e.g. a document), and P (N̂ ( l)|slj) is defined as global probability sinc i measures global coherence of neighbor mentions. The underlying idea is t achieve c nsist nt se antics in a piece of text assumi g that all ment ons inside it are talk ng about the same topic. In this paper, we regard the mention senses id ntified first as neighbors of the rest mentions.
P (slj) denot s prior probability of a mention sense occurring n texts p oport onal to the frequency of c rresponding entity in Wikipedia anchors:
P (slj) = ( |Aej | |A| )
2 [0, 1]
where is a hyper-parameter t smooth the gaps between different entity frequencies, namely smoothing parameter. It co trols the importance
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 48 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 ACL 2017 Submission ***. C nfidential Revie Copy. DO NOT DISTRIBUTE. predict the context words by m ximizing the following objective function: Lw = X wi,ml2D0 log P (C(wi)|wi) + log P (C(ml)|s⇤j ) ( ) where s⇤j = g(< ml, ej >) is obtained from anchors in wikipedia articles. Thus, similar w rds and me tion senses will be closed in text space, such as film and s⇤Independence Day (film), or wcelebrations and s⇤Independence Day (US) because hey frequently occur in the same contexts. Similar to WDS, we aintain a contex clu t r for each mention sense, which can be used for disambiguation given the contexts (Section 5). For example, in d1 of Figure , the context clu ter of s⇤ consists of all context vectors Wh n e cou t ring a mention, the context vect r we also maintain a context cluster center µ⇤j for each mention sense s⇤j , which is computed by averaging all the context ve tors b longi to the cluster. We define context vector s the average sum of context word mbeddings 1 |C(wi)| P wj2C(wi) j. The cluster cent r is helpful for inducing mention sense in contexts. Wh n encounter a mention, we map it to a s t of menti n senses, and then find the near st one ccording to the distance from its context vector to each m nti n sense cluster center, which will be discussed in Sect on 5. d1, d2, d3, s ⇤ j , wi/s ⇤ j s⇤Independence Day (US) P (ej |C(ml), s⇤j ) P (C(wi)|wi) · P (C(ml)|s⇤j ) (7) 4.5 Joint Training Considering all the above rep esentation learning components, we define the overall objective function as linear combinations: L = Lw + Le + Lm (8) The trai i g of MPME is to maximize the above function, nd iteratively update three types of mbeddings. Also, we use negative s pli g t chnique for efficiency (Mikolov t al., 2013a). 5 Mention S nse Disambiguation MPME learns each me tio with multiple sense embeddings, and each s se corr sponds to a context clust r. Given an a notat d document D0 including M m ntions, and their sense sets acc rding to Section ??: M⇤l = { lj | lj 2 g(ml), ml 2 M}. In this i n, we d cribe how to determine the m ntio sen for e ch ntion l in the document. Based on language model, i ntifyi g me ti senses in a docum nt can be g r ed as maximizing their jo nt probabili y. H wev r, the global opti um is expensive, i which each mention gets an optimum sens , to arch over the space of all mention senses of all m ti n in th document. Thus, we approximately assign each mention independently: P (D0, . . . , sj , . . . , ) ⇡ Y P (D0|slj) · P (slj) ⇡ Y P (C(ml)|slj) · P (N̂ (ml)|slj) · P (slj) (9) wher P (C(ml)|slj) is proport onal o cosine simlarity b tween cont xt vector and ention sense luster ce er µlj to m asur th m tion’s local simi arity, namely local probabili y. N̂ (ml) note nei hbor mentions of ml c - occurring in a piece of text (e.g. a document , and P (N̂ (ml)|slj) is defined as glob l robability since it measures global coherence f neighbor mentions. The und rlyin ide is to achieve onsistent s mantics in a piece of text assu ing that all ntions inside it are talking about the sa e topic. In this paper, w regard the menti n senses identifie first as n ighbors of the rest mentions. P (slj) de tes prior probability of a menti n sense occurring in texts proportional to the frequency of corresponding entity in Wikipedia anchors: P (slj) = ( |Aej | |A| ) 2 [0, 1] where i a hyper-parameter to smooth the gaps between different tity frequencies, namely smoothing parameter. It controls the importance
5
400 401 402 403 404 405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 1 452 53 454 455
456
457
8
459
60
461
462
463
464
65
466
67
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2017 Submission ***. Confidenti l Review Copy. DO NOT DISTRIBUTE. predict he c ext words by max izi g th following objective function: Lw = X
wi,ml2D0 log P (C(wi)|wi
+ log P (C( l)|s⇤j ) (6)
where s⇤j = g(< l, j >) is obtained from a - chors in wikipedia articl s.
Thus, similar word nd menti n senses will be closed in text space, such as wfilm and s⇤Independ ce Day (film), or wcelebrations and s⇤Independence Day (US) b cause t y fr qu ntly occur in the same co text .
Similar to WDS, we maintai a conte t cluster for each men on e se, w ch ca be us d for disambiguation given the cont x s (Sect on 5). For example, in d1 of Figure , the c text cluster of s⇤ c sists of all c ntext v ctors Wh n ncou t ring a mentio , the context vector
we also maintai a context cluster ce ter µ⇤j for each ment on sense s⇤j , which is computed by averaging all the c ntext vector l ngi g to the cluster. We define context vector as the average sum of context wor embeddings
1 |C(wi)| P wj2C(wi) wj. The cluster cent r is helpful for inducing mention sense in c ntexts. When encounter a mention, we map it to a set f mention senses, and n fi d the ne t one according to the distance from its context vector to each m - tion sense cluster center, which will be discussed in Sectio 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j
s⇤Independence Day (US)
P ( j |C(ml), s⇤j )
P (C(wi)|wi) · P (C(ml)|s⇤j ) (7)
4.5 Joint Training
Considering all the above representation learning components, we define the overall objective function as lin ar c mbinations:
L = Lw + Le + Lm (8)
Th a ing of MPME is t ximize th abov fu ction, nd it r tiv ly update three types of embeddings. Also, w us eg tive sampling t chnique for effici ncy (Mikolov et al., 2013a). 5 Me ti S Dis mbiguatio
MPME l arns each menti n w th multipl sense emb dings, and each sen e corr spo ds to a c - text clu er. Giv an annotat d d cument D0 includi g M m nt ons, and their sens sets according to Sectio ??: M⇤l = {slj |slj 2 g(ml), ml 2 M}. In this section, we escribe h w d termine the m ntion sens f e ch me tion ml in th d cument.
Ba ed o languag m del, identi ying mentio e ses in a doc me t can be eg d d a maximiz-
ing their joi t pr bability. However, the global optimum is exp nsiv , i which e ch mention gets an optimum sense, to search over the space of all me tion sens s f all mentio s in t e document. Thus, we approxim tely assign each mention i - dependently:
P (D0, . . . , slj , . . . , ) ⇡ Y
P (D0|slj) · P (slj)
⇡ Y P (C(ml)|slj) · P (N̂ (ml)|slj) · P (slj) (9)
where P ( l)|slj is proportional to cosine similarity between co ext vector and m ntion sens cluster ce ter µlj to mea ure the mention’s l cal similarity, namely local probability.
N̂ (ml) denotes neighbor mentions of ml cooccurring in a piece of text (e.g. a document), and P (N̂ (ml)|slj) is defined as global probability since it measures global coh re c of neig b r mentions. The underlying idea is to achi ve consistent semantics in a piece of text assuming that all mentions inside it are talking about the same topic. In this paper, we regard the mention senses identifi d first as nei hbors f the rest ntions.
P (slj) denotes prior pr bability of a mention sense occurring in t xts proportional to the fr - que cy f c rrespondi g entity in Wikipedia nchors:
P (slj) = ( |Aej | |A| )
2 [0, 1]
where is a yper-parameter to smooth he gaps between different entity frequencies, namely smoothing paramet r It controls the im ortance5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457 458 459 460 461 462 463 464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2017 Sub ission ***. Confiden ial R view Copy. DO NOT DISTRIBUTE. predict the ont xt word by max miz ng the following objective function: Lw = X wi,ml2D0 log P (C( i)|wi) + log P (C(ml)|s⇤j ) (6) here ⇤j = g(< ml, j >) i b ain d fr m anch rs in wikipedia articles. Thus, similar words a d n sen s will be c os d n text spac , such as wfil a d s⇤Independenc Day (film), or w lebra s a d s⇤Independence Day (US) because h y freq e tly - cur in the same contex s.
Similar to WDS, we ain ain a co text cluster for each me t on se se, which an b u d fo disambiguation given h contexts Section 5 . F r example, in d1 f Fi ur 2, th cont xt clust r f s⇤ con ist f ll context vec ors Wh n ountering a mention, the context v ctor
we also maintain a context lus er cent r µ⇤j fo each me ti n s ns s⇤j , hich is c mput d by av aging all the co text vect r belonging to the cluster. We defi c t xt ve tor as the average sum of cont xt ord edding
1 |C(wi)| P wj2C( i) wj Th cluster center h lpful for in ucing mention sense i c ntexts. When encounter a me io , we map it t set of ment on senses, and th n find the nearest o e according to the istance from its ontext v ctor to each menti n s nse cluster c nt r, which will be discussed in Section 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j
s⇤Indep dence Day (US)
P (ej |C(ml), s⇤j )
P (C(wi)|wi) · P (C(ml)|s⇤j ) (7)
4.5 Joint Trai ing
Considering all the above representation learning components, we define the overall obj ive functio as lin ar combinations:
L = Lw + Le + Lm (8)
The training of MPME is to aximize the bove function, and it ratively update three types of emb ddings. Also, we use negative sampling t chnique for efficie cy (Mikolov et al., 2013a). 5 M n i n Sens Disambiguation MPME learn each ention with multiple sense beddi gs, and ach sense corresponds to a cont xt cluster. Give an annotated document D0 including M mentions, d t eir ense sets according to S ctio ??: M⇤l = {slj |slj 2 g(ml), ml 2 M}. I this section, we d scribe how to determine t e m ntion s nse f r ach me tion ml in the document.
Based o language model, identifying mention sense in ocument can be r g rded as maximizi g their joint pr b bility. However, the global optimu is exp nsive, in wh ch each mention gets a opt mu sense, to search over the space of all mention senses of all m nt ns in the docume t. T us, we approxi at ly assign e c ment on independently:
P (D0, . . . , slj , . . . , ) ⇡ Y
P D0|slj) · P (slj)
⇡ Y P (C( l)| j) · P (N̂ (ml)|slj) · P (slj) (9)
where P (C(ml)|slj) is pr p rtional to c sine similarity between c ntext vector and m ntion sense cluster center µlj t m asur the mentio ’s local si ilarity, nam ly local probability.
N̂ (ml) denot s neighbor mentions of ml cooccurring in a p ece of text (e.g. a d cument), an P (N̂ (ml)|slj) is d fin d as global probability since t measur s global coh rence of neig bor m tion . The und rlyi g idea is to achieve consistent semantics in a piece of text assum ng that all men ions inside it ar t lking about the sa e topic. In this pap r, we regard the ention senses identified first as neighbors of the rest mentions.
P (slj) denotes prior probability of a mention s nse occurring in texts proportion l to the frequency of corresponding entity in Wikipedia anchors:
P ( lj) = ( |Aej | |A| )
2 [0, 1]
where is a hyper-p rameter to smooth th gaps between diff rent entity frequenc es, namely smoothing parameter. It controls the importance
5
400 401 402 403 404 405 406 407 408 409 410 411 412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457 4 8 459 460 461 462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2017 Submission ***. Confidential Review Copy. DO NOT DISTRIBUTE. predict the context words by maximizing the following objective function: Lw = X wi,ml2D0 log P (C(wi)|wi) + log P (C(ml)| ⇤j ) (6) where s⇤j = g(< ml, ej >) is obtained from anchors in wikipedia articles.
Thus, similar words and mention sens s will be closed in text space, such as wfilm and s⇤Independence Day (film), or wcelebrations and s⇤Independence Day (US) because they frequ ntly occur in the same contexts.
Similar to WDS, we maintain a context cluster for each mention sense, which can be u d for di - ambiguation given the contexts (Section 5). F r example, in d1 of Figure 2, the context c uster of s⇤ consists of all context v c ors When en unt ring a mention, the context vector
we also maintain a context cluster center µ⇤j for each mention sens s⇤j , hich is computed by averaging all the context vectors belonging to the cluster. W define context vector as the average sum of context word embe di gs
1 |C(wi)| P wj2C(wi) wj. The cluster center is helpful for inducing mention sense in cont xts. When encounter a mention, we map it to a set of mention senses, and then find the nearest one according to the distance from its context vector to ea h mention sense cluster center, which will be discussed in Section 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j
s⇤Independence Day (US)
P (ej |C(ml), s⇤j )
P (C(wi)|wi) · P (C(ml)|s⇤j ) (7)
4.5 Joint Training
Considering all the above representation learning components, we define the overall obj ctive function as linear combinations:
L = Lw + Le + Lm (8)
The training of MPME is to maximize the above function, and iterativ ly updat three types of embeddings. Also, we us nega ive ampli g t chique for efficiency (M kolov t al., 2013 ). 5 M ntion S nse D sambiguation MPME lear s ch nti n w th multiple sens embed ings, and each sens corresp ds to a ontext cluster. Given an annotat d d cumen D0 including M mentions, and their sense sets according to Sectio ??: M⇤l = {slj |slj 2 g(ml), l 2 M}. In this section, e d s rib h w to d t rmine the entio sens f r ach men io ml in t d cument.
Bas d on languag mod l, identifying m nti n senses in a d cument can be rega ded as aximizing t eir joint prob b l ty. Howeve , the gl b l optimum is xpen ive, i which each mention g ts an p imum sense, to search over p c f all mention senses of all me tions in th do ument. Thus, we app ximately s ig eac ment on independently:
P (D0, . . . , slj , . . . , ) ⇡ Y
P (D0|slj) · P ( lj)
⇡ Y P (C(ml)|slj) · P (N̂ (ml)|slj) · P (slj) (9)
where P (C(m )|slj) is pr porti al t co i e similarity betwe n context vector a mentio ense cluster center µlj to measure the nti n’s cal similarity, nam ly local probability.
N̂ (ml) denotes neighbor me ti ns of l cooccurring in a piece of text (e.g. a docume t), and P (N̂ (ml)| lj) is defined s global probability since it measures global coherence of neighbor m ntions. The underlyi idea is to achi ve consistent semantics i a piece of text assuming that all mentions inside it are talking about the same topic. In this paper, we regard the m tion enses identified first as neighbors of the rest mentions.
P (slj) denotes prior probability of a mention
sense occurring in texts proportio al to the fre-
quency of corresponding entity in Wikipedia an-
chors:
P (slj) = ( |Aej | |A| )
2 [0, ]
where is a hyp r-para eter o moo the gaps between different entity frequencies, amely smoothing parameter. It controls th importance
5
400 401 402 403 404 405 406 407 408 409 410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457 458 459 460
461
462
463
464
465
466
467
468
469
470
471
472
473
7
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
91
492
493
494
495
496
497
498
499
ACL 2017 Su missio ***. Confid tia Review Copy. DO NOT DISTRIBUTE. predict the context words by maximizing the following objective functi n: Lw = X wi,ml2D0 log P (C(wi)|wi) + log P (C(ml)|s⇤j ) (6) where s⇤j = g(< ml, ej >) is obtained from anchors in wikipedia articles.
Thus, similar words and mention senses will be closed in text space, such as wfilm and s⇤Independence Day (film), or wcelebrations and s⇤Independence Day (US) because they frequently occur in the same contexts.
Similar to WDS, we maintain a context cluster for each mention sense, which can be used for disambiguation given the contexts (Sectio 5). For example, in d1 of Figure 2, the context cluster of s⇤ consists of all context vectors When ncountering a mention, the context vector
we also maintain a context cluster center µ⇤j for each mention sense s⇤j , which is computed by averaging all the context vectors belonging to the cluster. We define context vector as the average sum of context word embedd ngs
1 |C(wi)| P wj2C(wi) wj. The cluster center is helpful for inducing mention sense in contexts. When encounter a mention, we map it to a set of mention senses, and then find the nearest one according to the distance from its context vector to each mention sense cluster center, which will be discussed in Section 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j
s⇤Independence Day (US)
P (ej |C(ml), s⇤j )
P (C(wi)|wi) · P (C(ml)|s⇤j ) (7)
4.5 Joint Training
Considering all the above representation learning components, we define the overall objective function as linear combinations:
L = Lw + Le + Lm (8)
The training of MPME to aximiz the bove fun tion, d iterativ ly update t ree types of - beddings. Also, us negativ sa pli g - niqu for effic ency (Mik lov et al., 013a). 5 Mention Sense Disa biguatio MPME learns each me tio with multipl sense mb dding , and ach sense resp n s o a con-
t xt cluster. Given an ann tat d docu t D0 i - cluding M mentions, and their se se ets ac rd ing to Section ??: M⇤l = {slj |slj 2 g(ml), ml 2 M}. In this secti , we describ how to d t min the mention sense for each m ntion l i the document.
Based on langu ge m d l, id n ifyi g me t on senses i a d cume t can be reg rd d as maxim zing their j int pr bability. However, the global optimum is x ensive, in which each enti n g s an optimum sense, to sear h over the space of all mention senses of all mentions in th document. Thus, w approximat y ssig ach e t on independ tly:
P (D0, . . . , slj , . . . , ) ⇡ Y
P (D0|slj) · P (slj)
⇡ Y P (C(ml)| lj) · P (N̂ (ml)|slj) · P (slj) (9)
where P (C(ml)|slj) is prop rtio al to cosi imilarity between context vector and menti n sense cluster center µlj to easure the ment on’s local simi arity, na ely local probability.
N̂ (ml) denotes neig bor mentions of ml cooccurring in a piece of text (e.g. a document), and P (N̂ (ml)|slj) is defi ed as global pr babi - ity since it measu es global coherence f neighbor mentions. The underlying idea is to achieve consistent semantics in a pi ce of text assuming that all mentions inside it are talking ab ut the same topic. In this paper, we regard the mention senses
identified first as neighbors of the rest mentions.
P (slj) denotes prior pro ability of a me tion sense occurri g in texts proporti n l to the fr - quency f corresponding entity in Wikipedia anchors:
P (slj) = ( |Aej | |A| )
2 [0, 1]
where is a hyper-parameter to smooth the gaps between different entity frequencies, namel smoothing para ete . It controls he importa ce
5
400 401 402 403 404 405 406 407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 454 455 456 457
458
459
460
461
462
463
46
465
466
467
468
469
470
471
47
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
ACL 2017 Su mission ***. Confid ntial Review Copy. DO NOT DISTRIBUTE. predict the context words by maximizing the following objective functi n: Lw = X wi,ml2D0 log P (C(wi)|wi) + log P (C(ml)|s⇤j ) (6)
where s⇤j = g(< ml, ej >) is obtained from anchors in wikipedia articles.
Thus, similar words and mention senses will be closed in text space, such as wfilm and s⇤Independence Day (film), or wcelebrations and s⇤Independence Day (US) because they frequently occur in the same contexts.
Similar to WDS, we maintain a co text cluster for each mention sense, which can be used for d sambiguation given the contexts (Section 5). For example, in d1 of Figure 2, the context clu ter of s⇤ consists of all context vectors When encountering a mention, the context vector
we also maintain a context cluster center µ⇤j for each mention sense s⇤j , which is computed by averaging all the context v ctors belonging to the cluster. We define c ntext vector s the average sum of context word embeddings
1 |C(wi)| P wj2C(wi) wj. The cluster c nter is helpful for inducing mention sense in contexts. When encounter a mention, we map it to a set of mention senses, and then find the nearest on accordi g to the distance from its context vector to each mention sense cluster center, which will be discussed in Section 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j
s⇤Independence Day (US)
P (ej |C(ml), s⇤j )
P (C(wi)|wi) · P (C(ml)|s⇤j ) (7)
4.5 Joint Training
Considering all the above representation learning components, we define the overall objective function as linear combinations:
L = Lw + Le + Lm (8)
The training of MPME is t m xi iz the above functio , and iteratively update thr e type of - beddings. Als , w use neg tive sampli g t chnique for efficie cy (Mikolov et al., 2013a). 5 Me tion Sens Disambigu tion
MPME l arns each mention with multiple sense embeddings, and each sense corr sp nds t a co - text cluster. Given an annotated d cument D0 incl d ng M m ntions, and their ns sets ccording to Section ??: M⇤l = {slj |slj 2 g(ml), ml 2 M}. In this section, e describ how to d termin the mention sense for each mention l i t d - ument.
Based on language mod l, identifying mention senses in a document can be r g rd d as maxi izing their joint probability. However, the global optimum is expensive, in hich each mentio gets an optimum sense, to search over the space of all mention senses of all me tions i the document. Thus, we approximately assign ea h menti n independently:
P (D0, . . . , slj , . . . , ) ⇡ Y
P (D0|slj) · P (slj)
⇡ Y P (C(ml)|slj) · P ( ˆ (ml)| lj) · P (slj) (9)
where P (C(ml)|slj) is proportional to c sin simil rity bet een context vector nd mention sense cluster center µlj to measur the me ti ’s l cal similarity, namely local probability.
N̂ (ml) denotes eighb r m ntions f ml cooccurring in a piece of tex (e.g. a document), and P (N̂ (ml)|slj) is defined as global probability since it measur global cohe e e of n ighbor mentions. The underlying idea is to ach eve con-
sistent manti s a piece of text assuming ha
all mentions inside it are talking about the sam
opi . In this pap r, w regard the mention senses identified first as neighbors of the rest mentions. P (slj) denotes prior probability of a mention sense occurring in texts proportional to the frequency of rresponding ntity in Wikipedia anchors:
P (slj) = ( |Aej | |A| )
2 [0, 1]
where is a hyper-parameter t s ooth the gaps between different entity frequencies, namely smoothing parameter. It controls the importance
5
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450 451 452 453 45 455 456 457 458 459 60 61 462 463 464 465 466 467 468 6
7
471
472
473
474
475
476
477
478
79
80
481
482
483
48
485
486
487
88
89
490
491
492
493
494
495
496
497
98
99
ACL 2017 Submissi n ***. Confidential Review Copy. DO NOT DISTRIBUTE. predict the context words by maximizing the following objective function: Lw = X wi, l2D0 log P (C(wi)|wi) + log P (C(ml)|s⇤j ) (6) where s⇤j = g(< ml, ej >) is obtained from anchors in wikipedia articles. Thus, similar words and mention senses will be closed in text space, such a wfilm and s⇤Independence Day (film), or wcelebrations and s⇤Independence Day (US) because they fr quently occur in the same contexts. Si ilar to WDS, we maintain a context cluster for e ch mention sense, which can be used for disambiguation given the contexts (Section 5). For example, in d1 f Figure 2, the co text cluster of s⇤ consists of all context vec ors When encountering a mention, the context vector
we also maintain a context cluster center µ⇤j for each mention sense s⇤j , which is computed by averaging all the ontext vectors belongi g to the cluster. We d fi e context vector as the average sum of context word embeddings
1 |C(wi)| P wj2C(wi) wj. The cluster center is helpful for inducing mention sense in c nt xts. When encounter a mention, we map it to a set of mention senses, an then fi d the nearest one accordi g to the distance from its context vector to each mention sense clust r c nter, which will be discuss d in Section 5.
d1, d2, d3, s ⇤ j , wi/s ⇤ j , e3
s⇤Independ nce Day (US)
P (ej |C(ml), s⇤j )
P (C(wi)|wi) · P (C(ml)|s⇤j ) (7)
4.5 Joint Training
Considering all the above representati n learni g components, we define the overall objective function as linear combinations:
L = Lw + Le + Lm (8)
The training f MPME is to maximize the ab ve function, and it ratively update three types of em beddings. Also, we use negative sampling technique for effici n y (Mikol v et al., 2013a). 5 Mention Sens Disambiguation MPME learns each mention ith multiple sen embeddings, and each se se correspo ds to a context cluster. Giv n an a not t d docume t D0 i - cluding M mentions, an th ir sen s ts cc rd i g to Section ??: M⇤ = {slj | lj 2 g( l), l 2 M}. In this section, we describe how t determine the mention ense for each mentio ml in the cument.
Based on language model, ide tifying mention senses in a d cument can be r g ded as aximizi g their joi t probability. However, the global optimum is expensive, in which each me tion gets an optimum sense, to search over the space of all mention s ses f all mentions i the ocum nt. Thus, w approxim t ly assign e ch enti n independently:
P (D0, . . . , slj , . . . , ) ⇡ Y
P (D0|slj) · P (slj)
⇡ Y P (C(ml)|slj) · P (N̂ (ml)|slj) · P (slj) (9)
wh re P (C(ml)|slj) is p oportion l t cosi e similarity bet een cont x vector and menti n sense clu ter center µlj to mea ure th mention’s l cal similarity, nam ly local probability.
N̂ (ml) d n tes neighb r mentions of ml cooccurring in a piece of text (e.g. a ocume t), and P (N̂ (ml)|slj) is defined as global probability since it measures gl bal c her nce of ighbor men ions. The underlying idea is to achiev consist nt semantics in a pi ce of text assuming that all mentions inside it are talki g about the ame topic. In this paper, we r gard the m ntion senses identifi d first as neighbors of th rest e tions.
P (slj) d notes p ior pr bability of a mention sense occurring in texts proportiona to th frequency of corres onding entity in Wikipedia anchors:
P (slj) = ( |Aej | |A| )
2 [0, 1]
wher is a hyper-par meter to smooth the gaps between different entity frequencies, namely smo thing aramete . It controls th importance
K owledge Space
Text Spac
Figure 2: Framework of Multi-Prot typ Menti Embedding model.
dictionary. Similar to (Shen et al., 2015), we build the dictionary by collecting mention-entity pairs < ml, ej > from Wikipedia page titl , redirec pages and anchors. According to ej , ach p ir prov des a candidate menti n sense for ml, so we formally define the mapping as: M∗l = ⋃ g(< ml, ej >) = {s∗j}, where g(·) denotes th mapping function from entity mention to its reference entity, as well as the mapping from each m on to its sense.
We directly use t e anchors contained in the annotated text corpus D ′ for training. As Figure 2
shows, we replace the anchor <July 4th, Independence Day (US)> with the corresponding mention sense: Independence Day (US).
Representation Learning Using KB, A and D′ as input, we design three separate m dels and a unified optimization objective to jointly learn entity, word and mention sense representations into two semantic spaces. Entity embeddings can reflect their relatedness in the network, e.g. Independence Day (US) (e1) and Memorial Day (e3) are close to each other because they share some common neighbors, e.g., United States and Public holidays in the United States, as shown in Figure 2.
Word and mention embeddings are learned from the same semantic space. As two basic units in D′, their embeddings represent their distributed semantics in texts. For example, mention Inde-
pe dence Day and ord celebrations co-occur frequently when it refers to the hol day: Indep nd nce Day (US), thus they have similar representations. Withou disambigu ti g the mention senses, some ds, e.g., film, will also are si ilar repre e tatio s a I dep dence D y.
Besides, by introducing entity embeddings into our MPME fra ework, the knowledge informati n will als be distilled into mention ense mbeddings, so that the mention sense Memorial Day can be similar as Independence Day (US).
Mention Sense Disambiguation Given a document, in order to disambiguate the sense for each me tion, we design a la guag model based approach to measure the similarity of each candidate mention sense with the local contexts as well as the neighbor mentions occurri g in a limited size of window.
4 Representation Learning
4.1 Skip-Gram and CBOW model
Skip-gram and CBOW models (Mikolov et al., 2013a,b) are widely used to learn distributed word representations. Given a sequence of wordsD, the optimization objective of Skip-gram model is to maximize the average log probability:
L = ∑
wi∈D
∑
wo∈C(wi) P (wo|wi) (1)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
In contrast, CBOW model aims to predict the current word given its context words:
L = ∑
wi∈D P (wi|C(wi)) (2)
Formally, the conditional probability P (wo|wi) is defined using a softmax function:
P (wo|wi) = exp(wi ·wo)∑
wo∈D exp(wi ·wo) (3)
where wi,wo denote the input and output word vectors during training. Furthermore, these two models can be accelerated by using hierarchical softmax or negative sampling (Mikolov et al., 2013a,b).
4.2 Entity Representation Learning
Given a knowledge base KB, we aim to learn entity embeddings by modeling “contextual” entities, so that the entities sharing more common neighbors tend to have similar representations. Therefore, we extend Skip-gram model to a network by maximizing the log probability of being a neighbor entity.
Le = ∑
ej∈E logP (N (ej)|ej) (4)
Clearly, the neighbor entities serve a similar role as the context words in Skip-gram model. As shown in Figure 2, entity Memorial Day (e3) also has the neighbors of United States and Public holidays in the United States, thus their embeddings are close in knowledge space. These entity embeddings will be later used to learn mention representations.
4.3 Mention Representation Learning
As mentioned above, the textual context information and reference entity are helpful to distinguish different senses for a mention. Thus, given an anchor < ml, ej > and its context words C(ml), we combine mention sense embeddings with its context word embeddings to predict the reference entity by extending CBOW model. The objective function is as follows:
Lm = ∑
<ml,ej>∈A logP (ej |C(ml), s∗j ) (5)
where s∗j = g(< ml, ej >). Thus, if two mentions refer to similar entities and share similar contexts, they tend to be close in semantic vector space. Take Figure 1 as an example again, mention Independence Day and Memorial Day refer to similar entities Independence Day (US) (e1) and Memorial Day (e2) and share some similar context words, e.g., celebrations in documents d2, d3, so their sense embeddings are close to each other in text space.
4.4 Text Representation Learning Instead of directly using a word or a mention to predict the context words, we incorporate mention sense to joint optimize word and sense representations, which can avoid some noise introduced by the multiple senses mentions. For example, in Figure 2, without identifying the mention Independence Day as the holiday or the film, various dissimilar context words such as celebrations and film in documents d1, d2 will share similar semantics, which will further affect the performance of entity representations during joint training.
Given the annotated corpus D′, we use a word wi or a mention sense s∗j to predict the context words by maximizing the following objective function:
Lw = ∑
wi,ml∈D′ logP (C(wi)|wi)
+ logP (C(ml)|s∗j ) (6)
where s∗j = g(< ml, ej >) is obtained from anchors in Wikipedia articles.
Thus, words and mention senses will share the same vector space, where similar words and mention senses are close to each other, such as celebrations and Independence Day (US) because they frequently occur in the same contexts.
Similar to WDS, we maintain a context cluster for each mention sense, which can be used for mention sense disambiguation (Section 5). The context cluster of a mention sense s∗j contains all the context vectors of its mentionml. We compute context vector of ml by averaging the sum of its context word embeddings: 1|C(ml)| ∑ wj∈C(ml)wj. Further, the center of a context cluster µ∗j is defined as the average of context vectors of all mentions which refer to the sense. These context clusters will be later used to disambiguate the sense of a given mention with its contexts.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
4.5 Joint Training Considering all the above representation learning components, we define the overall objective function as linear combinations:
L = Lw + Le + Lm (7) The training of MPME is to maximize the above
function, and iteratively update three types of embeddings. Also, we use negative sampling technique for efficiency (Mikolov et al., 2013a).
5 Mention Sense Disambiguation
As we have learned multiple sense representations for each mention, given a new mention with its context, we also need to disambiguate and link the mention to a pre-learned sense.
Given an annotated document D′ includingM mentions, we first map each mention ml ∈ M to a set of senses according to Section 3: M∗l = {s∗j |s∗j ∈ g(ml)}. Then, based on language model, we identify the correct sense by maximizing a joint probability of all mention senses contained in the document. However, the global optimum is expensive with a time complexity of O(|M||M∗l |). Thus, we approximately identify each mention sense independently:
P (D′, . . . , s∗j , . . . , ) ≈ ∏
P (D′|s∗j ) · P (s∗j )
≈ ∏ P (C(ml)|s∗j ) · P (N̂ (ml)|s∗j ) · P (s∗j ) (8)
where P (C(ml)|s∗j ) denotes the probability of the local contexts of ml given its mention sense s∗j , namely local similarity. we define it proportional to the cosine similarity between current context vector and the sense context cluster center µ∗j as described in Section 4.4. It measures how likely a mention sense occurring together with current context words. For example, given the mention sense Independence Day (film), word film is more likely to appear within the context than the word celebrations. P (N̂ (ml)|slj) denotes the probability of the contextual mentions of ml given its sense slj , namely global probability, where N̂ (ml) is the collection of the neighbor mentions occurring together with ml in a predefined context window. We define it proportional to the cosine similarity between mention sense embeddings and the neighbor mention vector, which is computed similar to
context vector: ∑ 1 |N̂ (ml)| ŝlj, where ŝ l j is the correct sense for ml. The correct sense can be induced using either L2R (left to right) or S2C (simple to complex) orders (Chen et al., 2014).
Global probability assumes that there should be consistent semantics in a context window, and measures whether all neighbor mentions are related. For instance, two mentions Memorial Day and Independence Day occurring in the same document. If we already know that Memorial Day denotes a holiday, then obviously Independence Day has higher probability of being the holiday than the film. P (s∗j ) is a prior probability of sense s ∗ j indicating how possible it occurs without considering any additional information. We define it proportional to the frequency of sense s∗j in Wikipedia anchors:
P (s∗j ) = ( |As∗j | |A| ) γ γ ∈ [0, 1]
where As∗j is the set of anchors annotated with s∗j , and γ is a smoothing hyper-parameter to control the impact of prior on the overall probability, which is set by experiments (Section 6.4.2).
6 Experiments
Setup We choose Wikipedia, the March 2016 dump, as training corpus, which contains nearly 75 millions of anchors, 180 millions of edges among entities and 1.8 billions of tokens after preprocessing. We then learn representations for 1.5 millions of words, 5 millions of entities and 1.7 millions of mentions. The entire training process in 10 iterations costs nearly 8 hours on the server with 64 core cpu and 188GB memory.
We use the default settings in word2vec1, and set our embedding dimension as 200 and context window size as 5. For each positive example, we sample 5 negative examples.
Baseline Methods As far as we know, this is the first work to deal with mention ambiguity in the integration of text and knowledge representations, so there is no exact baselines for comparison. We use the method in (Yamada et al., 2016) as a baseline, marked as ALIGN because (1) this is the most similar work that directly aligns word and entity embeddings. (2) it achieves the state-of-the-art performance in entity linking task.
1https://code.google.com/archive/p/word2vec/
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Mention Sense Nearest words Nearest entities SPME Independence
Day lee-jackson, thanksgiving, diwali, strassenfest, chiraghan National Aboriginal and Torres Strait Islander Education Policy, E. Chandrasekharan Nair, Jean Aileen Little, Thessalian barbel, 1825 in birding and ornithology
MPME IndependenceDay (US) thanksgiving, parades, leejackson, festivities, celebrations Memorial Day, Labor Day, Thanksgiving, Thanksgiving (United States), Saint Patrick’s Day
Independence Day (film)
robocop, clockstoppers, mindhunters, tarantino, terminator The Terminator, True Lies, Total Recall (1990 film), RoboCop 2, Die Hard
Table 1: The nearest neighbors of mention Independence Day.
To investigate the effect of multi prototype, we degrade our method to single prototype, which means to use one sense to represent all mentions with the same phrase, namely Single-Prototype Mention Embedding (SPME). For example, we only use one unique sense vector for Independence Day whatever it denotes the holiday or the film.
6.1 Qualitative Analysis
We use cosine similarity to measure the similarity of two vectors, and present the top 5 nearest words and entities for two most popular senses of the mention Independence Day. Because ALIGN is incapable of dealing with multiple words, we only give the results of SPME and MPME.
As shown in Figure 1, without considering mention sense, the mention Independence Day can only show an dominant holiday sense based on SPME and ignore all other senses. Instead, MPME successfully learns two clear and distinct senses. For the sense Independence Day (US), all of it nearest words and entities, such as parades, celebrations, and Memorial Day, are holiday related, while for another sense Independence Day (film), its nearest words and entities, such as robocop and The Terminator, are all science fiction films. All the results demonstrate the effectiveness of our framework in multi-prototype mention embedding learning.
6.2 Entity Relatedness
To evaluate the quality of entity embeddings, we conduct experiments using the dataset which is designed for measuring entity relatedness (Ceccarelli et al., 2013; Huang et al., 2015; Yamada et al., 2016). The dataset contains 3314 entities, and each mention has 91 candidate entities on average with gold-standard labels indicating whether they are semantic related.
We compute cosine similarity between entity embeddings to measure their relatedness, and rank them in a descending order. To evaluate the ranking quality, we use two standard metrics:
normalized discounted cumulative gain (NDCG) (Järvelin and Kekäläinen, 2002) and mean average precision (MAP) (Schütze, 2008).
We design another baseline method: Entity2vec, which learns entity embeddings using the method described in Section 4.2, without joint training with word and mention sense embeddings.
Table 2: Entity Relatedness.
NDCG@1 NDCG@5 NDCG@10 MAP ALIGN 0.416 0.432 0.472 0.410 Entity2vec 0.593 0.595 0.636 0.566 SPME 0.593 0.594 0.636 0.566 MPME 0.613 0.613 0.654 0.582
As shown in Table 2, ALIGN gets lower performance2 than Entity2vec, because it doesn’t consider the mention phrase ambiguity and yields lots of noise when forcing entity embeddings to satisfy word embeddings and aligning them into the unified space. For example, entity Gente (magazine) should be more relevance to entity France, the place where its company locates. However, ALIGN mixed various meanings of mention Gente (e.g. the song) and ranked some bands higher (e.g. entity Poolside (band)).
SPME also doesn’t consider the ambiguity of mentions but achieves comparative results with Entity2vec. We analyze the reasons and find that, it can avoid some noise by using word embeddings to predict entities. MPME outperforms all the other methods, which demonstrates that the unambiguous textual information is helpful to refine the entity embeddings.
6.3 Word Analogical Reasoning
Following (Mikolov et al., 2013a; Wang et al., 2014), we use word analogical reasoning task to evaluate the quality of word embeddings. The dataset consists of 8869 semantic questions
2We failed to reproduce the positive result in the original paper, meanwhile the authors are unable to release their code.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
694
695
696
697
698
699
(“Paris”:“France”::“Rome”:?), and 10675 syntactic questions (e.g. “sit”:“sitting”::“walk”:?). We solve it by finding the closest word vector w? to wFrance−wParis+wRome according to cosine similarity. We compute accuracy for top 1 nearest word to measure the performance.
Table 3: Word Similarity.
Word2vec ALIGN SPME MPME Semantic 66.78 68.34 71.65 71.65 Syntactic 61.58 59.73 55.28 54.75
We also adopt Word2vec3 as an additional baseline method, which provides a standard to measure the impact from other components on word embeddings.
Table 3 shows the results. We can see that ALIGN, SPME and MPME, achieve higher performance in semantic task, because relations among entities (e.g. country-capital relation for entity France and Paris) enhance the semantics in word embeddings through jointly training. On the other hand, their syntactic performance decrease. We analyze the results and find that, most of errors are caused by the bias of semantics. For example, given a query “pleasant”:“unpleasant”::“possibly”:?, our model tends to return the word which has high semantic similarity with query words, such as probably, instead of the syntactical similar words, e.g., impossibly.
The word embeddings of MPME achieve the best semantic performance even be forced to satisfy with entity embeddings in joint training, mainly because (1) text representation learning has better generalization ability due to the larger size of training examples than entities’ (i.e. 1.8b v.s. 0.18b) as well as relatively smaller size of vocabulary (i.e. 1.5m v.s. 5m). (2) unambiguous mention embeddings capture both textual context information and knowledge, and thus enhance word and entity embeddings.
6.4 A Case Study: Entity Linking As a study case, we further apply the multiprototype mention embeddings to entity linking task. Given mentions in text, entity linking aims to link them to a predefined knowledge base. One of the main challenges in this task is the ambiguity of entity mentions.
3https://code.google.com/archive/p/word2vec/
We use the public dataset AIDA created by (Hoffart et al., 2011), which includes 1393 documents and 27,816 mentions referring to Wikipedia entries. The dataset has been divided into 946, 216 and 231 documents for the purpose of training, developing and testing. Following (Pershina et al., 2015; Yamada et al., 2016), we use a public available dictionary to generate candidate entities and mention senses. For evaluation, we rank the candidate entities for each mention and report both standard micro (aggregates over all mentions) and macro (aggregates over all documents) precision over top-ranked entities.
6.4.1 Supervised Entity Linking Yamada et al. (2016) designs a list of features for each mention and candidate entity pair. By incorporating these features into a supervised learningto-rank algorithm, Gradient Boosting Regression Tree (GBRT), each pair is obtained a relevance score indicating whether they should be linked to each other. Following their recommended parameters, we set the number of trees as 10,000, the learning rate as 0.02 and the maximum depth of the decision tree as 4.
Based on word and entity embeddings learned by ALIGN, the key features in (Yamada et al., 2016) are from two aspects: (1) the cosine similarity between context words and candidate entity, and (2) the coherence among “contextual” entities in the same document.
To evaluate the performance of multi-prototype mention embeddings, we incorporate the following features into GBDT for comparison: (1) the cosine similarity between the current context vector and the sense context cluster center µ∗j , which denotes how likely the mention sense to refer to the candidate entity, (2) the cosine similarity between the current context vector and the mention sense embeddings.
As shown in Table 4, we can see that ALIGN performs better than SPME. This is because SPME learns word embeddings and entity embeddings in separate semantic spaces, and fails to measure the similarity between context words and candidate entities. However, MPME com-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Table 5: Performance of unsupervised methods
Cucerzan Kulkarni Hoffart Shirakawa Alhelbawy MPME (L2R) MPME (S2C) Micro P@1 0.510 0.729 0.818 0.823 0.842 0.882 0.885 Macro P@1 0.437 0.767 0.819 0.830 0.875 0.875 0.890
putes the similarity between context words with mention sense, instead of entities, thus achieves the best performance, which also demonstrates the high quality of the mention sense embeddings.
6.4.2 Unsupervised Entity Linking
Linking a mention to a specific entity equals to disambiguating mention senses since each candidate entity corresponds to a mention sense. As described in Section 5, we disambiguate senses in two orders: (1) L2R (from left to right), and (2) S2C (from simple to complex).
We evaluate our unsupervised disambiguation methods on the entire AIDA dataset. To be fair, we choose the state-of-the-art unsupervised methods which are proposed in (Hoffart et al., 2011; Alhelbawy and Gaizauskas, 2014; Cucerzan, 2007; Kulkarni et al., 2009; Masumi Shirakawa and Nishio, 2011) using the same dataset.
Table 5 shows the results. We can see that our two methods outperforms all other methods, while MPME (L2R) is more efficient and easy to apply.
We analyze the results and observe a disambiguation bias to popular senses. For example, there are three mentions in the sentence “Japan began the defence of their Asian Cup I title with a lucky 2-1 win against Syria in a Group C championship match on Friday”, where the country name Japan and Syria actually denote the entity of their national football teams, and the football match name Asian Cup I has little ambiguity. Compared to the team, the sense of country occurs more frequently and has a dominant prior probability, which greatly affects the disambiguation. By incorporating local similarity and global probability, both the context words (e.g. defence or match) and the neighbor mentions (e.g. Asian Cup I) provide us enough clues to identifying a soccer related mention sense instead of the country.
Influence of Smoothing Parameter As mentioned above, a mention sense may possess a dominant prior probability and greatly affect the disambiguation. So we introduce a smoothing parameter γ to controls its importance to the overall probability. Figure 3 shows the linking accuracy under different values of γ on the dataset of AIDA.
γ = 0 denotes we don’t use any prior knowledge, and γ = 1 indicates the case without smoothing parameter.
We can see that both micro and macro accuracy decrease a lot if we don’t use the parameter (γ = 1). Only using local and global probabilities for disambiguations (γ = 0) achieves a comparable performance, and when γ = 0.05, both accuracy reach their peaks, which is optimal and default value in our experiments.
7 Conclusion
In this paper, we propose a novel Multi-Prototype Mention Embedding model that jointly learns word, entity and mention sense embeddings. These mention sense capture both textual context information and knowledge from reference entities, and provide an efficient approach to disambiguate mention sense in text. We conduct a series of experiments to demonstrate that multiprototype mention embedding improves the quality of both word and entity representations. Using entity linking as a study case, we apply our disambiguation method as well as the multi-prototype mention embeddings on the benchmark dataset, and achieve the state-of-the-art.
In the future, we will improve the scalability of our model and learn multi-prototype embeddings for the mentions without reference entities in a knowledge base, and introduce compositional approaches to model the internal structures of multiword mentions.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
References Ayman Alhelbawy and Robert J Gaizauskas. 2014.
Graph ranking for collective named entity disambiguation. In ACL (2). pages 75–80.
Diego Ceccarelli, Claudio Lucchese, Salvatore Orlando, Raffaele Perego, and Salvatore Trani. 2013. Learning relatedness measures for entity linking. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management. ACM, pages 139–148.
Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In EMNLP. Citeseer, pages 1025– 1035.
Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on wikipedia data .
Xu Han, Zhiyuan Liu, and Maosong Sun. 2016. Joint representation learning of text and knowledge for knowledge graph completion. CoRR abs/1611.04125.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 782–792.
Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proc. ACL.
Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. arXiv preprint arXiv:1504.07678 .
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS) 20(4):422–446.
Sayali Kulkarni, Amit Singh, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2009. Collective annotation of wikipedia entities in web text. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pages 457–466.
Jiwei Li and Dan Jurafsky. 2015. Do multi-sense embeddings improve natural language understanding? In Proc. EMNLP.
Massimiliano Mancini, José Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2016. Embedding words and senses together via joint knowledgeenhanced training. CoRR abs/1612.02703.
Haixun Wang Yangqiu Song Zhongyuan Wang Kotaro Nakayama Takahiro Hara Masumi Shirakawa and Shojiro Nishio. 2011. Entity disambiguation based on a. technical report. In Technical Report MSR-TR2011-125. Microsoft Research.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR abs/1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS. pages 3111–3119.
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient nonparametric estimation of multiple embeddings per word in vector space. In Proc. EMNLP.
Maria Pershina, Yifan He, and Ralph Grishman. 2015. Personalized page rank for named entity disambiguation. In HLT-NAACL. pages 238–243.
Joseph Reisinger and Raymond J Mooney. 2010. Multi-prototype vector-space models of word meaning. In Proc. NAACL.
Hinrich Schütze. 2008. Introduction to information retrieval. In Proceedings of the international communication of association for computing machinery conference.
Wei Shen, Jianyong Wang, and Jiawei Han. 2015. Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Transactions on Knowledge and Data Engineering 27(2):443–460.
Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Tie-Yan Liu. 2014. A probabilistic model for learning multi-prototype word embeddings. In COLING.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. ACL Association for Computational Linguistics .
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph and text jointly embedding. In Proc. EMNLP.
Zhigang Wang and Juan-Zi Li. 2016. Text-enhanced representation learning for knowledge graph. In Proc. ACL.
Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for relation extraction. In Proc. ACL.
Jiawei Wu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2016. Knowledge representation via joint learning of sequential text and knowledge graphs. CoRR .
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. In Proc. CoNLL.
This paper addresses the problem of disambiguating linking textual entity mentions into a given background knowledge base (in this case, English Wikipedia).  (Its title and introduction are a little overblown misleading, since there is a lot more to bridging text and knowledge than the EDL task, but EDL is a core part of the overall task nonetheless.)  The method is to perform this bridging via an intermediate layer of representation, namely mention senses, thus following two steps: (1) mention to mention sense, and (2) mention sense to entity.  Various embedding representations are learned for the words, the mention senses, and the entities, which are then jointly trained to maximize a single overall objective function that maximizes all three types of embedding equally.    Technically the approach is fairly clear and conforms to the current deep processing fashion and known best practices regarding embeddings; while one can suggest all kinds of alternatives, it  not clear they would make a material difference.  Rather, my comments focus on the basic approach.  It is not explained, however, exactly why a two-step process, involving the mention senses, is better than a simple direct one-step mapping from word mentions to their entities.  (This is the approach of Yamada et al., in what is called here the ALIGN algorithm.)  Table 2 shows that the two-step MPME (and even its simplification SPME) do better.  By why, exactly?  What is the exact difference, and additional information, that the mention senses have compare4ed to the entities?  To understand, please check if the following is correct (and perhaps update the paper to make it exactly clear what is going on).    For entities: their profiles consist of neighboring entities in a relatedness graph.                    This graph is built (I assume) by looking at word-level relatedness of the entity definitions (pages in Wikipedia).  The profiles are (extended skip-gram-based) embeddings.    For words: their profiles are the standard distributional semantics approach, without sense disambiguation.    For mention senses: their profiles are the standard distributional semantics approach, but WITH sense disambiguation.  Sense disambiguation is performed using a sense-based profile ( language model ) from local context words and neighboring mentions, as mentioned briefly just before Section 4, but without details.  This is a problem point in the approach.  How exactly are the senses created and differentiated?  Who defines how many senses a mention string can have?  If this is done by looking at the knowledge base, then we get a bijective mapping between mention senses and entities -  that is, there is exactly one entity for each mention sense (even if there may be more entities).  In that case, are the sense collection  definitional profiles built starting with entity text as  seed words ?                    If so, what information is used at the mention sense level that is NOT used at the entity level?  Just and exactly the words in the texts that reliably associate with the mention sense, but that do NOT occur in the equivalent entity webpage in Wikipedia?  How many such words are there, on average, for a mention sense?                    That is, how powerful necessary is it to keep this extra differentiation information in a separate space (the mention sense space) as opposed to just loading these additional words into the Entity space (by adding these words into the Wikipedia entity pages)?    If the above understanding is essentially correct, please update Section 5 of the paper to say so, for (to me) it is the main new information in the paper.    It is not true, as the paper says in Section 6, that   u2026this is the first work to deal with mention ambiguity in the integration of text and knowledge representations, so there is no exact baselines for comparison d.  The TAC KBP evaluations for the past two years have hosted EDL tasks, involving eight or nine systems, all performing exactly this task, albeit against Freebase, which is considerably larger and more noisy than Wikipedia.  Please see http:  nlp.cs.rpi.edu kbp 2016  .    On a positive note: I really liked the idea of the smoothing parameter in Section 6.4.2.  Post-response: I have read the authors' responses.  I am not really satisfied with their reply about the KBP evaluation not being relevant, but that they are interested in the goodness of the embeddings instead.  In fact, the only way to evaluate such 'goodness' is through an application.  No-one really cares how conceptually elegant an embedding is, the question is: does it perform better?