Automatic Induction of Synsets from a Graph of Synonyms
This paper presents a new graph-based approach that induces synsets using synonymy dictionaries and word embeddings. Firstly, we build a weighted graph of synonyms extracted from commonly available resources, such as Wiktionary. Secondly, we apply word sense induction to deal with ambiguous words. Finally, we cluster the disambiguated version of the ambiguous input graph into synsets. Our meta-clustering approach lets us use an efficient hard clustering algorithm to perform fuzzy clustering of the graph. Despite its simplicity, our approach shows excellent results outperforming five analogous state-of-the-art methods in terms of F-score on four different gold standard datasets for English and Russian derived from large-scale manually constructed lexical resources.
1 Introduction
A synset is a set of mutual synonyms, which can be represented as a clique where nodes are words and edges are synonymy relations. Synsets represent word senses and are building blocks of WordNet (Miller, 1995) and similar resources as thesauri and lexical ontologies. These resources are crucial for many natural language processing applications which require common sense reasoning, such as information retrieval (Gong et al., 2005) and question answering (Kwok et al., 2001; Zhou et al., 2013). However, for most languages no manually-constructed resource comparable to the English WordNet in terms of coverage and quality is available. For instance, Kiselev et al. (2015) presents a comparative analysis of lexical resources available for the Russian language. This
lack of linguistic resources for many languages urges the development of new methods for automatic construction of WordNet-like resources.
Wikipedia1, Wiktionary2, OmegaWiki3 and other collaboratively-created resources contain a large amount of lexical semantic information— yet designed to be human-readable and not formally structured. While semantic relations can be automatically extracted using such tools as DKPro JWKTL4 and Wikokit5, words in these relations are not disambiguated. For instance, the synonymy pairs (bank, streambank) and (bank, banking company) will be connected via the word “bank”, while they refer to the different senses. This problem stems from the fact that articles in Wiktionary and similar resources list undisambiguated synonyms. They are easy to disambiguate for humans while reading a dictionary article, but can be a source of errors for a language processing system.
The contribution of this paper is a novel approach which resolves ambiguities of the input graph enabling fuzzy clustering. The method takes as an input synonymy relations between potentially ambiguous terms available in humanreadable dictionaries and transforms them into a disambiguated machine readable representation in the form of synsets. Our method, called WATSET, is based on a new meta-algorithm for fuzzy graph clustering. The name choice reflects the underlying principle: discover the correct word senses (“wat”) and then construct the synsets (“set”).
In contrast to the projects like BabelNet (Navigli and Ponzetto, 2012) and UBY (Gurevych et al., 2012), which rely on English WordNet as
1http://www.wikipedia.org 2http://www.wiktionary.org 3http://www.omegawiki.org 4https://dkpro.github.io/dkpro-jwktl 5https://github.com/componavt/wikokit
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
a pivot for mapping of existing resources, our approach requires no such pivot lexical ontologies. Besides, it outperforms analogous state-of-the-art methods for synset induction. An implementation of the WATSET method is available online, along with induced lexical resources.6
2 Related Work
Methods based on resource mapping, such as BabelNet and UBY, gather various existing lexical resources across multiple languages and perform their linking to obtain a machine-readable repository of lexical semantic knowledge. In its core, BabelNet was obtained by mapping the Princeton WordNet and Wikipedia enhanced by the machine translation of the results. Later, other resources were mapped to this core, including Wiktionary and OmegaWiki. UBY has a similar architecture, but relies on similarity of dictionary definitions and existing cross-lingual links for mapping.
The potential advantages of our approach as compared to BabelNet/UBY is the (1) absence of the error-prone procedures of mapping and machine translation; (2) possibility to model the target language more accurately as senses of the same word in different languages may be different; (3) no need for a pivot English resource.
A related branch of methods deals with coarsification of sense inventories of fine-grained lexical resources, such as (Snow et al., 2007).
Methods based on word sense induction try to induce sense representations without the need for any initial lexical resource by extracting semantic relations from text. In particular, word sense induction (WSI) based on word ego networks clusters graphs of words semantically related to the ambiguous word (Lin, 1998; Pantel and Lin, 2002; Dorow and Widdows, 2003; Véronis, 2004; Hope and Keller, 2013). Each cluster corresponds to a word sense. An ego network consists of a single node (ego) together with the nodes they are connected to (alters) and all the edges among those alters. In the case of WSI, such a network is a local neighbourhood of one word. Nodes of the ego network are the words which are semantically similar to the target word.
Such approaches are able to discover homonymous senses of words, e.g., “bank” as slope versus “bank” as organisation (Di Marco and Navigli, 2012). However, as the graphs are composed
6scheme://domain.tld/anonymized
of semantically related words obtained using distributional methods (Baroni and Lenci, 2010; Biemann and Riedl, 2013), the resulting clusters by no means can be considered synsets. Namely, (1) they contain words related not only via synonymy relation, but via a mixture of relations such as synonymy, hypernymy, co-hyponymy, antonymy, etc. (Heylen et al., 2008); (2) clusters are not unique, i.e., one word can occur in several clusters referring to the same sense, while in WordNet a word used in the given sense occurs only in a single synset.
In our approach, to induce synsets, we use word ego network clustering similarly as in word sense induction approaches, but apply them to the graph of semantically clean synonyms.
Methods based on clustering of synonyms, such as our approach, induce the resource from an ambiguous graph of synonyms where edges were extracted from manually-created resources. According to the best of our knowledge, most experiments either used graph-based word sense induction applied to text-derived graphs or used a mapping-based method which already assumes availability of a WordNet-like resource. A notable exception is the ECO approach by Gonçalo Oliveira and Gomes (2014), which was used to induce a WordNet of the Portuguese language called Onto.PT.7 We use this approach and five other state-of-the-art graph clustering algorithms as the baselines:
• ECO (Gonçalo Oliveira and Gomes, 2014) is a fuzzy clustering algorithm that was used to induce synsets of a Portuguese WordNet out of available synonymy dictionaries. The algorithm starts by adding random noise to edge weights. Then, the approach launches Markov Clustering of this graph several times to estimate the probability of each word pair being in the same synset. Finally, these candidate pairs passing through the θ threshold are added to output synsets.
• MaxMax (Hope and Keller, 2013) is a fuzzy clustering algorithm designed initially for the word sense induction task. In the nutshell, pairs of nodes are grouped if either node has a maximal affinity to the other. The algorithm starts by converting the input undirected graph into the directed graph by keep-
7http://ontopt.dei.uc.pt
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 1: Outline of the WATSET method for synset induction.
ing maximal nodes of each node. Next, all nodes are marked as root nodes. Finally, for each root node, the following procedure is repeated: all transitive children of this root form a cluster and marked as non-root nodes; a root node together with all its transitive children form a fuzzy cluster.
• Markov Clustering (MCL) (van Dongen, 2000) is a hard clustering algorithm for graphs based on simulation of stochastic flow in graphs. MCL simulates random walks within a graph by alternation of two operators called expansion and inflation, which recomputes the class labels. Notably, it has been successfully used for the word sense induction task (Dorow and Widdows, 2003).
• Chinese Whispers (Biemann, 2006) is a hard clustering algorithm for weighted graphs that can be considered as a special case of MCL with a simplified class update step. At each iteration, the labels of all the nodes are updated according to the majority labels among the neighbouring nodes. The algorithm has a meta-parameter that controls graph weights that can be set to three values: (1) top sums over the neighbourhood’s classes; (2) nolog downgrades the influence of a neighbouring node by its degree or (3) log of its degree.
• Clique Percolation Method (CPM) (Palla et al., 2005) is a fuzzy clustering algorithm for unweighted graphs that builds up the clusters from k-cliques corresponding to fully connected sub-graphs of k nodes. While this method is only commonly used in social network analysis, we decided to add it to the comparison as synsets are essentially cliques of synonyms, and it is natural to try using an algorithm based on detection of cliques.
3 The WATSET Method
The goal of the method is to induce a set of unambiguous synsets by grouping individual ambigu-
ous synonyms. An outline of the proposed approach is depicted in Figure 1. The method takes a dictionary of ambiguous synonymy relations and a text corpus as an input and outputs synsets. Note that the method can be used without a background corpus, yet as our experiments will show, corpusbased information improves the results, when utilizing it for weighting the word graph.
A synonymy dictionary can be perceived as a graph, where the nodes correspond to lexical entries (words) and the edges connect pairs of the nodes when the synonymy relation between them holds. The cliques in such a graph naturally form densely connected sets of synonyms corresponding to concepts (Kamps et al., 2004). Given the fact that the clique problem in a graph is NPcomplete (Bomze et al., 1999), an efficient graph clustering algorithm like the MCL algorithm can be used for finding a global segmentation of the graph. However, the hard clustering property of this algorithm does not handle polysemy: one word can have several senses but will be assigned to only one cluster. To deal with this limitation, a word sense induction procedure is used to induce senses for all words. Finally, the disambiguated word sense graph is clustered globally to induce the synsets from this disambiguated word graph.
More specifically, the method consists of five steps presented in Figure 1: (1) learning word embeddings; (2) constructing the ambiguous weighted graph of synonyms G; (3) inducing the word senses; (4) constructing the disambiguated weighted graph G′ by disambiguating of neighbours w.r.t. the induced word senses; (5) global clustering of the graph G′.
3.1 Learning Word Embeddings
Since the different graph clustering algorithms are sensitive to edge weighing, we consider the distributional semantic similarity measures based on word embeddings as a possible edge weighing approach for our synonymy graph. As we show further, this approach yields the best results.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
3.2 Construction of a Synonymy Graph
We construct the synonymy graph G = (V,E) as follows. The set of nodes V includes every lexeme appearing in the input synonymy dictionaries. The set of undirected edgesE is composed of all edges (u, v) ∈ V × V retrieved from one of the input synonymy dictionaries. We consider three edge weight representations:
• ones that assigns every edge the constant weight of 1;
• count that weights the edge (u, v) as the number of times the synonymy pair appeared in the input dictionaries;
• sim that assigns every edge (u, v) a weight equal to the cosine similarity of the skip-gram word vectors.
As the graph G is likely to have polysemous words, the goal is to separate the individual word senses using graph-based word sense induction.
3.3 Word Sense Induction
We use a graph-based word sense induction method that is similar to the curvature-based approach of Dorow and Widdows (2003). In particular, removal of the nodes participating in many triangles tends to separate the original graph into several connected components. Thus, given a word u, we extract a network of its nearest neighbours from the synonymy graph G. Then, we remove the original word u from this network and run a hard graph clustering algorithm that assigns one node to one and only one cluster. In our experiments, we test Chinese Whispers and Markov Clustering. As the result, each cluster is hopefully representing a different sense of the word u, e.g.:
bank1 {streambank, riverbank, . . . } bank2 {bank company, . . . } bank3 {bank building, building, . . . } bank4 {coin bank, penny bank, . . . }
We denote man1, man2 and other items as word senses referred to as, e.g., senses(man). We denote as ctx(s) a cluster corresponding to the word sense s. Note that the context words have no sense labels. They are recovered by the disambiguation approach described below.
3.4 Disambiguation of Neighbours
The result of the previous step is splitting word nodes into (one or more) sense nodes. However,
nearest neighbours of each new node are still ambiguous, e.g., (bank3, building?).
For recovering these sense labels of the neighbouring words, we employ the following sense disambiguation approach. For each word u in the context ctx(s) of the sense s, we estimate the most similar sense of that word û using the cosine similarity measure between the context of the sense s and the context of the candidate sense u′ in a vector space model:
û = argmax u′∈ senses(u)
cos(ctx(s), ctx(u′)).
This approach makes it possible to construct a disambiguated context ĉtx(s) that corresponds to the certain word senses appearing in ctx(s):
ĉtx(s) = {û : u ∈ ctx(s)}.
3.5 Global Clustering
Finally, we construct the word sense graph G′ = (V ′, E′) using the disambiguated senses instead of the regular words and establishing the edges between these disambiguated senses:
V ′ = {s : u ∈ V ∧ s ∈ senses(u)}, E′ = {(s, û) : s ∈ V ′ ∧ û ∈ ĉtx(s)}.
As the result, running a hard clustering algorithm on G′ produces the desired set of synsets.
Figure 2 illustrates the process of disambiguation of an input ambiguous graph on the example of the word “bank”. As one may observe, disambiguation of the nearest neighbours is a necessity to be able to construct a global version of the sense-aware graph. Note that current approaches to WSI, e.g., (Véronis, 2004; Biemann, 2006; Hope and Keller, 2013), do not perform this step, but perform only local analysis of the graph.
4 Evaluation
We conduct our experiments on the data for two different languages. We evaluate our approach on two datasets for English to demonstrate its performance on a resource-rich language. Additionally, we evaluate it on two Russian datasets since Russian is a good example of an under-resourced language with a clear need for synset induction.
4.1 Gold Standard Datasets
For each language, we used two differently constructed lexical semantic resources listed in Table 1 to obtain gold standard synsets.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Ambiguous Graph before Local Clustering (WSI)
bank?
streambank?
riverbank?
streamside?
building?
bank building?
Sense Inventory of Nodes, Undisambiguated Neighbours
bank2 building?
bank building?
streamside1
bank1
riverbank?
streambank?
Disambiguated Graph before Global Clustering
bank2 building2
bank building1
bank1 streambank3
riverbank2 streamside1
Figure 2: Disambiguation of an input ambiguous graph using local clustering (WSI) to facilitate global clustering of words into synsets.
English. We use WordNet8 (Miller, 1995), a popular English lexical database crafted by a group of expert lexicographers. WordNet contains general vocabulary and appears to be de facto gold standard in similar tasks. We used WordNet 3.1 to derive the synonymy pairs from the synsets for the evaluation. Also, we use BabelNet9 (Navigli and Ponzetto, 2012), a large-scale multilingual semantic network constructed automatically using WordNet, Wikipedia and other resources. We retrieved all the synonymy pairs from the BabelNet 3.7 synsets that were marked as English. We also considered using TWSI by Biemann (2013), but found that it contains mostly hypernymy and co-hyponymy relations instead of being a viable source of synonymy information.
Russian. We use RuThes10 (Loukachevitch, 2011), a lexical ontology for Russian containing both general vocabulary and domain-specific synsets related to sport, finance, economics, etc. Up to a half of the words in this resource are multiword expressions (Kiselev et al., 2015), which is due to the coverage of domain-specific vocabulary. RuThes is constructed in the traditional way, namely by a small group of expert lexicog-
8https://wordnet.princeton.edu/ 9http://babelnet.org/
10http://www.labinform.ru/pub/ruthes
raphers. Also, we use Yet Another RussNet11 (YARN) by Braslavski et al. (2016) as yet another gold standard for Russian. The resource is constructed using crowdsourcing and mostly covers general vocabulary. Particularly, non-expert users are allowed to edit synsets in a collaborative way loosely supervised by a team of project curators. Due to the ongoing development of the resource, we selected as the gold standard only those synsets that were edited at least eight times in order to filter out noisy incomplete synsets.
Resource # words # synsets # synonyms WordNet En 148 730 117 659 152 254 BabelNet En 11 710 137 6 667 855 28 822 400 RuThes Ru 119 836 31 528 474 537 YARN Ru 9 141 2 210 48 291
Table 1: Statistics of the gold standard datasets.
4.2 Evaluation Metrics
To evaluate the quality of the induced synsets, we transformed them into binary synonymy relations and computed precision, recall, and F-score on the basis of the overlap of these binary relations with the binary relations from the gold standard datasets. Given a synset uniting n words, we generate a set of n(n−1)2 pairs of synonyms.
4.3 Word Embeddings
English. We use the standard 300-dimensional word embeddings trained on the 100 billion tokens Google News corpus (Mikolov et al., 2013).
Russian. We use the 500-dimensional word embeddings trained using the skip-gram model with negative sampling (Mikolov et al., 2013) using a context window size of 10 with the minimal word frequency of 5 on a 12.9 billion tokens corpus of books lib.rus.ec. These embeddings were shown to produce state-of-the-art results for Russian in the RUSSE shared task12 and are part of the Russian Distributional Thesaurus13 (RDT).
4.4 Input Dictionary of Synonyms
For each language, we constructed a synonymy graph using openly available language resources. The statistics of the graphs used as the input in the further experiments are shown in Table 2.
11https://russianword.net/en/ 12http://www.dialog-21.ru/en/
evaluation/2015/semantic_similarity 13http://russe.nlpub.ru/downloads
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
WordNet (English)
F −
s c o re
CW MCL MaxMax ECO CPM Watset
0.00
0.05
0.10
0.15
RuThes (Russian)
F −
s c o re
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
BabelNet (English)
F −
s c o re
CW MCL MaxMax ECO CPM Watset
0.0
0.1
0.2
0.3
0.4
YARN (Russian)
F −
s c o re
Figure 3: Impact of the different graph weighting schemas on the performance of synset induction:
ones, count, sim. Here, each bar corresponds to the top performance of each method.
English. The synonyms were extracted from the English Wiktionary, which is the largest Wiktionary as the present moment according to the lexical coverage, using the DKPro JWKTL tool by Zesch et al. (2008). Only the words marked as English have been extracted.
Russian. The synonyms from three sources were combined to improve lexical coverage of the input dictionary and to enforce confidence in jointly observed synonyms: (1) synonyms listed in the Russian Wiktionary extracted using the Wikokit tool by Krizhanovsky and Smirnov (2013); (2) the dictionary of Abramov (2007); and (3) the Universal Dictionary of Concepts (Dikonov, 2013). While these two latter resources are specific to Russian, Wiktionary is available for most languages. Note that the same input synonymy dictionary was used by authors of YARN to construct synsets using crowdsourcing. Therefore, results on the YARN dataset show how close an automatic synset induction method can approximate manually created synsets provided the same starting raw linguistic materials.
Language # words # synonyms English 77 871 71 816 Russian 74 395 202 313
Table 2: Statistics of the input datasets.
5 Results
We compare WATSET with five state-of-the art graph clustering methods presented in Section 2: Chinese Whispers (CW), Markov Clustering (MCL), MaxMax, ECO clustering, and
the clique percolation method (CPM). In our experiments, we rely on our own implementation of MaxMax and ECO as reference implementations are not available. For CW14, MCL15 and CPM16, the available implementations have been used. During the evaluation, we delete the clusters equal or larger than the threshold of 150 words as they hardly can represent any meaningful synset. The notation WATSET[MCL, CWtop] means using MCL for word sense induction and Chinese Whispers in the top mode for global clustering.
5.1 Impact of Graph Weighting Schema
Figure 3 presents an overview of the evaluation results on both datasets. The first step being common for all of the tested synset induction methods is the graph construction. Thus, we started with an analysis of the three ways to weight edges of the graph introduced in Section 3.2: binary scores (ones), frequencies (count), and semantic similarity scores (sim). The results across various configurations and methods indicate that using the weights based on the similarity scores provided by word embeddings is the best strategy for all the methods but MaxMax on both English datasets. However, in these cases, its performance using the ones weighing does not exceed the other methods using the sim weighing. Therefore, we report all further results on the basis of the sim weights.
14https://github.com/tudarmstadt-lt/ chinese-whispers
15http://java-ml.sourceforge.net 16https://networkx.github.io
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
WordNet BabelNet Method # words # synsets # synonyms Precision Recall F-score Precision Recall F-score WATSET[CWtop, MCL] 77 871 55 874 78 303 0.290 0.397 0.335 0.272 0.478 0.346 WATSET[CWtop, CWtop] 77 871 55 873 78 337 0.290 0.396 0.335 0.271 0.478 0.346 WATSET[MCL, MCL] 77 871 33 948 127 331 0.328 0.334 0.331 0.340 0.414 0.374 WATSET[MCL, CWlog] 77 871 32 843 143 120 0.341 0.305 0.322 0.350 0.379 0.364 MCL 77 871 25 467 141 042 0.316 0.327 0.321 0.343 0.399 0.369 CWnolog 77 871 24 220 170 499 0.338 0.276 0.304 0.357 0.343 0.350 CPMk=2 54 374 19 749 80 325 0.148 0.551 0.234 0.243 0.525 0.332 MaxMax 77 871 25 532 436 571 0.363 0.131 0.192 0.380 0.163 0.228 ECO 77 870 56 152 25 726 0.075 0.765 0.136 0.109 0.717 0.190
Table 3: Comparison of the synset induction methods on datasets for English. All methods rely on the similarity edge weighting (sim); best configurations of each method in terms of F-scores are shown for each dataset. Results are sorted by F-score on WordNet, top three values of each metric are boldfaced.
RuThes YARN Method # words # synsets # synonyms Precision Recall F-score Precision Recall F-score WATSET[CWlog, MCL] 74 395 48 484 317 040 0.224 0.101 0.139 0.466 0.396 0.428 WATSET[MCL, MCL] 74 395 32 407 378 769 0.220 0.093 0.131 0.459 0.400 0.427 WATSET[CWnolog, CWnolog] 74 395 48 443 327 125 0.225 0.097 0.136 0.471 0.386 0.424 MCL 74 395 19 277 328 789 0.187 0.128 0.152 0.345 0.546 0.423 WATSET[MCL, CWnolog] 74 395 31 129 429 823 0.231 0.084 0.123 0.500 0.352 0.413 CWnolog 74 395 16 683 660 424 0.220 0.069 0.105 0.460 0.357 0.402 MaxMax 74 395 24 250 560 096 0.186 0.147 0.164 0.230 0.566 0.327 CPMk=3 14 662 3 729 46 007 0.047 0.189 0.075 0.061 0.561 0.110 ECO 74 395 61 126 15 784 0.022 0.630 0.042 0.002 0.898 0.004
Table 4: Results on Russian sorted by F-score on YARN, top three values of each metric are boldfaced.
5.2 Performance Analysis
Table 3 and 4 present the evaluation results on the all the datasets for both languages. For each method, we show the best configurations in terms of F-score. One may note that the granularity of the resulting synsets, especially on the Russian dataset, is very different, ranging from 3 729 synsets for the CPMk=3 method to 61 126 induced by the ECO method. Both tables report the number of words, synsets and synonyms after pruning huge clusters larger than 150 words. Without this pruning, the MaxMax and CPM methods tend to discover giant components obtaining almost zero precision as we generate all possible pairs of nodes in such clusters. The other methods did not demonstrate such behavior.
WATSET robustly outperformed all other methods according to F-score on both English datasets (Table 3) and on the YARN dataset for Russian (Table 4). Also, it outperformed all other methods according to precision on both Russian datasets. The disambiguation of the input graph performed by the WATSET method splits nodes belonging to several local communities to several nodes significantly facilitating the clustering task otherwise complicated by the presence of the hubs that wrongly link semantically unrelated nodes.
Amusingly, in all the cases, the toughest competitor was a hard clustering algorithm—MCL (van Dongen, 2000). We observed that the “plain” MCL successfully groups the monosemous words,
but produces nonsensical clusters of moderate size, e.g., 20–50 words, when a polysemous word appears, resulting in the precision drop w.r.t. WATSET. Chinese Whispers, a simplified version of MCL, converges faster due to node label randomization which leads to a stricter stopping condition. CW thus does not amplify the hubs between the unrelated nodes and therefore produces smaller clusters in average. As the result, the “plain” CW offers higher precision than the “plain” MCL at the cost of lower recall.
Using CW instead of MCL for word sense induction in WATSET expectedly produces finegrained senses. Interestingly, at the global clustering step, these senses erroneously tend to form coarse-grained synsets connecting unrelated senses of the ambiguous words. This explains the generally higher precision of WATSET[MCL, ·].
The MaxMax algorithm showed mixed results. On the one hand, it outputs large clusters uniting more than hundred nodes. This inevitably leads to a high recall, as it is clearly seen in the results for Russian because such synsets passed through our threshold of 150 words. The synsets produced on English datasets were even larger and did not pass, which resulted in low recall. On the other hand, the smaller synsets having at most 10–15 words were identified correctly. MaxMax appeared to be extremely sensible to the edge weighing, which complicates its practical use.
The CPM algorithm showed unsatisfactory results which indicated by emitting giant compo-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
nents uniting thousands words. Such clusters have been automatically pruned, but the rest clusters connect virtually every node left after the pruning. This is confirmed by the high values of recall. As one increase the minimal number of elements in the clique k, precision effectively grows, but at the cost of a dramatic drop in recall. We suppose that the network structure assumptions exploited by CPM do not accurately model the structure of the synonymy graphs in the present task.
Finally, the ECO method yielded the worst results because the most cluster candidates failed to pass through the constant threshold used for estimating whether a pair of words should be included in the same cluster. Most synsets produced by this method were trivial, i.e., containing only one word. The remaining synsets for both languages have at most three words having been connected by a chance due to the edge noising procedure used in this method resulting in low precision.
6 Discussion
On difference in absolute scores. While the results obtained for the English datasets are in agreement (Tables 3 and 5), the results measured on RuThes and YARN datasets for Russian show a similar global picture in terms of relevant ranking of the methods, yet absolute scores are largely different (cf. Figure 3). This difference stems from the difference of the gold standards. RuThes is more domain-specific in terms of vocabulary, so our input set of generic synonymy dictionaries has a limited coverage on this dataset. On the other hand, recall calculated on YARN reaching significantly higher levels as this resource was manually built on the basis of exactly the same initial resources. Low performance of the cross-evaluation of two resources presented in Table 5 confirms this observation. No single resource for Russian can obtain high precision scores on another one. Surprisingly, even BabelNet, which integrates most of available lexical resources, still does not reach recall largely higher than 0.5.17 Finally, note that the results of this cross-dataset evaluation are not directly comparable to results in Table 4 as in our experiments we use much smaller input dictionaries than those used by BabelNet.
On sparseness of the input dictionary. Table 6 presents some examples of the obtained synsets
17We used BabelNet 3.7 as in our study for English and retrieved all 3 497 327 synsets that were marked as Russian.
of various sizes for the best WATSET configuration on both languages. As one might observe, the quality of the results is highly plausible. However, one limitation of all approaches considered in this paper is the dependence on completeness of the input dictionary of synonyms. In some parts of the input synonymy graph, important bridges between words can be missing, leading to smallerthan-desired synsets. A promising extension of the present methodology is using the distributional models to enhance connectivity of the graph to further improve recall of the method by adding extra relations.
7 Conclusion
In this paper, we presented a new robust approach to fuzzy graph clustering that relies on a hard clustering method. Using ego network clustering, the nodes belonging to several local communities are split into several nodes each belonging to one local community. The transformed “disambiguated” graph is then clustered using an efficient hard graph clustering algorithm, obtaining a fuzzy clustering as the result. The disambiguated graph contains fewer hubs connecting unrelated nodes from different communities and thus facilitates clustering. We apply this meta-clustering algorithm to the task of synset induction, obtaining the best results on two datasets for two different natural language in terms of precision and competitive results in terms of F-score, as compared to five state-ofthe-art graph clustering methods.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This paper presents a graph-based approach for producing sense-disambiguated synonym sets from a collection of undisambiguated synonym sets.  The authors evaluate their approach by inducing these synonym sets from Wiktionary and from a collection of Russian dictionaries, and then comparing pairwise synonymy relations (using precision, recall, and F1) against WordNet and BabelNet (for the English synonym sets) or RuThes and Yet Another RussNet (for the Russian synonym sets).  The paper is very well written and structured.              The experiments and evaluations (or at least the prose parts) are very easy to follow.              The methodology is sensible and the analysis of the results cogent.  I was happy to observe that the objections I had when reading the paper (such as the mismatch in vocabulary between the synonym dictionaries and gold standards) ended up being resolved, or at least addressed, in the final pages.  The one thing about the paper that concerns me is that the authors do not seem to have properly understood the previous work, which undercuts the stated motivation for this paper.  The first instance of this misunderstanding is in the paragraph beginning on line 064, where OmegaWiki is lumped in with Wiktionary and Wikipedia in a discussion of resources that are  not formally structured  and that contain  undisambiguated synonyms .  In reality, OmegaWiki is distinguished from the other two resources by using a formal structure (a relational database) based on word senses rather than orthographic forms.              Translations, synonyms, and other semantic annotations in OmegaWiki are therefore unambiguous.  The second, and more serious, misunderstanding comes in the three paragraphs beginning on lines 092, 108, and 120.  Here the paper claims that both BabelNet and UBY  rely on English WordNet as a pivot for mapping of existing resources  and criticizes this mapping as being  error-prone .  Though it is true that BabelNet uses WordNet as a pivot, UBY does not.  UBY is basically a general-purpose specification for the representation of lexical-semantic resources and of links between them.  It exists independently of any given lexical-semantic resource (including WordNet) and of any given alignment between resources (including ones based on  similarity of dictionary definitions  or  cross-lingual links ).  Its maintainers have made available various databases adhering to the UBY spec; these contain a variety of lexical-semantic resources which have been aligned with a variety of different methods.  A given UBY database can be *queried* for synsets, but UBY itself does not *generate* those synsets.  Users are free to produce their own databases by importing whatever lexical-semantic resources and alignments thereof are best suited to their purposes.  The three criticisms of UBY on lines 120 to 125 are therefore entirely misplaced.  In fact, I think at least one of the criticisms is not appropriate even with respect to BabelNet.  The authors claim that Watset may be superior to BabelNet because BabelNet's mapping and use of machine translation are error-prone.  The implication here is that Watset's method is error-free, or at least significantly less error-prone.  This is a very grandiose claim that I do not believe is supported by what the authors ought to have known in advance about their similarity-based sense linking algorithms and graph clustering algorithms, let alone by the results of their study.  I think this criticism ought to be moderated.              Also, I think the third criticism (BabelNet's reliance on WordNet as a pivot) somewhat misses the point -- surely the most important issue to highlight isn't the fact that the pivot is English, but rather that its synsets are already manually sense-annotated.  I think the last paragraph of  u00a71 and the first two paragraphs of  u00a72 should be extensively revised. They should focus on the *general* problem of generating synsets by sense-level alignment translation of LSRs (see Gurevych et al., 2016 for a survey), rather than particularly on BabelNet (which uses certain particular methods) and UBY (which doesn't use any particular methods, but can aggregate the results of existing ones).  It may be helpful to point out somewhere that although alignment translation methods *can* be used to produce synsets or to enrich existing ones, that's not always an explicit goal of the process.  Sometimes it's just a serendipitous (if noisy) side-effect of aligning translating resources with differing granularities.  Finally, at several points in the paper (lines 153, 433), the  synsets  of TWSI of JoBimText are criticized for including too many words that are hypernyms, co-hypnomyms, etc. instead of synonyms.  But is this problem really unique to TWSI and JoBimText?  That is, how often do hypernyms, co-hypernyms, etc. appear in the output of Watset?  (We can get only a very vague idea of this from comparing Tables 3 and 5, which analyze only synonym relations.)  If Watset really is better at filtering out words with other semantic relations, then it would be nice to see some quantitative evidence of this.  Some further relatively minor points that should nonetheless be fixed:  * Lines 047 to 049: The sentence about Kiselev et al. (2015) seems rather useless.  Why bother mentioning their analysis if you're not going to tell us what they found?  * Line 091: It took me a long time to figure out how  wat  has any relation to  discover the correct word sense .  I suppose this is supposed to be a pun on  what .  Maybe it would have been better to call the approach  Whatset ?  Or at least consider rewording the sentence to better explain the pun.  * Figure 2 is practically illegible owing to the microscopic font.  Please increase the text size!  * Similarly, Tables 3, 4, and 5 are too small to read comfortably.  Please use a larger font.              To save space, consider abbreviating the headers ( P,  R ,  F1 ) and maybe reporting scores in the range 0 100 instead of 0 1, which will eliminate a leading 0 from each column.  * Lines 517 522: Wiktionary is a moving target.  To help others replicate or compare against your work, please indicate the date of the Wiktionary database dump you used.  * Throughout: The constant switching between Times and Computer Modern is distracting.  The root of this problem is a longstanding design flaw in the ACL 2017 LaTeX style file, but it's exacerbated by the authors' decision to occasionally set numbers in math mode, even in running text.  Please fix this by removing    usepackage{times}  from the preamble and replacing it with either    usepackage{newtxtext}   usepackage{newtxmath}  or    usepackage{mathptmx}  References:  I Gurevych, J. Eckle-Kohler, and M. Matuschek, 2016. Linked Lexical Knowledge Bases: Foundations and Applications, volume 34 of Synthesis Lectures on Human Language Technologies, chapter 3: Linking Algorithms, pages 29-44. Morgan & Claypool.  ----  I have read the author response.


Most methods to learn bilingual word embeddings rely on large parallel corpora, which is difficult to obtain for most language pairs. This has motivated an active research line to relax this requirement, with methods that use document-aligned corpora or bilingual dictionaries of a few thousand words instead. In this work, we further reduce the need of bilingual resources using a very simple self-learning approach that can be combined with any dictionary-based mapping technique. Our method exploits the structural similarity of embedding spaces, and works with as little bilingual evidence as a 25 word dictionary or even an automatically generated list of numerals, obtaining results comparable to those of systems that use richer resources.
1 Introduction
Multilingual word embeddings have attracted a lot of attention in recent times. In addition to having a direct application in inherently cross-lingual tasks like machine translation (Zou et al., 2013) and cross-lingual entity linking (Tsai and Roth, 2016), they provide an excellent mechanism for transfer learning, where a model trained in a resourcerich language is transferred to a less-resourced one, as shown with part-of-speech tagging (Zhang et al., 2016), parsing (Xiao and Guo, 2014) and document classification (Klementiev et al., 2012), among others.
Most methods to learn these multilingual word embeddings make use of large parallel corpora (Gouws et al., 2015; Luong et al., 2015), but there have been several proposals to relax this requirement, given its scarcity in most language pairs. A possible relaxation is to use document-aligned or
label-aligned comparable corpora (Søgaard et al., 2015; Vulic and Moens, 2016; Mogadala and Rettinger, 2016), but large amounts of such corpora are not always available for some language pairs.
An alternative approach that we follow here is to independently train the embeddings for each language on monolingual corpora, and then learn a linear transformation to map the embeddings from one space into the other by minimizing the distances in a bilingual dictionary, usually in the range of a few thousand entries (Mikolov et al., 2013a; Artetxe et al., 2016). However, dictionaries of that size are not readily available for many language pairs, specially those involving less-resourced languages.
In this work, we reduce the need of large bilingual dictionaries to much smaller seed dictionaries. Our method can work with as little as 25 word pairs, which are straightforward to obtain assuming some basic knowledge of the languages involved. The method can also work with trivially generated seed dictionaries of numerals (i.e. 1-1, 2-2, 3-3, 4-4...) making it possible to learn bilingual word embeddings without any real bilingual data. In either case, we obtain very competitive results, comparable to other state-of-the-art methods that make use of much richer bilingual resources.
The proposed method is an extension of existing mapping techniques, where the dictionary is used to learn the embedding mapping and the embedding mapping is used to induce a new dictionary iteratively in a self-learning fashion (see Figure 1). In spite of its simplicity, our analysis of the implicit optimization objective reveals that the method is exploiting the structural similarity of independently trained embeddings.
Section 2 analyzes previous work. Section 3 describes the self-learning framework, while Section 4 presents the experiments. Section 5 analyzes the underlying optimization objective.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Learn D using nearest neighbor
D = 1-a, 2-b, 3-c, 4-x, 5-y
XW and Y in same space
W
Learn W using D
and rotate X D = 1-a, 2-b, 3-c.
XWYX
Figure 1: A general schema of the proposed self-learning framework. Previous works learn a mapping W based on the seed dictionary D, which is then used to learn the full dictionary. In our proposal we use the new dictionary to learn a new mapping, iterating until convergence.
2 Related work
We will first focus on bilingual embedding mappings, which are the basis of our proposals, and then on other unsupervised and weakly supervised methods to learn bilingual word embeddings.
2.1 Bilingual embedding mappings
Methods to induce bilingual mappings work by independently learning the embeddings in each language using monolingual corpora, and then learning a transformation from one embedding space into the other based on a bilingual dictionary.
The first of such methods is due to Mikolov et al. (2013a), who learn the linear transformation that minimizes the sum of squared Euclidean distances for the dictionary entries. The same optimization objective is used by Zhang et al. (2016), who constrain the transformation matrix to be orthogonal. Xing et al. (2015) incorporate length normalization in the training of word embeddings and maximize the cosine similarity instead, enforcing the orthogonality constraint to preserve the length normalization after the mapping. Finally, Lazaridou et al. (2015) use max-margin optimization with intruder negative sampling.
While all the previous methods learn a single linear transformation from the source language into the target language, Faruqui and Dyer (2014) use canonical correlation analysis to learn a separate linear transformation for each language that maps them to a shared vector space. Lu et al. (2015) extend this work and apply deep canonical correlation analysis to learn non-linear transformations.
Artetxe et al. (2016) propose a general framework that clarifies the relation between Mikolov
et al. (2013a), Xing et al. (2015), Faruqui and Dyer (2014) and Zhang et al. (2016) as variants of the same core optimization objective, and show that a new variant is able to surpass them all. While most of the previous methods use gradient descent, Artetxe et al. (2016) propose an efficient analytical implementation for those same methods, which we use in this work.
A prominent application of bilingual embedding mappings, with a direct application in machine translation (Zhao et al., 2015), is bilingual lexicon extraction, which is also the main evaluation method for bilingual embedding mappings. More specifically, the learned mapping is used to induce the translation of source language words that were missing in the original dictionary, usually by taking their nearest neighbor word in the target language according to cosine similarity, although Dinu et al. (2015) propose a more complex globally-corrected approach.
2.2 Unsupervised and weakly supervised bilingual embeddings
As mentioned before, our method works with as little as 25 word pairs, while the methods discussed previously use thousands of pairs. The only exception in this regard is the work by Zhang et al. (2016), who only use 10 word pairs to learn the orthogonal mapping minimizing the sum of squared Euclidean distances, with good results on transfer learning for part-of-speech tagging. Our experiments will show that, although their method captures coarse-grained relations, it fails on finergrained tasks like bilingual lexicon induction.
Miceli Barone (2016) attempts to learn bilingual embedding mappings without any bilingual
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Algorithm 1 Traditional framework Input: X (source embeddings) Input: Z (target embeddings) Input: D (seed dictionary)
1: W ← LEARN MAPPING(X , Z, D) 2: D ← LEARN DICTIONARY(X , Z, W ) 3: EVALUATE DICTIONARY(D)
data. He uses adversarial autoencoders (Makhzani et al., 2016), combining an encoder that maps the source language embeddings into the target language, a decoder that reconstructs the original embeddings, and a discriminator that distinguishes mapped embeddings from real target language embeddings. Although promising, the reported performance of the learned embeddings is poor in comparison to other methods.
Finally, the induction of bilingual knowledge from monolingual corpora is closely related to the decipherment scenario, for which models that incorporate word embeddings have also been proposed (Dou et al., 2015). However, decipherment is only concerned with translating text from one language to another and relies on complex statistical models that are designed specifically for that purpose, while our approach is more general and learns task-independent multilingual embeddings.
3 Proposed self-learning framework
As discussed in Section 2.1, a common evaluation task (and practical application) of bilingual embedding mappings is to induce bilingual lexicons, that is, to obtain the translation of source words that were missing in the training dictionary, which are then compared to a gold standard test dictionary for evaluation. This way, one can say that the seed (train) dictionary is used to learn a mapping, which is then used to induce a better dictionary (at least in the sense that it is larger). Algorithm 1 summarizes this framework.
Following this observation, we propose to use the output dictionary in Algorithm 1 as the input of the same system in a self-learning fashion which, assuming that the output dictionary was indeed better than the original one, should serve to learn a better mapping and, consequently, an even better dictionary the second time. The process can then be repeated iteratively to obtain a hopefully better mapping and dictionary each time until some convergence criterion is met. Algorithm 2 summa-
Algorithm 2 Proposed self-learning framework Input: X (source embeddings) Input: Z (target embeddings) Input: D (seed dictionary)
1: repeat 2: W ← LEARN MAPPING(X , Z, D) 3: D ← LEARN DICTIONARY(X , Z, W ) 4: until convergence criterion 5: EVALUATE DICTIONARY(D)
rizes this alternative framework that we propose. Our method can be combined with any embedding mapping and dictionary induction technique (see Section 2.1). However, efficiency turns out to be critical for a variety of reasons. First of all, by enclosing the learning logic in a loop, the total training time is increased by the number of iterations. Even more importantly, our framework requires to explicitly build the entire dictionary at each iteration, whereas previous work tends to induce the translation of individual words ondemand later at runtime. Moreover, from the second iteration onwards, it is this induced, full dictionary that has to be used to learn the embedding mapping, and not the considerably smaller seed dictionary as it is typically done. In the following two subsections, we respectively describe the embedding mapping method and the dictionary induction method that we adopt in our work with these efficiency requirements in mind.
3.1 Embedding mapping
As discussed in Section 2.1, most previous methods to learn embedding mappings use variants of gradient descent. As the only exceptions in this regard, both Faruqui and Dyer (2014) and Artetxe et al. (2016) provide more efficient exact methods, with the second being simpler, more efficient, and yielding better results as reported in their paper. In the following, we present their method, explicitly incorporating the dictionary in the formalization so it can be smoothly integrated in our selflearning algorithm.
Let X and Z denote the word embedding matrices in two languages so that Xi∗ corresponds to the ith source language word embedding and Zj∗ corresponds to the jth target language embedding. While Artetxe et al. (2016) assume these two matrices are aligned according to the dictionary, we drop this assumption and represent the dictionary explicitly as a binary matrix D, so that Dij = 1
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
if the ith source language word is aligned with the jth target language word. The goal is then to find the optimal mapping matrix W ∗ so that the sum of squared Euclidean distances between the mapped source embeddings Xi∗W and target embeddings Zj∗ for the dictionary entries Dij is minimized:
W ∗ = arg min W ∑ i ∑ j Dij ||Xi∗W − Zj∗||2
Following Artetxe et al. (2016), we length normalize and mean center the embedding matrices X and Z in a preprocessing step, and constrain W to be an orthogonal matrix (i.e. WW T = W TW = I), which serves to enforce monolingual invariance, preventing a degradation in monolingual performance while yielding to better bilingual mappings. Under such orthogonality constraint, minimizing the squared Euclidean distance becomes equivalent to maximizing the dot product, so the above optimization objective can be reformulated as follows:
W ∗ = arg max W
Tr ( XWZTDT ) where Tr (·) denotes the trace operator (the sum of all the elements in the main diagonal). The optimal orthogonal solution for this problem is given by W ∗ = UV T , where XTDZ = UΣV T is the singular value decomposition of XTDZ. Since the dictionary matrix D is sparse, this can be efficiently computed in linear time with respect to the number of dictionary entries.
3.2 Dictionary induction
As discussed in Section 2.1, practically all previous work uses nearest neighbor retrieval for word translation induction based on embedding mappings. In nearest neighbor retrieval, each source language word is assigned the closest word in the target language. In our work, we use the dot product between the mapped source language embeddings and the target language embeddings as the similarity measure, which is roughly equivalent to cosine similarity given that we apply length normalization followed by mean centering as a preprocessing step (see Section 3.1). This way, following the notation in Section 3.1, we set Dij = 1 if j = argmaxk (Xi∗W ) ·Zk∗ and Dij = 0 otherwise1.
1Note that we induce the dictionary entries starting from the source language words. We experimented with other alternatives in development, with minor differences.
While we find that independently computing the similarity measure between all word pairs is prohibitively slow, the computation of the entire similarity matrix XWZT can be easily vectorized using popular linear algebra libraries, obtaining big performance gains. However, the resulting similarity matrix is often too large to fit in memory when using large vocabularies. For that reason, instead of computing the entire similarity matrix XWZT in a single step, we iteratively compute submatrices of it using vectorized matrix multiplication, find their corresponding minima each time, and then combine the results.
4 Experiments and results
In this section, we experimentally test the proposed method in bilingual lexicon induction and crosslingual word similarity. Subsection 4.1 describes the experimental settings, while Subsections 4.2 and 4.3 present the results obtained in each of the tasks. The code and resources necessary to reproduce our experiments are available at https://github.com/XXXXX2.
4.1 Experimental settings
For easier comparison with related work, we evaluated our mappings on bilingual lexicon induction using the public English-Italian dataset by Dinu et al. (2015), which includes monolingual word embeddings in both languages together with a bilingual dictionary split in a training set and a test set3. The embeddings were trained with the word2vec toolkit with CBOW and negative sampling (Mikolov et al., 2013b)4, using a 2.8 billion word corpus for English (ukWaC + Wikipedia + BNC) and a 1.6 billion word corpus for Italian (itWaC). The training and test sets were derived from a dictionary built form Europarl word alignments and available at OPUS (Tiedemann, 2012), taking at random 1,500 uniformly distributed entries as the test set and the 5,000 most frequent of the remaining word pairs as the training set.
In addition to English-Italian, we selected two other languages from different language families with publicly available resources. We thus cre-
2To be released upon acceptance. 3http://clic.cimec.unitn.it/
˜georgiana.dinu/down/ 4The context window was set to 5 words, the dimension of the embeddings to 300, the sub-sampling to 1e-05 and the number of negative samples to 10, and the vocabulary was restricted to the 200,000 most frequent words
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
English-Italian English-German English-Finnish 5,000 25 num. 5,000 25 num. 5,000 25 num.
Mikolov et al. (2013a) 34.93% 0.00% 0.00% 35.00% 0.00% 0.07% 25.91% 0.00% 0.00% Xing et al. (2015) 36.87% 0.00% 0.13% 41.27% 0.07% 0.53% 28.23% 0.07% 0.56% Zhang et al. (2016) 36.73% 0.07% 0.27% 40.80% 0.13% 0.87% 28.16% 0.14% 0.42% Artetxe et al. (2016) 39.27% 0.07% 0.40% 41.87% 0.13% 0.73% 30.62% 0.21% 0.77% Our method 39.67% 37.27% 39.40% 40.87% 39.60% 40.27% 28.72% 28.16% 26.47%
Table 1: Accuracy on bilingual lexicon induction for different seed dictionaries
ated analogous datasets for English-German and English-Finnish. In the case of German, the embeddings were trained on the 0.9 billion word corpus SdeWaC, which is part of the WaCky collection (Baroni et al., 2009), which was also used for English and Italian. Given that Finnish is not included in this collection, we used the 2.8 billion word Common Crawl corpus provided at WMT 20165 instead, which we tokenized using the Stanford Tokenizer (Manning et al., 2014). In addition to that, we created training and test sets for both pairs from their respective Europarl dictionaries from OPUS following the exact same procedure used for English-Italian, and the word embeddings were also trained using the same configuration as Dinu et al. (2015).
Given that the main focus of our work is on small seed dictionaries, we created random subsets of 2,500, 1,000, 500, 250, 100, 75, 50 and 25 entries from the original training dictionaries of 5,000 entries. This was done by shuffling once the training dictionaries and taking their first k entries, so it is guaranteed that each dictionary is a strict subset of the bigger dictionaries.
In addition to that, we explored using automatically generated dictionaries as a shortcut to practical unsupervised learning. For that purpose, we created numeral dictionaries, consisting of words matching the [0-9]+ regular expression in both vocabularies (e.g. 1-1, 2-2, 3-3, 1992-1992 etc.). The resulting dictionary had 2772 entries for English-Italian, 2148 for English-German, and 2345 for English-Finnish. While more sophisticated approaches are possible (e.g. involving the edit distance of all words), we believe that this method is general enough that should work with practically any language pair, as arabic numerals are often used even in languages with a different writing system (e.g. Chinese and Russian).
While bilingual lexicon induction is a standard
5http://www.statmt.org/wmt16/ translation-task.html
evaluation task for seed dictionary based methods like ours, it is unsuitable for bilingual corpus based methods, as statistical word alignment already provides a reliable way to derive dictionaries from bilingual corpora and, in fact, this is how the test dictionary itself is built in our case. For that reason, we carried out some experiments in crosslingual word similarity as a way to test our method in a different task and allowing compare it to systems that use richer bilingual data. There are no many crosslingual word similarity datasets, and we used the RG-65 and WordSim353 crosslingual datasets for English-German and the WordSim-353 crosslingual dataset for EnglishItalian as published by Camacho-Collados et al. (2015) 6.
As for the convergence criterion required by our method, we decide to stop training when the improvement on the average similarity for the automatically generated dictionary falls below a given threshold from one iteration to the next. As the average similarity ranges from -1 to 1, we decide to set this threshold at 1e-6, which we find to be a very conservative value yet enough that training takes a reasonable amount of time. The curves in the next section show that this threshold was a reasonable choice.
This convergence criterion is usually met in less than 100 iterations, each of them taking 5 minutes on a modest desktop computer (Intel Core i5-4670 CPU with 8GiB of RAM), including the induction of a dictionary of 200,000 words at each iteration.
4.2 Bilingual lexicon induction
For the experiments on bilingual lexicon induction, we compared our method with those proposed by Mikolov et al. (2013a), Xing et al. (2015), Zhang et al. (2016) and Artetxe et al. (2016)7. The results obtained with the 5,000 en-
6http://lcl.uniroma1.it/ similarity-datasets/
7We used the publicly available implementation at https://github.com/artetxem/vecmap
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
● ●●
●● ●
● ●
0
10
20
30
40
0 1000 2000 3000 4000 5000 Seed dictionary size
A cc
ur ac
y (%
)
Method ● Our method
Artetxe et al. (2016)
Xing et al. (2015)
Zhang et al. (2016)
Mikolov et al. (2013a)
Figure 2: Accuracy on English-Italian bilingual lexicon induction for different seed dictionaries
try, 25 entry and the numerals dictionaries for all the 3 language pairs are given in Table 1.
The results for the 5,000 entry dictionaries show that our method is comparable or even better than the other systems. As another reference, the best published results for the full English-Italian dictionary are due to Lazaridou et al. (2015)8, who report an accuracy of 40.20%, almost at pair with our system (39.67%).
In any case, the main focus of our work is on smaller dictionaries, and it is under this setting that our method really stands out. The 25 entry and numerals columns in Table 1 show the results for this setting, where all previous methods drop dramatically, falling below 1% accuracy in all cases. The method by Zhang et al. (2016) also obtains poor results with small dictionaries, which reinforces our hypothesis in Section 2.2 that their method can only capture coarse-grain bilingual relations for small dictionaries. In contrast, our proposed method obtains very competitive results for all dictionaries, with a difference of only 1-2 points between the full dictionary and both the 25 entry dictionary and the numerals dictionary in all three languages. Figure 2 shows the curve of the English-Italian accuracy for different seed dictionary sizes, confirming this trend.
Finally, it is worth mentioning that, even if all the three language pairs show the same general behavior, there are clear differences in their ab-
8The code for this method is not publicly available but, given that our best baseline system obtains similar results on English-Italian (39.27% vs 40.20%), we consider that the state-of-the-art is already well represented in our experiments.
solute accuracy numbers, which can be attributed to the linguistic proximity of the languages involved. In particular, the results for EnglishFinnish are about 10 points below the rest, which are explained by the fact that Finnish is a nonindoeuropean agglutinative language, making the task considerably more difficult for this language pair. In this regard, we believe that the good results with small dictionaries are a strong indication of the robustness of our method, showing that it is able to learn good bilingual mappings from very little bilingual evidence even for distant language pairs where the structural similarity of the embedding spaces is presumably weaker.
4.3 Crosslingual word similarity
In addition to the baseline systems in Section 4.2, in the crosslingual similarity experiments we also tested the method by Luong et al. (2015), which is the state-of-the-art for bilingual word embeddings based on parallel corpora (Upadhyay et al., 2016)9. As this method is an extension of word2vec, we used the same hyperparameters as for the monolingual embeddings when possible (see Section 4.1), and leave the default ones otherwise. We used Europarl as our parallel corpus to train this method as done by the authors, which consists of nearly 2 million parallel sentences.
As shown in the results in Table 2, our method obtains the best results in all cases, surpassing the rest of the dictionary-based methods by 1-3 points depending on the dataset. But, most importantly, it does not suffer from any significant degradation for using smaller dictionaries and, in fact, our method gets better results using the 25 entry dictionary or the numeral list as the only bilingual evidence than any of the baseline systems using much richer resources.
The relatively poor results of Luong et al. (2015) can be attributed to the fact that the dictionary based methods make use of much bigger monolingual corpora, while methods based on parallel corpora are restricted to smaller corpora. However, it is not clear how to introduce monolingual corpora on those methods. We did run some experiments with BilBOWA (Gouws et al., 2015), which supports training in monolingual corpora in
9We also tested English-German pre-trained embeddings from Klementiev et al. (2012) and Chandar A P et al. (2014). They both had coverage problems that made the results hard to compare, and, when considering the correlations for the word pairs in their vocabulary, their performance was poor.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
IT DE Bi. data WS RG WS
Luong et al. (2015) Europarl .331 .335 .356 Mikolov et al. (2013a) 5k dict .627 .643 .528 Xing et al. (2015) 5k dict .614 .700 .595 Zhang et al. (2016) 5k dict .61.6 .704 .596 Artetxe et al. (2016) 5k dict .617 .716 .597 Our method 5k dict .624 .742 .616 25 dict .626 .749 .612 num. .628 .739 .604
Table 2: Spearman correlations on English-Italian and English-German crosslingual word similarity
addition to bilingual corpora, but obtained very poor results10. All in all, our experiments show that it is better to use large monolingual corpora in combination with very little bilingual data rather than a bilingual corpus of a standard size alone.
5 Global optimization objective
It might seem somehow surprising at first that, as seen in the previous section, our simple selflearning approach is able to learn high quality bilingual embeddings from small seed dictionaries instead of falling in degenerated solutions. In this section, we try to shed light on our approach, and give empirical evidence supporting our claim.
More concretely, we argue that, for the embedding mapping and dictionary induction methods described in Section 3, the proposed selflearning framework is implicitly solving the following global optimization problem11:
W ∗ = arg max W ∑ i max j (Xi∗W ) · Zj∗
s.t. WW T = W TW = I
Contrary to the optimization objective for W in Section 3.1, the global optimization objective does not refer to any dictionary, and maximizes the similarity between each source language word and its closest target language word. Intuitively, a random solution would map source language embeddings to seemingly random locations in the target language space, and it would thus be unlikely that they have any target language word nearby, making the optimization value small. In contrast, a good solution would map source language words
10Upadhyay et al. (2016) report similar problems using BilBOWA.
11While we restrict our formal analysis to the embedding mapping and dictionary induction method that we use, the general reasoning should be valid for other choices as well.
close to their translation equivalents in the target language space, and they would thus have their corresponding embeddings nearby, making the optimization value large. While it is certainly possible to build degenerated solutions that take high optimization values for small subsets of the vocabulary, we think that the structural similarity between independently trained embedding spaces in different languages is strong enough that optimizing this function yields to meaningful bilingual mappings when the size of the vocabulary is much larger than the dimensionality of the embeddings.
The reasoning for how the self-learning framework is optimizing this objective is as follows. At the end of each iteration, the dictionary D is updated to assign, for the current mapping W , each source language word to its closest target language word. This way, when we update W to maximize the average similarity of these dictionary entries at the beginning of the next iteration, it is guaranteed that the value of the optimization objective will improve (or at least remain the same). The reason is that the average similarity between each word and what were previously the closest word will be improved if possible, as this is what the updated W directly optimizes (see Section 3.1). In addition to that, it is also possible that, for some source words, some other target words get closer after the update. Thanks to this, our self-learning algorithm is guaranteed to converge to a local optimum of the above global objective, behaving like an alternating optimization algorithm for it.
It is interesting to note that the above reasoning is valid no matter what the the initial solution is, and, in fact, the optimization objective does not depend on the seed dictionary nor any other bilingual resource. For that reason, it should be possible to use a random initialization instead of a small seed dictionary. However, we empirically observe that this works poorly in practice, as our algorithm tends to get stuck in poor local optima when the initial solution is not good enough.
The general behavior of our method is reflected in Figure 3, which shows the learning curve for different seed dictionaries according to both the objective function and the accuracy on bilingual lexicon induction. As it can be seen, the objective function is improved from iteration to iteration and converges to a local optimum just as expected. At the same time, the learning curves show a strong correlation between the optimization objective and
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0.25
0.30
0.35
0.40
0.45
10 20 30 40 Iteration
O bj
ec tiv
e fu
nc tio
n
Seed dict.
5,000
2,500
1,000
500
250
100
75
50
25
num.
none
0
10
20
30
40
10 20 30 40 Iteration
A cc
ur ac
y (%
)
Seed dict.
5,000
2,500
1,000
500
250
100
75
50
25
num.
none
Figure 3: Learning curve on English-Italian according to the global objective function (left) and the accuracy on bilingual lexicon induction (right)
the accuracy, as it can be clearly observed that improving the former leads to an improvement of the latter, confirming our explanations. Regarding random initialization, the figure shows that the algorithm gets stuck in a poor local optimum of the objective function, which is the reason of the bad performance (0% accuracy) on bilingual lexicon induction, but the proposed optimization objective itself seems to be adequate.
Finally, we empirically observe that our algorithm learns similar mappings no matter what the seed dictionary was. We first repeated our experiments on English-Italian bilingual lexicon induction for 5 different dictionaries of 25 entries, obtaining an average accuracy of 38.15% and a standard deviation of only 0.75%. In addition to that, we observe that the overlap between the predictions made when starting with the full dictionary and the numerals dictionary is 76.00% (60.00% for the 25 entry dictionary). At the same time, 37.00% of the test cases are correctly solved by both instances, and it is only 5.07% of the test cases that one of them gets right and the other wrong (34.00% and 8.94% for the 25 entry dictionary). This clearly shows that these different systems do not only obtain a similar accuracy, but also have a very similar behavior. In other words, our algorithm tends to converge to similar solutions even for disjoint seed dictionaries, which is in line with our view that we are implicitly optimizing an objective that is independent from the seed dictionary, yet a seed dictionary is necessary to build a good enough initial solution to avoid getting stuck in poor local optima. For that reason, it is likely that better methods to tackle this opti-
mization problem would allow learning bilingual word embeddings without any bilingual evidence at all and, in this regard, we believe that our work opens exciting opportunities for future research.
6 Conclusions and future work
In this work, we propose a simple self learning framework to learn bilingual word embedding mappings in combination with any embedding mapping and dictionary induction technique. Our experiments on bilingual lexicon induction and crosslingual word similarity show that our method is able to learn high quality bilingual embeddings from as little bilingual evidence as a 25 word dictionary or an automatically generated list of numerals, obtaining results that are competitive with state-of-the-art systems using much richer bilingual resources like larger dictionaries or parallel corpora. In spite of its simplicity, a more detailed analysis shows that our method is implicitly optimizing a meaningful objective function that is independent from any bilingual data which, with a better optimization method, might allow to learn bilingual word embeddings in a completely unsupervised manner.
In the future, we would like to delve deeper into this direction and fine-tune our method so it can reliably learn high quality bilingual word embeddings without any bilingual evidence at all. In addition to that, we would like to explore non-linear transformations (Lu et al., 2015) and alternative dictionary induction methods (Dinu et al., 2015). Finally, we would like to apply our model in the decipherment scenario (Dou et al., 2015).
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

This work proposes a self-learning bootstrapping approach to learning bilingual word embeddings, which achieves competitive results in tasks of bilingual lexicon induction and cross-lingual word similarity although it requires a minimal amount of bilingual supervision: the method leads to competitive performance even when the seed dictionary is extremely small (25 dictionary items!) or is constructed without any language pair specific information (e.g., relying on numerals shared between languages).   The paper is very well-written, admirably even so. I find this work 'eclectic' in a sense that its original contribution is not a breakthrough finding (it is more a 'short paper idea' in my opinion), but it connects the dots from prior work drawing inspiration and modelling components from a variety of previous papers on the subject, including the pre-embedding work on self-learning bootstrapping (which is not fully recognized in the current version of the paper). I liked the paper in general, but there are few other research questions that could should have been pursued in this work. These, along with only a partial recognition of related work and a lack of comparisons with several other relevant baselines, are my main concern regarding this paper, and they should be fixed in the updated version(s).  *Self-learning bootstrapping of bilingual vector spaces: While this work is one of the first to tackle this very limited setup for learning cross-lingual embeddings (although not the first one, see Miceli Barone and more works below), this is the first truly bootstrapping self-learning approach to learning cross-lingual embeddings. However, this idea of bootstrapping bilingual vector spaces is not new at all (it is just reapplied to learning embeddings), and there is a body of work which used exactly the same idea with traditional 'count-based' bilingual vector spaces. I suggest the authors to check the work of Peirsman and Pado (NAACL 2010) or Vulic and Moens (EMNLP 2013), and recognize the fact that their proposed bootstrapping approach is not so novel in this domain. There is also related work of Ellen Riloff's group on bootstrapping semantic lexicons in monolingual settings.  *Relation to Artetxe et al.: I might be missing something here, but it seems that the proposed bootstrapping algorithm is in fact only an iterative approach which repeatedly utilises the previously proposed model formulation of Artetxe et al. The only difference is the reparametrization (line 296-305). It is not clear to me whether the bootstrapping approach draws its performance from this reparametrization (and whether it would work with the previous parametrization), or the performance is a product of both the algorithm and this new parametrization. Perhaps a more explicit statement in the text is needed to fully understand what is going on here.  *Comparison with prior work: Several very relevant papers have not been mentioned nor discussed in the current version of the paper. For instance, the recent work of Duong et al. (EMNLP 2016) on 'learning crosslingual word embeddings without bilingual corpora' seems very related to this work (as the basic word overlap between the two titles reveals!), and should be at least discussed if not compared to. Another work which also relies on mappings with seed lexicons and also partially analyzes the setting with only a few hundred seed lexicon pairs is the work of Vulic and Korhonen (ACL 2016) 'on the role of seed lexicons in learning bilingual word embeddings': these two papers might also help the authors to provide more details for the future work section (e.g., the selection of reliable translation pairs might boost the performance further during the iterative process). Another very relevant work has appeared only recently: Smith et al. (ICLR 2017) discuss 'offline bilingual word vectors, orthogonal transformations and the inverted softmax'. This paper also discusses learning bilingual embeddings in very limited settings (e.g., by relying only on shared words and cognates between two languages in a pair). As a side note, it would be interesting to report results obtained using only shared words between the languages (such words definitely exist for all three language pairs used in the experiments). This would also enable a direct comparison with the work of Smith et al. (ICLR 2017) which rely on this setup.  *Seed dictionary size and bilingual lexicon induction: It seems that the proposed algorithm (as discussed in Section 5) is almost invariant to the starting seed lexicon, yielding very similar final BLI scores regardless of the starting point. While a very intriguing finding per se, this also seems to suggest an utter limitation of the current 'offline' approaches: they seem to have hit the ceiling with the setup discussed in the paper; Vulic and Korhonen (ACL 2016) showed that we cannot really improve the results by simply collecting more seed lexicon pairs, and this work suggests that any number of starting pairs (from 25 to 5k) is good enough to reach this near-optimal performance, which is also very similar to the numbers reported by Dinu et al. (arXiv 2015) or Lazaridou et al. (ACL 2015). I would like to see more discussion on how to break this ceiling and further improve BLI results with such 'offline' methods. Smith et al. (ICLR 2017) seem to report higher numbers on the same dataset, so again it would be very interesting to link this work to the work of Smith et al.  In other words, the authors state that in future work they plan to fine-tune the method so that it can learn without any bilingual evidence. This is an admirable 'philosophically-driven' feat, but from a more pragmatic point of view, it seems more pragmatic to detect how we can go over the plateau ceiling which seems to be hit with these linear mapping approaches regardless of the number of used seed lexicon pairs (Figure 2).  *Convergence criterion training efficiency: The convergence criterion, although crucial for the entire algorithm, both in terms of efficiency and efficacy, is mentioned only as a side note, and it is not entirely clear how the whole procedure terminates. I suspect that the authors use the vanishing variation in crosslingual word similarity performance as the criterion to stop the procedure, but that makes the method applicable only to languages which have a cross-lingual word similarity dataset. I might be missing here given the current description in the paper, but I do not fully understand how the procedure stops for Finnish, given that there is no crosslingual word similarity dataset for English-Finnish.  *Minor: - There is a Finnish 'Web as a Corpus' (WaC) corpus (lines 414-416): https:  www.clarin.si repository xmlui handle 11356 1074 - Since the authors claim that the method could work with a seed dictionary containing only shared numerals, it would be very interesting to include an additional language pair which does not share the alphabet (e.g., English-Russian, English-Bulgarian or even something more distant such as Arabic and or Hindi).  *After the response: I would like to thank the authors for investing their time into their response which helped me clarify some doubts and points raised in my initial review. I hope that they would indeed clarify these points in the final version, if given the opportunity.