One-Shot Neural Cross-Lingual Transfer for Paradigm Completion
We present a novel cross-lingual transfer method for paradigm completion, the task of mapping a lemma to its inflected forms, using a neural encoder-decoder model, the state of the art for the monolingual task. We use labeled data from a high-resource language to increase performance on a lowresource language. In experiments on 21 language pairs from four different language families, we obtain up to 58% higher accuracy than without transfer and show that even zero-shot and one-shot learning are possible. We further find that the degree of language relatedness strongly influences the ability to transfer morphological knowledge.
1 Introduction
Low-resource natural language processing remains an open problem for many tasks of interest. Furthermore, for most languages in the world, high-cost linguistic annotation and resource creation are unlikely to be undertaken in the near future. In the case of morphology, out of the 7000 currently spoken (Lewis, 2009) languages, only about 200 have computer-readable annotations (Sylak-Glassman et al., 2015) – although morphology is easy to annotate compared to syntax and semantics. Transfer learning is one solution to this problem: it exploits annotations in a high-resource language to train a system for a low-resource language. In this work, we present a method for cross-lingual transfer of inflectional morphology using an encoder-decoder recurrent neural network (RNN). This allows for the development of tools for computational morphology with limited annotated data.
In morphologically rich languages, individual lexical entries may be realized as distinct inflec-
tions of a single lemma depending on the syntactic context. For example, the 3SgPresInd of the English verbal lemma to bring is brings. In many languages, a lemma can have hundreds of individual forms. Thus, both generation and analysis of such morphological inflections are active areas of research in NLP and morphological processing has been shown to be a boon to several other down-stream applications, e.g., machine translation (Dyer et al., 2008), speech recognition (Creutz et al., 2007), parsing (Seeker and Çetinoğlu, 2015), keyword spotting (Narasimhan et al., 2014) and word embeddings (Cotterell et al., 2016b), inter alia. In this work we focus on paradigm completion, a form of morphological generation that maps a given lemma to a target inflection, e.g., (bring, Past) 7→ brought (with Past being the target tag).
RNN sequence-to-sequence models (Sutskever et al., 2014; Bahdanau et al., 2015) are the state of the art for paradigm completion (Faruqui et al., 2016; Kann and Schütze, 2016a; Cotterell et al., 2016a). However, these models require a large amount of data to achieve competitive performance; this makes them unsuitable for out-of-thebox application to paradigm completion in the low-resource scenario. To mitigate this, we consider transfer learning: we train an end-to-end neural system jointly with limited data from a lowresource language and a larger amount of data from a high-resource language. This technique allows the model to apply knowledge distilled from the high-resource training data to the low-resource language as needed.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
We conduct experiments on 21 language pairs from four language families, emulating a lowresource setting. Our results demonstrate successful transfer of morphological knowledge. We show improvements in accuracy and edit distance of up to 58% (accuracy) and 4.62 (edit distance) over the same model with only in-domain language data on the paradigm completion task. We further obtain up to 44% (resp. 14%) improvement in accuracy for the one-shot (resp. zero-shot) setting, i.e., one (resp. zero) in-domain language sample per target tag. We also show that the effectiveness of morphological transfer depends on language relatedness, measured by lexical similarity.
2 Inflectional Morphology and Paradigm Completion
Many languages exhibit inflectional morphology, i.e., the form of an individual lexical entry mutates to show properties such as person, number or case. The citation form of a lexical entry is referred to as the lemma and the collection of its possible inflections as its paradigm. Tab. 1 shows an example of a partial paradigm; we display several forms for the Spanish verbal lemma soñar. We may index the entries of a paradigm by a morphological tag, e.g., the 2SgPresInd form sueñas in Tab. 1. In generation, the speaker must select an entry of the paradigm given the form’s context. In general, the presence of rich inflectional morphology is problematic for NLP systems as it greatly increases the token-type ratio and, thus, word form sparsity.
An important task in inflectional morphology is paradigm completion (Durrett and DeNero, 2013; Ahlberg et al., 2014; Nicolai et al., 2015; Cotterell et al., 2015; Faruqui et al., 2016). Its goal is to map a lemma to all individual inflections, e.g., (soñar, 1SgPresInd) 7→ sueño. There are good solutions for paradigm completion when a large amount of annotated training data is available (Cotterell et al., 2016a).1 In this work, we address the lowresource setting, an up to now unsolved challenge.
2.1 Transferring Inflectional Morphology
In comparison to other NLP annotations, e.g., partof-speech (POS) and named entities, morphological inflection does not lend itself easily to transfer.
1The SIGMORPHON 2016 shared task (Cotterell et al., 2016a) on morphological reinflection, a harder generalization of paradigm completion, found that ≥ 98% accuracy can be achieved in many languages with neural sequence-to-sequence models, improving the state of the art by 10%.
We can define a universal set of POS tags (Petrov et al., 2012) or of entity types (e.g., coarse-grained types like person and location or fine-grained types (Yaghoobzadeh and Schütze, 2015)), but inflection is much more language-specific. It is infeasible to transfer morphological knowledge from Chinese to Portuguese as Chinese does not use inflected word forms. Transferring named entity recognition, however, among Chinese and European languages works well (Wang and Manning, 2014a). But even transferring inflectional paradigms from morphologically rich Arabic to Portuguese seems difficult as the inflections often mark dissimilar subcategories. In contrast, transferring morphological knowledge from Spanish to Portuguese, two languages with similar conjugations and 89% lexical similarity, appears promising. Thus, we conjecture that transfer of inflectional morphology is only viable among related languages.
2.2 Formalization of the Task
We now offer a formal treatment of the crosslingual paradigm completion task and develop our notation. Let Σ` be a discrete alphabet for language ` and let T` be a set of morphological tags for `. Given a lemma w` in `, the morphological paradigm (inflectional table) π can be formalized as a set of pairs
π(w`) = {( fk[w`], tk )} k∈T (w`)
(1)
where fk[w`] ∈ Σ+` is an inflected form, tk ∈ T` is its morphological tag and T (w`) is the set of slots in the paradigm; e.g., a Spanish paradigm is:
π(SOÑAR)= {( sueño, 1SgPresInd ) , . . . , ( soñaran, 3PlPastSbj )}
Paradigm completion consists of predicting the entire paradigm π(w`) given the lemma w`.
In cross-lingual paradigm completion, we consider a high-resource source language `s (lots of training data available) and a low-resource target language `t (little training data available). We denote the source training examples as Ds (with |Ds| = ns) and the target training examples as Dt (with |Dt| = nt). The goal of cross-lingual paradigm completion is to populate paradigms in the low-resource target language with the help of data from the high-resource source language, using only few in-domain examples.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
3 Cross-Lingual Transfer as Multi-Task Learning
We describe our probability model for morphological transfer using terminology from multi-task learning (Caruana, 1997; Collobert et al., 2011). We consider two tasks, training a paradigm completor (i) for a high-resource language and (ii) for a low-resource language. We want to train jointly so we reap the benefits of having related languages. Thus, we define the log-likelihood as
L(θ)= ∑
(k,w`t )∈Dt
log pθ (fk[w`t ] | w`t , tk) (2)
+ ∑
(k,w`s )∈Ds
log pθ(fk[w`s ] | w`s , tk)
where we tie parameters θ for the two languages together to allow the transfer of morphological knowledge between languages. Each probability distribution pθ defines a distribution over all possible realizations of an inflected form, i.e., a distribution over Σ∗. For example, consider the related Romance languages Spanish and French; focusing on one term from each of the summands in Eq. (2) (the past participle of the translation of to visit in each language), we arrive at
Lvisit(θ) = log pθ(visitado | visitar, PastPart) + log pθ(visité | visiter, PastPart) (3)
Our cross-lingual setting forces both transductions to share part of the parameter vector θ, to represent morphological regularities between the two languages in a common embedding space and, thus, to enable morphological transfer. This is no different from monolingual multi-task settings, e.g., jointly training a parser and tagger for transfer of syntax.
Based on recent advances in neural transducers, we parameterize each distribution as an encoderdecoder RNN, as in (Kann and Schütze, 2016b). In their setup, the RNN encodes the input and predicts the forms in a single language. In contrast, we force the network to predict two languages.
3.1 Encoder-Decoder RNN
We parameterize the distribution pθ as an encoderdecoder gated RNN with attention (Bahdanau et al., 2015), the state-of-the-art solution for the monolingual case (Kann and Schütze, 2016b). A bidirectional gated RNN encodes the input sequence (Cho et al., 2014) – the concatenation of (i) the language
! h1
! h2
! h3
! hN
h1
h2
h3
hN
s o ñ r
s u e s1 s2 s3 sN
y1= y2= y3=M
…
Figure 1: Encoder-decoder RNN for paradigm completion. The lemma soñar is mapped to a target form (e.g., sueña). For brevity, language and target tags are omitted from the input. Thickness of red arrows symbolizes the degree to which the model attends to the corresponding hidden state of the encoder.
tag, (ii) the morphological tag of the form to be generated and (iii) the characters of the input word – represented by embeddings. The input to the decoder consists of concatenations of −→ hi and ←− hi , the forward and backward hidden states of the encoder. The decoder, a unidirectional RNN, uses attention: it computes a weight for each hi. Each weight reflects the importance given to that input position. Using the attention weights αij , the probability of the output sequence given the input sequence is:
p(y | x1, . . . , x|X|) = |Y |∏
t=1
g(yt−1, st, ct) (4)
where y = (y1, . . . , y|Y |) is the output sequence (a sequence of |Y | characters), x = (x1, . . . x|X|) is the input sequence (a sequence of |X| characters), g is a non-linear function, st is the hidden state of the decoder and ct is the sum of the encoder states hi, weighted by attention weights αi(st−1) which depend on the decoder state:
ct =
|X|∑
i=1
αi(st−1)hi (5)
Fig. 1 shows the encoder-decoder. See Bahdanau et al. (2015) for further details.
3.2 Input Format
Each source form is represented as a sequence of characters; each character is represented as an embedding. In the same way, each source tag is represented as a sequence of subtags, and each subtag is represented as an embedding. More formally,
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
we define the alphabet Σ = ∪`∈LΣ` as the set of characters in the languages in L, with L being the set of languages in the given experiment. Next, we define S as the set of subtags that occur as part of the set of morphological tags T = ∪`∈LT`; e.g., if 1SgPresInd ∈ T , then 1, Sg, Pres, Ind ∈ S . Note that the set of subtags S is defined as attributes from the UNIMORPH schema (Sylak-Glassman, 2016) and, thus, is universal across languages; the schema is derived from research in linguistic typology.2 The format of the input to our system is S+Σ+. The output format is Σ+. Both input and output are padded with distinguished BOW and EOW symbols.
What we have described is the representation of Kann and Schütze (2016b). In addition, we preprend a symbol λ ∈ L to the input string (e.g., λ = Es, also represented by an embedding), so the RNN can handle multiple languages simultaneously and generalize over them.
4 Languages and Language Families
To verify the applicability of our method to a wide range of languages, we perform experiments on example languages from several different families.
Romance languages, a subfamily of IndoEuropean, are widely spoken, e.g., in Europe and Latin America. Derived from the common ancestor Vulgar Latin (Harris and Vincent, 2003), they share large parts of their lexicon and inflectional morphology; we expect knowledge among them to be easily transferable.
We experiment on Catalan, French, Italian, Portuguese and Spanish. Tab. 2 shows that Spanish – which takes the role of the low-resource language in our experiments – is closely related with the other four, with Portuguese being most similar. We hypothesize that the transferability of morphological knowledge between source and target corresponds to the degree of lexical similarity; thus, we expect Portuguese and Catalan to be more beneficial for Spanish than Italian and French.
The Indo-European Slavic language family has its origin in eastern-central Europe (Corbett and Comrie, 2003). We experiment on Bulgarian, Macedonian, Russian and Ukrainian (Cyrillic script) and on Czech, Polish and Slovene (Latin script). Macedonian and Ukranian are low-resource
2Note that while the subtag set is universal, which subtags a language actually uses is language-specific; e.g., Spanish does not mark animacy as Russian does. We contrast this with the universal POS set (Petrov et al., 2012), where it is reasonable to expect that we see all 17 tags in every language.
languages, so we assign them the low-resource role. For Romance and for Uralic, we experiment with groups containing three or four source languages. To arrive at a comparable experimental setup for Slavic, we run two experiments, each with three source and one target language: (i) from Russian, Bulgarian and Czech to Macedonian; and (ii) from Russian, Polish and Slovene to Ukrainian.
We hope that the paradigm completor learns similar embeddings for, say, the characters “e” in Polish and “ ” in Ukrainian. Thus, the use of two scripts in Slavic allows us to explore transfer across different alphabets.
We further consider a non-Indo-European language family, the Uralic languages. We experiment on the three most commonly spoken languages – Finnish, Estonian and Hungarian (Abondolo, 2015) – as well as Northern Sami, a language used in Northern Scandinavia. While Finnish and Estonian are closely related (both are members of the Finnic subfamily), Hungarian is a more distant cousin. Estonian and Northern Sami are lowresource languages, so we assign them the lowresource role, resulting in two groups of experiments: (i) Finnish, Hungarian and Estonian to Northern Sami; (ii) Finnish, Hungarian and Northern Sami to Estonian.
Arabic (baseline) is a Semitic language (part of the Afro-Asiatic family (Hetzron, 2013)) that is spoken in North Africa, the Arabian Peninsula and other parts of the Middle East. It is unrelated to all other languages used in this work. Both in terms of form (new words are mainly built using a templatic system) and categories (it has tags such as construct state), Arabic is very different. Thus, we do not expect it to support morphological knowledge transfer and we use it as a baseline for all target languages.
5 Experiments
We run three experiments on 21 distinct pairings of languages to show the feasibility of morphological transfer and analyze our method. We first discuss details common to all experiments.
We keep hyperparameters during all experiments (and for all languages) fixed to the following values. Encoder and decoder RNNs each have 100
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
50·20 50·21 50·22 50·23 50·24 50·25 50·26 50·27 Number of Samples
0.0
0.2
0.4
0.6
0.8 1.0 A cc ur ac y
Languages Pt Ca It Fr Ar Es
Figure 2: Learning curves showing the accuracy on Spanish test when training on language λ ∈ {PT, CA, IT, FR, AR, ES}. Except for λ=ES, each model is trained on 12,000 samples from λ and “Number of Samples” (x-axis) of Spanish.
hidden units and the size of all subtag, character and language embeddings is 300. For training we use ADADELTA (Zeiler, 2012) with minibatch size 20. All models are trained for 300 epochs. Following Le et al. (2015), we initialize all weights in the encoder, decoder and the embeddings except for the GRU weights in the decoder to the identity matrix. Biases are initialized to zero.
Evaluation metrics: (i) 1-best accuracy: the percentage of predictions that match the true answer exactly; (ii) average edit distance between prediction and true answer. The two metrics differ in that accuracy gives no partial credit and incorrect answers may be drastically different from the annotated form without incurring additional penalty. In contrast, edit distance gives partial credit for forms that are closer to the true answer.
5.1 Exp. 1: Transfer Learning for Paradigm Completion
In this experiment, we investigate to what extent our model transfers morphological knowledge from a high-resource source language to a low-resource target language. We experimentally answer three questions. (i) Is transfer learning possible for morphology? (ii) How much annotated data do we need in the low-resource target language? (iii) How closely related must the two languages be to achieve good results?
Data. Based on complete inflection tables from unimorph.org (Kirov et al., 2016), we create datasets as follows. Each training set consists of 12,000 samples in the high-resource source language and nt∈{50, 200} samples in the lowresource target language. We create target lan-
guage dev and test sets of sizes 1600 and 10,000, respectively.3 For Romance and Arabic, we create learning curves for nt∈{100, 400, 800, 1600, 3200, 6400, 12000}. Lemmata and inflections are randomly selected from all available paradigms.
Results and Discussion. Tab. 3 shows the effectiveness of transfer learning. There are two baselines. (i) “0”: no transfer, i.e., we consider only in-domain data; (ii) “AR”: Arabic, which is unrelated to all target languages.
With the exception of the 200 sample case of ET→SME, cross-lingual transfer is always better than the two baselines; the maximum improvement is 0.58 (0.58 vs. 0.00) in accuracy for the 50 sample case of CA→ES. More closely related source languages improve performance more than distant ones. French, the Romance language least similar to Spanish, performs worst for →ES. For the target language Macedonian, Bulgarian provides most benefit. This can again be explained by similarity: Bulgarian is closer to Macedonian than the other languages in this group. The best result for Ukrainian is RU→UK. Unlike Polish and Slowenian, Russian is the only language in this group that uses the same script as Ukrainian, showing the importance of the alphabet for transfer. Still, the results also demonstrate that transfer works across alphabets (although not as well); this suggests that similar embeddings for similar characters have been learned. Finnish is the language that is closest to Estonian and it again performs best as a source language for Estonian. For Northern Sami, transfer works least well, probably because the distance between sources and target is largest in this case. The distance of the Sami languages from the Finnic (Estonian, Finnish) and Ugric (Hungarian) languages is much larger than the distances within Romance and within Slavic.4 However, even for Northern Sami, adding an additional language is still always beneficial compared to the monolingual baseline.
Learning curves for Romance and Arabic further support our finding that language similarity is important. In Fig. 2, knowledge is transferred to Spanish, and a baseline – a model trained only on Spanish data – shows the accuracy obtained without any transfer learning. Here, Catalan and Italian help the most, followed by Portuguese, French and,
3For Estonian, we use 7094 (not 12,000) train and 5000 (not 10,000) test samples as more data is unavailable.
4We have enlisted the expert for Uralic at our university and are in the process of analyzing SME results in more detail.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Romance Slavic I Slavic II Uralic I Uralic II source 0 AR PT CA IT FR 0 AR RU BG CS 0 AR RU PL SL 0 AR FI HU ET 0 AR FI HU SME target →ES →MK →UK →SME →ET
5 0 acc 0.00 0.04 0.48 0.58 0.46 0.29 0.00 0.00 0.23 0.47 0.13 0.01 0.01 0.47 0.16 0.07 0.00 0.00 0.03 0.01 0.01 0.02 0.01 0.35 0.21 0.17
ED 5.42 4.06 0.85 0.80 1.15 1.82 5.71 5.59 1.61 0.87 2.32 5.23 4.80 0.77 2.14 3.12 8.47 8.64 5.41 6.53 7.03 4.50 4.51 1.55 2.19 2.60 2 0 0 acc 0.38 0.54 0.62 0.78 0.74 0.60 0.21 0.40 0.62 0.77 0.57 0.16 0.21 0.64 0.55 0.50 0.05 0.09 0.20 0.18 0.06 0.34 0.53 0.74 0.71 0.66 ED 1.37 0.87 0.57 0.78 0.44 0.82 1.93 1.12 0.68 0.36 0.72 2.09 1.60 0.49 0.73 0.82 5.43 4.93 3.20 3.37 5.01 1.47 0.98 0.41 0.48 0.62
Table 3: Accuracy (acc) and edit distance (ED) of cross-lingual transfer learning for paradigm completion. The target language is indicated by “→”, e.g., it is Spanish for “→ES”. Sources are indicated in the row “source”; “0” is the monolingual case. Except for Estonian, we train on ns = 12,000 source samples and nt ∈ {50, 200} target samples (as indicated by the row). There are two baselines in the table. (i) “0”: no transfer, i.e., we consider only in-domain data; (ii) “AR”: the Semitic language Arabic is unrelated to all target languages and functions as a dummy language that is unlikely to provide relevant information. All languages are denoted using the official codes (SME=Northern Sami).
finally, Arabic. This corresponds to the order of lexical similarity with Spanish, except for the performance of Portuguese (cf. Tab. 2). A possible explanation is the potentially confusing overlap of lemmata between the two languages – cf. discussion in the next subsection. That the transfer learning setup improves performance for the unrelated language Arabic as source is at first surprising. But adding new samples to a small training set helps prevent overfitting (e.g., rote memorization) even if the source is a morphologically unrelated language; effectively acting as a regularizer.5 This will also be discussed below.
Error Analysis for Romance. Even for only 50 Spanish instances, many inflections are correctly produced in transfer. For, e.g., (criar, 3PlFutSbj) 7→ criaren, model outputs are: fr: criaren, ca: criaren, es: crntaron, it: criaren, ar: ecriren, pt: criaren (all correct except for the two baselines). Many errors involve accents, e.g., (contrastar, 2PlFutInd) 7→ contrastaréis; model outputs are: fr: contrastareis, ca: contrastareis, es: conterarı́an, it: contrastareis, ar: contastarı́as, pt: contrastareis. Some inflections all systems get wrong, mainly because of erroneously applying the inflectional rules of the source to the target. Finally, the output of the model trained on Portuguese contains a class of errors that are unlike those of other systems. Example: (contraatacar, 1SgCond) 7→ contraatacarı́a with those solutions: fr: contratacarı́am, ca: contraatacarı́a, es: concarnar, it: contratacé, ar: cuntatarı́a and pt: contra-atacarı́a. The Portuguese model inserts “-” because Portuguese train data contains contraatacar and “-” appears in its inflected form.6
5Following (Kann and Schütze, 2016b) we did not use standard regularizers. To verify that the effect of Arabic is a regularization effect, we ran a small monolingual experiment on ES (200 setting) with dropout 0.5 (Srivastava et al., 2014). The resulting accuracy is 0.57, very similar to the comparable Arabic number of 0.54 in the table.
6To investigate this in more detail we retrain the Portuguese model with 50 Spanish samples, but exclude all lemmata
0 PT CA IT FR AR →ES
on e sh ot acc 0.00 0.44 0.39 0.23 0.13 0.00 ED 6.26 1.01 1.27 1.83 2.87 7.00
ze ro sh ot acc 0.00 0.14 0.08 0.01 0.02 0.00 ED 7.18 1.95 1.99 3.12 4.27 7.50
Table 4: Results for one-shot and zero-shot transfer learning. Formatting is the same as for Tab. 3. We still use ns = 12000 source samples. In the one-shot (resp. zero-shot) case, we observe exactly one form (resp. zero forms) for each tag in the target language at training time.
An example for the generally improved performance across languages for 200 Spanish training samples is (contrastar, 2PlIndFut) 7→ contrastaréis: all models now produce the correct form.
5.2 Exp. 2: Zero-Shot/One-Shot Transfer
In §5.1, we investigated the relationship between indomain (target) training set size and performance. Here, we look at the extreme case of training set sizes 1 (one-shot) and 0 (zero-shot) for a tag. We train our model on a single sample for half of the tags appearing in the low-resource language, i.e., if T` is the set of morphological tags for the target language, train set size is |T`|/2. As before, we add 12,000 source samples.
We report one-shot accuracy (resp. zero-shot accuracy), i.e., the accuracy for samples with a tag that has been seen once (resp. never) during training. Note that the model has seen the individual subtags each tag is composed of.7
Data. Our experimental setup is similar to §5.1: we use the same dev, test and high-resource train sets as before. However, the low-resource data is created in the way specified above. To remove a potentially confounding variable, we impose the condition that no two training samples belong to
that appear in Spanish train/dev/test, resulting in only 3695 training samples. Accuracy on test increases by 0.09 despite the reduced size of the training set.
7It is very unlikely that due to random selection a subtag will not be in train; this case did not occur in our experiments.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
the same lemma. Results and Discussion. Tab. 4 shows that the Spanish and Arabic systems do not learn anything useful for either half of the tags. This is not surprising as there is not enough Spanish data for the system to generalize well and Arabic does not contribute exploitable information. The systems trained on French and Italian, in contrast, get a nonzero accuracy for the zero-shot case as well as 0.13 and 0.23, respectively, in the one-shot case. This shows that a single training example is sometimes sufficient for successful generation although generalization to tags never observed is rarely possible. Catalan and Portuguese show the best performance in both settings; this is intuitive since they are the languages closest to the target (cf. Tab. 2). In fact, adding Portuguese to the training data yields an absolute increase in accuracy of 0.44 (0.44 vs. 0.00) for one-shot and 0.14 (0.14 vs. 0.00) for zero-shot with corresponding improvements in edit distance.
Overall, this experiment shows that with transfer learning from a closely related language the performance of zero-shot morphological generation improves over the monolingual approach, and, in the one-shot setting, it is possible to generate the right form nearly half the time.
5.3 Exp. 3: True Transfer vs. Other Effects
We would like to separate the effects of regularization that we saw for Arabic from true transfer.
To this end, we generate a random cipher (i.e., a function γ : Σ ∪ S 7→ Σ ∪ S) and apply it to all word forms and morphological tags of the high-resource train set; target language data are not changed. Ciphering makes it harder to learn true “linguistic” transfer of morphology. Consider the simplest case of transfer: an identical mapping in two languages, e.g., (visitar, 1SgPresInd) 7→ visito in both Portuguese and Spanish. If we transform Portuguese using the cipher γ(iostv...) = kltqa..., then visito becomes aktkql in Portuguese and its tag becomes similarly unrecognizable as being identical to the Spanish tag 1SgPresInd. Our intuition is that ciphering will disrupt transfer of morphology.8 On the other hand, the regularization effect we observed with Arabic should still be effective.
Data. We use the Portuguese-Spanish and 8Note that ciphered input is much harder than transfer between two alphabets (Latin/Cyrillic) because it creates ambiguous input. In the example, Spanish “i” is totally different from Portuguese “i” (which is really “k”), but the model must use the same representation.
Arabic-Spanish data from Experiment 1. We generate a random cipher and apply it to morphological tags and word forms for Portuguese and Arabic. The language tags are kept unchanged. Spanish is also not changed. For comparability with Tab. 3, we use the same dev and test sets as before.
Results and Discussion. Tab. 5 shows that performance of PT→ES drops a lot: from 0.48 to 0.09 for 50 samples and from 0.62 to 0.54 for 200 samples. This is because there are no overt similarities between the two languages left after applying the cipher, e.g., the two previously identical forms visito are now different.
The impact of ciphering on AR→ES varies: slightly improved in one case (0.54 vs. 0.56), slightly worse in three cases. We also apply the cipher to the tags and Arabic and Spanish share subtags, e.g., Sg. Just the knowledge that something is a subtag is helpful because subtags must not be generated as part of the output. We can explain the tendency of ciphering to decrease performance on AR→ES by the “masking” of common subtags.
For 200 samples and ciphering, there is no clear difference in performance between Portuguese and Arabic. However, for 50 samples and ciphering, Portuguese (0.09) seems to perform better than Arabic (0.02) in accuracy. Portuguese uses suffixation for inflection whereas Arabic is templatic and inflectional changes are not limited to the end of the word. This difference is not affected by ciphering. Perhaps even ciphered Portugese lets the model learn better that the beginnings of words just need to be copied. For 200 samples, the Spanish dataset may be large enough, so that ciphered Portuguese no longer helps in this regard.
Comparing no transfer with transfer from a ciphered language to Spanish, we see large performance gains, at least for the 200 sample case: 0.38 (0→ES) vs. 0.54 (PT→ES) and 0.56 (AR→ES). This is evidence that our conjecture is correct that the baseline Arabic mainly acts as a regularizer that prevents the model from memorizing the training set and therefore improves performance. So performance improves even though no true transfer of morphological knowledge takes place.
6 Related Work
Cross-lingual transfer learning has been used for many tasks: automatic speech recognition (Huang et al., 2013), parsing (Cohen et al., 2011; Søgaard, 2011; Naseem et al., 2012), entity recog-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0→ES PT→ES AR→ES orig ciph orig ciph
5 0 acc 0.00 0.48 0.09 0.04 0.02
ED 5.42 0.85 3.25 4.06 4.62 2 0 0 acc 0.38 0.62 0.54 0.54 0.56
ED 1.37 0.57 0.95 0.87 0.93
Table 5: Results for ciphering. “0→ES” and “orig” are original results, copied from Tab. 3; “ciph” is the result after the cipher has been applied.
nition (Wang and Manning, 2014b) and machine translation (Johnson et al., 2016; Ha et al., 2016). One straightforward method is to translate datasets and then train a monolingual model (Fortuna and Shawe-Taylor, 2005; Olsson et al., 2005). Also, aligned corpora have been used to project information from annotations in one language to another (Yarowsky et al., 2001; Padó and Lapata, 2005). The drawback is that machine translation errors cause errors in the target. Therefore, alternative methods have been proposed, e.g., to port a model trained on the source language to the target language (Shi et al., 2010).
In the realm of morphology, Buys and Botha (2016) recently adapted methods for the training of POS taggers to learn weakly supervised morphological taggers with the help of parallel text. Snyder and Barzilay (2008a, 2008b) developed a non-parametric Bayesian model for morphological segmentation. They performed identification of cross-lingual abstract morphemes and segmentation simultaneously and reported, similar to us, best results for related languages.
Work on paradigm completion has recently been encouraged by the SIGMORPHON 2016 shared task on morphological reinflection (Cotterell et al., 2016a). Some work first applies an unsupervised alignment model to source and target string pairs and then learns a string-to-string mapping (Durrett and DeNero, 2013; Nicolai et al., 2015), using, e.g., a semi-Markov conditional random field (Sarawagi and Cohen, 2004). Encoderdecoder RNNs (Aharoni et al., 2016; Faruqui et al., 2015; Kann and Schütze, 2016b), a method which our work further develops for the cross-lingual scenario, define the current state of the art.
Encoder-decoder RNNs were developed in parallel by Cho et al. (2014) and Sutskever et al. (2014) for machine translation and extended by Bahdanau et al. (2015) with an attention mechanism, supporting better generalization. They have been applied to NLP tasks like speech recognition (Graves
and Schmidhuber, 2005; Graves et al., 2013), parsing (Vinyals et al., 2015) and segmentation (Kann et al., 2016). More recently, a number of papers have used encoder-decoder RNNs in multitask and transfer learning settings; this is mainly work in machine translation (MT): (Dong et al., 2015; Zoph and Knight, 2016; Chu et al., 2017; Johnson et al., 2016; Luong et al., 2016; Firat et al., 2016; Ha et al., 2016), inter alia. Each of these papers has both similarities and differences with our approach. (i) Most train several distinct models whereas we train a single model on input augmented with an explicit encoding of the language (similar to (Johnson et al., 2016)). (ii) Let k and m be the number of different input and output languages. We address the case k ∈ {1, 2} and m = k. Other work has addressed cases with k > 2 or m > 2; this would be an interesting avenue of future research for paradigm completion. (iii) Whereas training RNNs in MT is hard, we only experienced one difficult issue in our experiments (due to the low-resource setting): regularization. (iv) Some work is word- or subword-based, our work is character-based. The same way that similar word embeddings are learned for the inputs cow and vache (French for “cow”) in MT, we expect similar embeddings to be learned for similar Cyrillic/Latin characters. (v) Similar to work in MT, we show that zero-shot (and, by extension, one-shot) learning is possible.
(Ha et al., 2016) (which was developed in parallel to our transfer model although we did not prepublish our paper on arxiv) is most similar to our work. Whereas Ha et al. (2016) address MT, we focus on the task of paradigm completion in low-resource settings and establish the state of the art for this problem.
7 Conclusion
We presented a cross-lingual transfer learning method for paradigm completion, based on an RNN encoder-decoder model. Our experiments showed that information from a high-resource language can be leveraged for paradigm completion in a related low-resource language. Our analysis showed that the degree to which the source language data helps for a certain target language depends on their relatedness. Our method led to significant improvements in settings with limited training data – up to 58% absolute improvement in accuracy – and, thus, enables the use of state-of-the-art models for paradigm completion in low-resource languages.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

The paper introduces a simple and effective method for morphological paradigm completion in low-resource settings. The method uses a character-based seq2seq model trained on a mix of examples in two languages: a resource-poor language and a closely-related resource-rich language; each training example is annotated with a paradigm properties and a language ID. Thus, the model enables transfer learning across languages when the two languages share common characters and common paradigms. While the proposed multi-lingual solution is not novel (similar architectures have been explored in syntax, language modeling, and MT), the novelty of this paper is to apply the approach to morphology. Experimental results show substantial improvements over monolingual baselines, and include a very thorough analysis of the impact of language similarities on the quality of results. The paper is interesting, very clearly written, I think it ll be a nice contribution to the conference program.   Detailed comments:     My main question is why the proposed general multilingual methodology was limited to pairs of languages, rather than to sets of similar languages? For example, all Romance languages could be included in the training to improve Spanish paradigm completion, and all Slavic languages with Cyrillic script could be mixed to improve Ukrainian. It would be interesting to see the extension of the models from bi-lingual to multilingual settings.     I think Arabic is not a fair (and fairly meaningless) baseline, given how different is its script and morphology from the target languages. A more interesting baseline would be, e.g., a language with a partially shared alphabet but a different typology. For example, a Slavic language with Latin script could be used as a baseline language for Romance languages. If Arabic is excluded, and if we consider a most distant language in the same the same family as a baseline, experimental results are still strong.     A half-page discussion of contribution of Arabic as a regularizer also adds little to the paper; I d just remove Arabic from all the experiments and would add a regularizer (which, according to footnote 5, works even better than adding Arabic as a transfer language).                  Related work is missing a line of work on  language-universal d RNN models that use basically the same approach: they learn shared parameters for inputs in multiple languages, and add a language tag to the input to mediate between languages. Related studies include a multilingual parser (Ammar et al., 2016), language models (Tsvetkov et al., 2016), and machine translation (Johnson et al., 2016 )  Minor:    I don t think that the claim is correct in line 144 that POS tags are easy to transfer across languages. Transfer of POS annotations is also a challenging task.    References:   Waleed              Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith.  Many languages, one parser. d TACL 2016.   Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W. Black, Lori Levin, and Chris Dyer.  Polyglot neural language models: A case study in cross-lingual phonetic representation learning. d NAACL 2016.  Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat et al.  Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation.  arXiv preprint arXiv:1611.04558 2016.  -- Response to author response:   Thanks for your response & I'm looking forward to reading the final version!


Current state-of-the-art models for lexical substitution – the task of nominating substitutes for a word in context – ignore word sense, instead relying on powerful vector and embedded word representations to find good substitutes. We present a simple method for improving the lexical substitution rankings of existing models by integrating word sense inventories, filtering substitutes from the correct sense to the top of the rankings. To enable maximum coverage of our method, we also propose a novel method for clustering paraphrases by word sense with substitutability in mind. Our method results in sense clusters that are more substitutable and have wider coverage than existing sense inventories. They can be applied as a filter over lexical substitution rankings generated by existing vectorand embeddingbased ranking models to significantly improve their performance.
1 000
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
1 Introduction
Paraphrases are alternate ways of expressing the same meaning. Natural language processing applications like machine translation (Denkowski and Lavie, 2010) and semantic representation (Yu and Dredze, 2014; Faruqui et al., 2015) benefit from large paraphrase resources that enable them to identify different words and phrases with equivalent meaning. To meet this need, there have been several efforts to automatically acquire lexical and phrasal paraphrases. The largest of these is the Paraphrase Database (PPDB) (Pavlick et al., 2015), which contains over 20M English paraphrase pairs.
Since words and phrases can be polysemous, their paraphrases can be divided into subsets representing the different meanings. For example,
paraphrases of the noun paper include sheet, page, notepaper, newspaper, daily, publication, press which can be grouped into two sense clusters c1: {sheet, page, notepaper} and c2: {journal, daily, publication, press}, analagous to WordNet synsets (Miller, 1995). Earlier work (Apidianaki et al., 2014; Cocos and Callison-Burch, 2016) proposed automatically clustering paraphrases by sense, rather than manually specifying sense clusters as WordNet does, in order to organize large paraphrase resources by sense.
There is a clear relationship between the sense of a word or phrase in the context of some sentence, and our ability to replace it with one or more of its paraphrases. For instance, in the sentence The local papers took photographs of the footprint, the paraphrases from c2 would be good substitutes that preserve the overall meaning of the sentence, while paraphrases from c1 would be poor substitutes because they change the meaning.
The task of automatically replacing a word with its meaning-equivalent paraphrases in context is called lexical substitution (lexsub). It was initially conceived as a practical alternative evaluation for Word Sense Disambiguation (WSD) systems (McCarthy and Navigli, 2007). Accordingly, many early lexsub models approached the problem in two steps: first, identify the sense of the target word by reference to an existing sense inventory (e.g. WordNet), and second, propose substitutes corresponding to the predicted sense (Hassan et al., 2007; Yuret, 2007). But current stateof-the-art lexical substitution systems ignore the sense disambiguation step altogether, instead relying on vector space models to find substitutes that are semantically similar to the target, and contextually appropriate (Melamud et al., 2015; Roller and Erk, 2016; Apidianaki, 2016).
Our hypothesis is that a high-coverage, substitutable sense inventory – one in which words belonging to a single sense are mutually substitutable
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
Sentence Annotated Substitutes (Count) Applicable WordNet+ sense
Tasha snatched it from him to rip away the paper. sheet (3), page (3), note (1), notepaper (1), parchment (1) c1: sheet, piece of paper, sheet of paper An hour later, the paper’s circulation manager, Miguel Soler, was shot and killed near his home. newspaper (4), daily (1), publication (1), press (1) c2: press, daily, gazette, tabloid The operators at the East Fishkill factory wear light nylon uniforms, light blue shoe coverings and translucent hair nets made of paper. sheet (1), paper pulp (1), parchment (1), paper stock (1), card stock (1) c3: material, stuff, substance
Table 1: Example annotated sentences for the target paper.N from the CoInCo dataset and the corresponding WordNet+ senses (see sec. 2.1). Numbers after each substitute indicate the number of annotators who made that suggestion.
in most contexts – can be used in combination with a word-based lexsub model to improve the quality of proposed substitutes. In this paper we propose a novel method for clustering paraphrases in PPDB which improves performance on the lexical substitution task. We use a multi-view clustering algorithm that incorporates different views of paraphrase similarity – such as lexical substitution rankings, overlap in foreign translations, and similarity to WordNet synsets – and which can be tailored specifically for the task of lexical substitution. It is generalizable to all parts of speech and languages.
Our method results in sense clusters that are more substitutable than existing sense inventories as measured by their agreement with humangenerated lexsub annotations. They can be applied as a filter over lexsub rankings generated by existing vector- and embedding-based ranking models to significantly improve lexsub performance.
2 Sense Inventories and their Substitutability
A crucial question when evaluating a sense inventory for its fitness for the lexsub task is, how well does it match human judgments about the interchangeability of words? We propose a cluster quality metric, B-Cubed F-Score (Amigó et al., 2009), as a way of comparing an automaticallygenerated sense inventory against a humangenerated set of substitutability judgements.
We can describe a sense inventory, Ct, as a set of sets, {c1, c2, . . . ck}, where each subset contains synonyms of the target t corresponding to one of its senses. For example, the first sense of paper in Table 1 is the set c1 = {sheet, piece of paper, sheet of paper}.
A (human-annotated) lexsub dataset contains multiple sentences for each target word, and a set of annotated substitutes for each sentence (Table 1). We can view the annotated substitutions as a clustering, Lt = {l1, l2, . . . l3}, where each cluster contains the proposed substitutes for a single sentence. In Table 1, for example, the cluster l1 contains {sheet, page, note, notepaper, parchment}.
We call Et the set of all t’s synonyms that appear in both its sense inventory and its lexsub dataset. We then use the extended B-Cubed F-score cluster evaluation metric to quantify the level of agreement between the gold standard sense labels and the proposed substitute clusters for words e ∈ Et. B-Cubed F-score (FB 3
β ) measures cluster quality in terms of B-Cubed precision (measuring the extent to which each sentence’s annotated substitutes come from the same gold standard sense(s)), and B-Cubed recall (measuring the extent to which each sentence’s annotated substitutes cover an entire sense). Full details of our BCubed F-Score implementation are in the supplemental material.
We define the substitutability of t’s sense inventory, Ct, with respect to its annotated substitutes, Lt, as FB 3
β (Et). Given many target words for which we have senses and annotations, we can average the substitutability of an entire sense inventory C over its targets T (for which |Et|≥ 2) with annotations in L into a single substitutability score:
(1)subst(C,L) = Avgt∈T (FB 3 β (Et))
2.1 Measuring Substitutability
We now assess the substitutability of the following existing sense inventories with respect to human-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Sense Inventory
|T | Avg Synonyms per Tgt
Avg Senses per Tgt
Avg CoInCo Overlap CoInCo B3F0.5
WordNet+ 155K 29.7 4.5 5.5 0.841 TWSI 1K 16.7 2.42 6.0 0.842 PPDBClus 1.7M 132.8 13.7 16.6 0.736
Table 2: Characteristics and substitutability (B3F0.5) of each sense inventory analyzed in our study. The Avg CoInCo Overlap indicates the average number of words appearing in both the sense inventory and CoInCo annotations, by target word.
annotated judgements. This will establish a score that we aim to outperform with our new method for generating paraphrase sense clusters.
WordNet+. The first sense inventory we evaluate, WordNet+, is formed from WordNet 3.0. For each CoInCo target word that appears in WordNet, we take its senses to be its synsets, with lemmas belonging to hypernyms and hyponyms of each synset included.
PPDBClus. We also evaluate a much larger, abeit noisier, sense inventory of words in the Paraphrase Database (PPDB) XXL version (Cocos and Callison-Burch, 2016). This sense inventory was automatically generated via spectral clustering, with PPDB2.0 Score serving as the similarity metric. The resulting sense inventory has more than four times the coverage of WordNet, but is also quite noisy.
TWSI. The Turk Bootstrap Word Sense Inventory v2 (TWSI) (Biemann, 2013) is a crowdsourced sense inventory for nouns. It was developed specifically with substitutability in mind using an iterative, bootstrapped process for word sense induction and lexical coverage.
The size and characteristics of each sense inventory are given in Table 2.
For each sense inventory, we evaluate its BCubed F-Score over all sentences in our Concepts In Context (CoInCo) (Kremer et al., 2014) test set (see sec. 4.1). We ignore targets for which |Et|< 2. We find that the average substitutability scores for WordNet and TWSI are significantly higher than that of PPDBClus when aggregated over the entire CoInCo dataset.
3 Multi-View Clustering for Substitutability
We propose a novel method for automatically generating a sense inventory that is tailored to a substitutability metric. Our method produces a sense inventory that has greater coverage than WordNet, and that results in much better lexical substitutions than previous automatically sense-clustered paraphrases (Cocos and Callison-Burch, 2016).
Our method uses a multi-view clustering algorithm to automatically create sense clusters of paraphrases for a given target word. Different ’views’, or representations, allow us to encode properties of paraphrases that are related to our goal of substitutability, including:
• Substitutability of the paraphrase for the target word, over a large number of sentences (Context Substitutability View),
• Strength of the paraphrase’s relationship with other paraphrases of the target word (Paraphrase Similarity View),
• Translation probability between the paraphrase and foreign words across multiple languages (Shared Translations View), and
• Affinity between the paraphrase, and WordNet synsets (WordNet Synsets View)
3.1 Multi-View Clustering
In order to cluster the paraphrases of a target word, we use the multi-view non-negative matrix factorization model of Liu et al. (2013). Nonnegative matrix factorization (NMF) approaches cluster an input matrix by finding two smaller matrices that approximately equal the input when multiplied together. If the input matrix isX ∈ RM×N , where the N columns represent paraphrases to be clustered and the M rows represent features, NMF finds non-negative matricesU ∈ RM×K and V ∈ RN×K such that their product is approximately X; UV T ≈ X . K indicates the number of clusters.
Multi-view NMF is an adaptation of NMF that jointly factorizes n input matrices {X1, X2, . . . Xn}, each having the same number of columns to be clustered (but an arbitrary number of rows, or features). Each Xi gives a different view or representation of the data. Multi-view NMF assumes that the views are
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
complementary, and that each view is likely to cluster the data the same way on its own. Under this assumption, Multi-view NMF works by factorizing each Xi as above, but adds an additional regularization component that coaxes each resulting coefficient matrix, V i, toward a common consensus matrix V ∗. Full details of our multi-view NMF implementation are in the supplemental material.
Using multi-view NMF allows us to incorporate multiple types of information about paraphrases into the clustering algorithm. We now give an overview of the views used in our experiments.
3.1.1 View 1: Context Substitutability
Our ultimate goal is to find clusters of paraphrases that are mutually substitutable in context. The most direct way to do this is to represent each paraphrase in terms of its ’fit’ within a large number of contexts. Intuitively, paraphrases that fit well within the same subset of sentences should be clustered into the same sense.
The question of how to measure the ’fit’ of a paraphrase in context remains. One option might be to use existing human-annotated lexsub data, or to crowdsource a new set of human annotations for each target word. But we want to create a method that is scalable and generalizable. Therefore, we choose to measure lexical substitutability using a simple but high-performing vector space model, AddCos (Melamud et al., 2015).
The AddCos method quantifies the fit of substitute word for target word in context by measuring the semantic similarity of the substitute to the target, and the similarity of the substitute to the context, by comparing word and context embeddings generated by the skip-gram with negative sampling model (Mikolov et al., 2013b,a). Full details on our implementation of the AddCos metric are in the supplemental material.
The first view that we provide as input to the multi-view NMF algorithm is a paraphrasesentence matrix defined as XS ∈ RMS×N , where N denotes the number of paraphrases to be clustered, and MS is an arbitrarily large number (we use 1000 in our experiments). Each row in XS corresponds to a sentence containing the target word that we extract randomly from AGiga. Each entry xij gives the AddCos score for paraphrase j and target t in context Wi: xij = AddCos(pj , t,Wi).
3.1.2 View 2: Paraphrase Similarity
PPDB contains a measure of paraphrase strength called the PPDB2.0 Score. It is a supervised metric designed to align with human judgements of paraphrase quality (Pavlick et al., 2015). If the paraphrases for a target pertain to different senses, then we can expect paraphrases within a given sense to be connected with high PPDB2.0 Scores. Previous work validated this idea, using PPDB2.0 Score as the basis for clustering paraphrases by sense (Cocos and Callison-Burch, 2016).
The third view used in our clustering is a paraphrase-paraphrase matrix XP ∈ RN×N , where N is the number of paraphrases to be clustered. Each entry xi,j contains the PPDB2.0 Score between paraphrases pi and pj . The PPDB2.0 Score is not symmetric, so we use the maximum score between paraphrases pi and pj as the entry for both xij and xji.
3.1.3 View 3: WordNet Synsets
We also experiment with incorporating the clean structure of WordNet to encourage words belonging to the same synset, and others similar to them, to be clustered into the same sense.
The last view that we provide as input to multiview NMF is defined as the paraphrase-synset matrix XW ∈ RMW×N , where N again gives the number of paraphrases for the target, and MW gives the number of WordNet synsets that the target belongs to. Each entry in the matrix, xij gives the cosine similarity between the word embedding for paraphrase pj and a compositional synset embedding for synset ci. To compute synset embeddings, we take the weighted average of the word embeddings for each lemma in the synset, where each lemma embedding is weighted by the maximum of its PPDB2.0 Score with the target word and 1:
vc = ∑ l∈cmax(1, PPDB2.0Score(l, t)) · vl∑ l∈cmax(1, PPDB2.0Score(l, t))
3.1.4 View 4: Shared Translations
Foreign translations can also be used as a proxy for word sense (see, for example, ?, ?, ?, McCarthy et al. (2016) and others.). This is based on the idea that if the foreign translations of a single English lemma can be partitioned into clusters, then the the English lemma is likely polysemous, and the clusters of translations represent its different senses.
PPDB contains the translation probabilities for each foreign translation of its paraphrases over
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
486
487
488
489
490
491
492
493
494
495
496
497
498
499
multiple languages. We use the translation probabilities to construct a paraphrase-translation matrix XF ∈ RMF×N , where N denotes the number of paraphrases to be clustered, and MF is the size of the set of all translations for any paraphrase of t. Practically this number can be very large, so we arbitrarily set a threshold translation probability of 10−9 as the minimum for a translated word’s inclusion in the set. Each row in XF corresponds to a foreign word f . Each entry xij gives the translation probability prob(fi|pj) between the foreign word fi and the corresponding column’s paraphrase, pj .
4 Clustering Experiments
We now apply our proposed sense-clustering method to generate substitutable sense clusters within PPDB.
4.1 Data
We draw the target words for our clustering experiment from the Concepts In Context (CoInCo) dataset, containing over 15K sentences corresponding to nearly 4K unique target words (Kremer et al., 2014). Specifically, we use the 327 target words in CoInCo that have at least 10 example sentences. For each of the 327 eligible targets, we randomly divide the corresponding sentences into 60% training instances and 40% test instances. The resulting training and test sets have 4061 and 2091 sentences/annotations respectively. We will use the training annotations to tune a PPDB Score cutoff threshold to reduce noise, and we will use the test annotations to evaluate the substitutability of the resulting clusters.
4.2 Experimental Setup
For each target word in the CoInCo train and test sets, we take its paraphrases from PPDB2.0-XXL1 with a paraphrase score of at least 2.3 as the set of words to cluster into senses. We chose 2.3 as the threshold score because it gave the best precision and recall when used as a threshold over the CoInCo training data. This reduces the coverage of our resulting sense inventory, but decreases its noise.
The multi-view clustering algorithm requires us to choose the number of clusters, k. For each target word to be clustered, we try clustering with a range of k between 2 and 10. For each resulting
1http://paraphrase.org
clustering, we calculate the mean Silhouette Coefficient – an intrinsic measure of cluster quality that balances inter-cluster similarity and intra-cluster distance. We use the paraphrase-paraphrase matrix XP to as the basis for the Silhouette Coefficient calculation, as it was found to be effective for distinguishing senses in Cocos and CallisonBurch (2016). We choose as the final clustering solution, that clustering which produces the highest Silhouette Coefficient.
We experiment with the number of views used to produce the clusters. We first cluster the paraphrases using each view individually, and then combine views to see how they complement one another. Altogether we experiment with seven unique combinations of views. In the remaining discussion, we use the notation SubstClusX to denote a sense inventory derived from the view(s) included in the superscript. The superscript C denotes the Context Substitutability view, P denotes the Paraphrase Similarity View, T denotes the Shared Translations view, and W denotes the WordNet Synsets view. For example, SubstClusCP denotes the sense inventory resulting from clustering the Context Substitutability and Paraphrase Similarity views.
4.3 Optimized Clustering Results
Of the resulting sense inventories, all multiview inventories and the SubstClusP single-view inventory are more substitutable than WordNet and TWSI in terms of B-Cubed F-Score.
5 Sense Filtering for Lexical Substitution
We now investigate whether it is possible to improve the rankings of vector and embedding-based lexsub ranking systems by using the optimized
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
sense inventory as a filter. Our general approach is to take a set of ranked substitutes generated by an existing lexsub model. Then, we see whether ’filtering’ the ranked substitutes to elevate words belonging to the correct sense of the target in context will improve the overall ranking results. We conduct an oracle experiment – in which we assume we know the correct sense of the target in context for a particular instance – to find the maximum possible improvement in lexsub score using this method. We also evaluate how well the sense filtering method works in practice, by using a simple WSD method to predict the correct sense of the target in context. In both cases, if sense filtering successfully improves the quality of ranked substitutes, it indicates that the sense inventory captures substitutability well.
5.1 Data
We use the test portion of the CoInCo subset defined in Section 4.1 to evaluate the extent to which our baseline and optimized sense inventories improve lexical substitution rankings. The test portion consists of 2091 annotated sentences for 327 target words.
5.2 Ranking Models
Our approach requires a set of rankings produced by a high-quality lexsub model to start. We generate substitution rankings for each target/sentence pair in the test set using two models. The first is the syntactic vector space model of Apidianaki (2016) (Syn.VSM), which demonstrated an ability to correctly choose appropriate PPDB paraphrases for a target word in context. Its vector features correspond to syntactic dependency triples extracted from the English Gigaword corpus. Syn.VSM ranks substitutes in context based on the cosine similarity of the substitute’s basic vector with the target’s contextualized vector. The second model we use is the AddCos model introduced in section 3.1.1, using a window of one word to either side of the target as context. Each ranking model produces a score for each (target, sentence, substitute) tuple in the test set, and ranks substitutes based on the predicted score.
5.3 Metrics
Lexical substitution experiments are typically evaluated using generalized average precision (GAP) (Kishida, 2005). GAP compares a set of
predicted rankings to a set of gold standard rankings. GAP scores range from 0 to 1; a perfect ranking, in which all high-scoring substitutes outrank low-scoring substitutes, has a GAP score of 1.
For each sentence in the CoInCo test set, we consider the PPDB paraphrases for the target word to be the substitution candidates, and we set the CoInCo annotator frequency to be the gold score. Words in PPDB that were not suggested by annotators receive a gold score of 0.001. The predicted rankings are given by each lexsub model (Syn.VSM and AddCos).
5.4 Method
Sense filtering is intended to boost the lexsub rank of substitutes that belong to the most appropriate sense of the target given the context. We run our experiment as a two-step process, which is depicted in Table 4.
First, given a target and sentence from the CoInCo test set, we obtain the PPDB paraphrases for the target word and rank them using each lexsub model model. We calculate the overall unfiltered GAP score for each model as the average GAP over sentences in the CoInCo test set.
Next, we evaluate the ability of a sense inventory to improve the GAP score through filtering. In the oracle experiment, we find the maximum GAP score achieveable by adding a large number (10000) to the lexsub model’s score for words belonging to a single sense. This elevates the words from the chosen sense to the top of the rankings, while preserving their relative order. If the sense inventory corresponds well to substitutability, we should expect this filtering to improve the ranking by eliminating proposed substitutes that do not fall within the correct sense cluster.
Next, having estimated the maximum achievable improvement in GAP score with sense filtering, we apply a simple word sense disambiguation method to see how well this method could work in practice. For each target word in context, we choose the ’best-fit’ sense to be the sense most frequently represented among the top-5 ranked substitutes by the lexsub ranking model. We elevate the scores of all words in the ’best-fit’ sense by adding 10000 to their ranking model score, and calculate the Best-Fit GAP score.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Sentence Last year they contributed $34 million, before tax. Human-annotated Substitutes (score) period (3), time (1), term (1), season (1), annum (1), annual (1) AddCos Model-generated Ranked Substitutes (0.47 GAP) month, week, summer, decade, season, spring, fall, semester, fiscal, autumn, winter, century, quarter, day, beginning, period, time, millennium, harvest, revenue, report, calendar, dispensation, cycle, date, annum, vintage, appropriation, increase, occasion, assignment, percent, edition, budget, age, sophomore, end, trend, term, grade, ... SubstClusCPWT sense with maximum gold overlap (oracle) period, time, age Sense-filtered Ranked Substitutes (0.78 GAP)
period, time, age, month, week, summer, decade, season, spring, fall, semester, fiscal, autumn, winter, century, quarter, day, beginning, millennium, harvest, revenue, report, calendar, dispensation, cycle, date, annum, vintage, appropriation, increase, occasion, assignment, percent, edition, budget, sophomore, end, trend, term, grade, ...
Table 4: Sense-filtering for lexsub. Human-annotated (gold) substitutes for the target year in the given sentence are ranked. The AddCos model ranks the gold words (bold text) in positions 5, 16, 17, 26, and 39. Our SubstClusCPWT sense with the greatest gold overlap is given (matching 2 words). We filter the model-generated substitutes by elevating the ranks of words from the best SubstClusCPWT sense, in the same relative order (italic text). The sense-filtered resulting rankings place the gold words in positions 1, 2, 8, 27, and 39, improving GAP by 0.31.
decline decrease dip
drop
fall slide tumble
escape infiltrate
sneakspin
squeeze
(a) SubstClusC
decline decrease
dip
drop
fall
slidetumble
escape
infiltrate sneakspin
squeeze
(b) SubstClusP
decline
decrease dip drop
fall
slide
tumble
escape infiltrate
sneak spin
squeeze
(c) SubstClusW
decline decrease dip drop fall
slide
tumble
escape
infiltrate sneakspin
squeeze
(d) SubstClusT
decline decrease dipdrop
fallslide
tumble
escape
infiltrate sneakspin
squeeze
(e) SubstClusCPWT
Figure 1: Clustering results under different views for slip.V
6 Results
We report the unfiltered, oracle sense-filtered, and best-fit sense-filtered GAP scores achieved using each of our sense inventories in Table 5.
All baseline and clustered sense inventories can improve the GAP score of the Syn.VSM and AddCos ranking models when used as a filter, based on Oracle GAP scores. This shows that filtering with a sense inventory can improve lexsub results. Further, in most cases, all sense inventories except PPDBClus can improve the GAP score when a simple WSD method is applied (Best-Fit GAP). Thus the sense filtering method is generally effective in practice.
Comparing the B-Cubed F-Score for the sense inventories to the Oracle GAP scores, we find that our substitutability metric is generally a good indicator of the potential for a sense inventory to improve lexsub rankings. In general, sense inventories that score highest in terms of B-Cubed FScore also score well in terms of Oracle GAP. The Best-Fit GAP scores are not perfectly correlated with Oracle GAP scores, however, indicating that our simple WSD method may introduce bias. The simple WSD method fails to find the best sense cluster in many cases; this is an area for improve-
ment. Our automatically-generated sense inventories are well-suited to the task of sense filtering for lexical substitution, as indicated by their performance exceeding all baselines in terms of Oracle GAP and Best-Fit GAP when evaluated over all parts of speech. The area where our SubstClus inventories are weakest is in improving the lexsub rankings for nouns. The crowdsourced TWSI significantly beats all other sense inventories in the nouns-only rankings. This suggests that performance in this task may be part-of-speech specific, and different optimization methods may be required for each part of speech.
7 Related Work
Early lexical substitution models performed disambiguation to propose substitutes in context (Hassan et al., 2007; Yuret, 2007). More recent works ignore explicit sense representation, instead modeling targets with contextualized vectors, against which the vectors of the substitutes are compared and ranked (Erk and Padó, 2008; Dinu and Lapata, 2010; Thater et al., 2011). Most of these works pool together the total set of substitutes available for a given target word in an anno-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Syn.VSM Ranking Results
All Words Nouns Only
Sense Inventory Unfiltered GAP
Oracle GAP Best-Fit GAP
Unfiltered GAP
Oracle GAP Best-Fit GAP
SubstClusC
0.520
0.641 0.576
0.544 0.647 0.585 SubstClusP 0.692 0.540 0.683 0.547 SubstClusW 0.652 0.572 0.652 0.581 SubstClusT 0.654 0.565 0.660 0.562 SubstClusCP 0.681 0.550 0.670 0.544 SubstClusCPW 0.675 0.549 0.669 0.555 SubstClusCPWT 0.675 0.552 0.669 0.555 WordNet+ 0.655 0.563 0.682 0.577 TWSI – – 0.756 0.691 PPDBClus 0.652 0.432 0.681 0.474
AddCos Ranking Results
All Words Nouns Only
Sense Inventory Unfiltered GAP
Oracle GAP Best-Fit GAP
Unfiltered GAP
Oracle GAP Best-Fit GAP
SubstClusC
0.532
0.630 0.571
0.544 0.627 0.568 SubstClusP 0.686 0.541 0.668 0.541 SubstClusW 0.639 0.565 0.634 0.564 SubstClusT 0.644 0.560 0.647 0.537 SubstClusCP 0.670 0.549 0.656 0.540 SubstClusCPW 0.671 0.544 0.657 0.540 SubstClusCPWT 0.671 0.542 0.657 0.539 WordNet+ 0.651 0.567 0.662 0.563 TWSI – – 0.731 0.687 PPDBClus 0.648 0.458 0.667 0.469
Table 5: Average oracle and best-fit sense-filtered GAP scores over all sentences in the CoInCo test set for each sense inventory and lexsub ranking model.
tated dataset (e.g. the SemEval-2007 Lexical Substitution dataset (McCarthy and Navigli, 2007) or the CoInCo corpus (Kremer et al., 2014)) and produce a ranking for the total substitute set. This task is called candidate ranking.
A more difficult, but more realistic, task is substitute prediction (Melamud et al., 2015) or allwords ranking (Roller and Erk, 2016), where the model does not have access to a gold list of candidates but needs to find possible substitutes from the entire vocabulary. This task is very challenging and the substitutes proposed by the embedding models frequently correspond to rare words. An alternative is to use existing semantic resources. Apidianaki (2016) collects substitution candidates from PPDB and ranks them using the Thater et al. (2011) models. We take this approach, nominating substitutes from a target’s PPDB paraphrases.
PPDB is an enormous collection of paraphrases automatically compiled using bilingual pivoting (Bannard and Callison-Burch, 2005): the idea that two words in one language that align to the same words in a different language should be synonymous. Cocos and Callison-Burch (2016) clustered the contents of the PPDB resource by sense us-
ing a spectral clustering algorithm. The generated clusters are high coverage but contain many erroneous paraphrases, as well as paraphrases linked by different types of (non-substitutable) relations. In the substitutability-focused clustering that we propose, the resulting paraphrase clusters are more substitutable.
8 Conclusion
We have presented a novel method for clustering paraphrases by word sense that unites various paraphrase representations in a multi-view approach. By incorporating a view that encodes the mutual substitutability of paraphrases, our method generates paraphrase sense clusters that are more substitutable and coherent than previous results.
We have also showed that it is possible to use word senses as a filter over automaticallygenerated lexsub rankings to improve their agreement with human-annotated substitutes – marrying the strengths of vector- and embeddingbased models with semantic information encoded in sense inventories. Our paraphrase sense clusters outperform existing sense inventories in this application when evaluated over all parts of speech.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Supplemental Material
Here we expand on the technical details of our substitutability measurement and clustering implementation.
A.1 Measuring Substitutability Here we provide further details on our implementation of our substitutability metric, the extended B-Cubed F-Score (Amigó et al., 2009).
B-Cubed F-score measures cluster quality in terms of precision and recall with respect to each item being clustered. We choose this metric over other commonly-used metrics because it meets formal constraints as analyzed in Amigó et al.
(2009), and its extended version2 allows us to assess situations where a single item belongs to multiple clusters.
Given a set of items E, extended B-Cubed defines the gold standard categories of element e ∈ E as C(e) and the predicted clusters of e as L(e). It then defines precision and recall between two items as:
P (e, e′) = Min(|L(e) ∩ L(e′)|, |C(e) ∩ C(e′)|)
|L(e) ∩ L(e′)|
R(e, e′) = Min(|L(e) ∩ L(e′)|, |C(e) ∩ C(e′)|)
|C(e) ∩ C(e′)|
In other words, precision indicates the extent to which two items which are clustered together, belong together. And recall indicates the extent to which two items that belong to the same gold category are clustered together.
To calculate precision (PB 3 ) and recall (RB 3 ) over the entire set E of clustered items, B-Cubed averages the pairwise precision over all pairs of elements that are clustered together, and the pairwise recall over all pairs of elements that are in the same gold category:
PB 3 (E) = Avge(Avge′:L(e)∩L(e′)6=∅P (e, e ′))
RB 3 (E) = Avge(Avge′:C(e)∩C(e′) 6=∅R(e, e ′))
Finally, we can calculate B-Cubed F-score in the usual way:
FB 3 β (E) = (1 + β 2) · P B3(E) ·RB3(E) (β2 · PB3(E)) +RB3(E)
where β is a parameter that corresponds to the emphasis on recall over precision. In our work we use β = 0.5 to emphasize precision over recall in the substitutability score.
To use B-Cubed F-score for measuring substitutability, we first define the set Et of words that appear in both the sense inventory for target word t and the annotated substitutions for t. We view the annotated substitutions as a clustering, Lt, where Lt(e) denotes the set of sentences for which word e is suggested as a substitute for t. Finally we use the sense inventory as the set of gold standard category labels over Et, such that Ct(e) gives the set of senses to which e belongs.
2https://github.com/hhromic/ python-bcubed
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
We can then define the substitutability of t’s sense inventory, Ct, with respect to its annotated substitutes, Lt, as FB 3
β (Et). Finally, given many target words for which we have senses and annotations, we can aggregate the substitutability of the entire sense inventory C over the set of targets T with annotations in L into a single substitutability score:
(2)subst(C,L) = Avgt(FB 3 β (Et))
where FB 3
β (Et) is calculated in terms of Ct and Lt as above.
A.2 Multi-View Non-Negative Matrix Factorization
Given the set of PPDB paraphrases for a target word, there are several sources of information we can use to discover subsets that are mutually substitutable. Thus our setting is a good fit for multiview clustering approaches that incorporate data from multiple views, or representations, of the items to be clustered.
In order to cluster the paraphrases of a target word, we use the multi-view non-negative matrix factorization model of Liu et al. (2013)3. Nonnegative matrix factorization (NMF) approaches cluster an input matrix by finding two smaller matrices that approximately equal the input when multiplied together. If the input matrix isX ∈ RM×N , where the N columns represent paraphrases to be clustered and the M rows represent features, NMF finds non-negative matricesU ∈ RM×K and V ∈ RN×K such that their product is approximately X:
UV T ≈ X
The resulting matrices U and V are called the basis and coefficient matrices respectively. In the result, if K represents the number of clusters of items in X , then the coefficient matrix V provides a transformation from paraphrases in X to clusters. The basis matrix U also provides a mapping from features to clusters.
In order to find U and V , NMF uses a multiplicative update method to minimize the Frobenius norm of the difference between X and UV T , while constraining U and V to be nonnegative:
min U,V ‖X − UV T ‖2F , U ≥ 0, V ≥ 0
3Code available from http://jialu.cs. illinois.edu/
Multi-view NMF is an adaptation of NMF that jointly factorizes n input matrices {X1, X2, . . . Xn}, each having the same number of columns to be clustered (but an arbitrary number of rows, or features). Each Xi gives a different view or representation of the data. Multi-view NMF assumes that the views are complementary, and that each view is likely to cluster the data the same way on its own. Under this assumption, Multi-view NMF works by factorizing each Xi as above, but adds an additional regularization component that coaxes each resulting coefficient matrix, V i, toward a common consensus matrix V ∗. Specifically, the optimization problem becomes to minimize, over U i, V i, V ∗ for 1 ≤ i ≤ n: n∑ i =1 ‖Xi − U iV iT ‖2F + n∑ i=1 λi‖V i − V ∗‖2F s.t.∀1 ≤ k ≤ K, ‖U (i).,k ‖1 = 1 and U i, V i, V ∗ ≥ 0
where each λi parameter tunes the relative weight between views. In the original paper and in our experiments, we set λi = 0.01 for all i. Setting the sum of each column in the basis matrix to 1 is a normalization technique introduced in (Liu et al., 2013) that confers a probabilistic interpretation on the basis and coefficient matrices, and ensures the magnitude of each is comparable for optimization.
A.3 AddCos Lexical Substitution Model We use the AddCos (Melamud et al., 2015) lexsub model several places in our work. We use it first to measure the similarity between a paraphrase and a sentence in the paraphrase-sentence matrix used as the first view for clustering. We also use it as a ranking model for the CoInCo dataset, which we then improve upon using our sense-filtering approach. Here we provide details of our implementation.
The AddCos method quantifies the fit of substitute word s for target word t in context W by measuring the semantic similarity of the substitute to the target, and the similarity of the substitute to the context:
AddCos(s, t,W )
= |W |·cos(s, t) +
∑ w∈W cos(s, w)
2 · |W |
The vectors s and t are word embeddings of the substitute and target generated by the skip-gram
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
with negative sampling model (Mikolov et al., 2013b,a). The context W is the set of words appearing within a fixed-width window of the target t in a sentence (we use a window of 1), and the embeddings c are context embeddings generated by skip-gram. In our implementation, we train 300-dimensional word and context embeddings over the 4B words in the Annotated Gigaword (AGiga) corpus (Napoles et al., 2012) using the gensim word2vec package (Mikolov et al., 2013b,a; Řehůřek and Sojka, 2010). The word2vec training parameters we use are a context window of size 3, learning rate alpha from 0.025 to 0.0001, minimum word count 100, sampling parameter 1e−4, 10 negative samples per target word, and 5 training epochs.

This paper proposes integrating word sense inventories into existing approaches for the lexical substitution task by using these inventories to filter candidates. To do so, the authors first propose a metric to measure the mutual substitutability of sense inventories with human judgments for the lexsub task, and empirically measure the substitutability of inventories from various sources such as WordNet and PPDB. Next, they propose clustering different paraphrases of a word from PPDB using a multi-view clustering approach, to automatically generate a sense inventory instead of using the aforementioned inventories. Finally, they use these clusters with a naive (majority in top 5) WSD technique to filter existing ranked list of substitution candidates.  - Strengths:  * The key idea of marrying vector space model based approaches and sense inventories for the lexsub task is useful since these two techniques seem to have complementary information, especially since the vector space models are typically unaware of sense and polysemy.  * The oracle evaluation is interesting as it gives a clear indication of how much gain can one expect in the best case, and while there is still a large gap between the oracle and actual scores, we can still argue for the usefulness of the proposed approach due to the large difference between the unfiltered GAP and the oracle GAP.  - Weaknesses:  * I don't understand effectiveness of the multi-view clustering approach. Almost all across the board, the paraphrase similarity view does significantly better than other views and their combination. What, then, do we learn about the usefulness of the other views? There is one empirical example of how the different views help in clustering paraphrases of the word 'slip', but there is no further analysis about how the different clustering techniques differ, except on the task directly. Without a more detailed analysis of differences and similarities between these views, it is hard to draw solid conclusions about the different views.                                    * The paper is not fully clear on a first read. Specifically, it is not immediately clear how the sections connect to each other, reading more like disjoint pieces of work. For instance, I did not understand the connections between section 2.1 and section 4.3, so adding forward backward pointer references to sections should be useful in clearing up things. Relatedly, the multi-view clustering section (3.1) needs editing, since the subsections seem to be out of order, and citations seem to be missing (lines 392 and 393).  * The relatively poor performance on nouns makes me uneasy. While I can expect TWSI to do really well due to its nature, the fact that the oracle GAP for PPDBClus is higher than most clustering approaches is disconcerting, and I would like to understand the gap better. This also directly contradicts the claim that the clustering approach is generalizable to all parts of speech (124-126), since the performance clearly isn't uniform.  - General Discussion:  The paper is mostly straightforward in terms of techniques used and experiments. Even then, the authors show clear gains on the lexsub task by their two-pronged approach, with potentially more to be gained by using stronger WSD algorithms.  Some additional questions for the authors :  * Lines 221-222 : Why do you add hypernyms hyponyms? * Lines 367-368 : Why does X^{P} need to be symmetric? * Lines 387-389 : The weighting scheme seems kind of arbitrary. Was this indeed arbitrary or is this a principled choice? * Is the high performance of SubstClus^{P} ascribable to the fact that the number of clusters was tuned based on this view? Would tuning the number of clusters based on other matrices affect the results and the conclusions? * What other related tasks could this approach possibly generalize to? Or is it only specific to lexsub?