Morph-fitting: Fine-Tuning Word Vector Spaces with Simple Language-Specific Rules
Morphologically rich languages accentuate two properties of distributional vector space models: 1) the difficulty of inducing accurate representations for lowfrequency word forms; and 2) insensitivity to distinct lexical relations that have similar distributional signatures. These effects are detrimental for language understanding systems, which may infer that inexpensive is a rephrasing for expensive or may not associate acquire with acquires. In this work, we propose a novel morph-fitting procedure which moves past the use of curated semantic lexicons for improving distributional vector spaces. Instead, our method injects morphological constraints generated using simple language-specific rules, pulling inflectional forms of the same word close together and pushing derivational antonyms far apart. In intrinsic evaluation over four languages, we show that our approach: 1) improves low-frequency word estimates; and 2) boosts the semantic quality of the entire word vector collection. Finally, we show that morph-fitted vectors yield large gains in the downstream task of dialogue state tracking, highlighting the importance of morphology for tackling long-tail phenomena in language understanding tasks.
1 Introduction
Word representation learning has become a research area of central importance in natural language processing (NLP), with its usefulness demonstrated across many application areas such as parsing (Chen and Manning, 2014), machine translation (Zou et al., 2013), and many others (Turian et al., 2010; Collobert et al., 2011). Most promi-
nent word representation techniques are grounded in the distributional hypothesis, relying on word co-occurrence information in large textual corpora (Curran, 2004; Turney and Pantel, 2010; Mikolov et al., 2013; Mnih and Kavukcuoglu, 2013; Levy and Goldberg, 2014; Schwartz et al., 2015, i.a.).
Morphologically rich languages, in which “substantial grammatical information. . . is expressed at word level” (Tsarfaty et al., 2010), pose specific challenges for NLP. This is not always considered when techniques are evaluated on languages such as English or Chinese, which do not have rich morphology. In the case of distributional vector space models, morphological complexity brings two challenges to the fore:
1. Estimating Rare Words: A single lemma can have many different surface realisations. Naively treating each realisation as a separate word leads to sparsity problems and a failure to exploit their shared semantics. On the other hand, lemmatising the entire corpus can obfuscate the differences that exist between different word forms even though they share some aspects of meaning.
2. Embedded Semantics: Morphology can encode semantic relations such as antonymy (e.g. literate and illiterate, expensive and inexpensive) or synonymy (north, northern, northerly).
In this work, we tackle the two challenges jointly by introducing a resource-light vector space finetuning procedure termed morph-fitting. The proposed method does not require curated knowledge bases or gold lexicons. Instead, it makes use of the observation that morphology implicitly encodes semantic signals pertaining to synonymy (e.g., German word inflections katalanisch, katalanischem, katalanischer denote the same semantic concept in different grammatical roles), and antonymy (e.g., mature vs. immature), capitalising on the proliferation of word forms in morphologically
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
en_expensive de_teure it_costoso en_slow de_langsam it_lento en_book de_buch it_libro costly teuren dispendioso fast allmählich lentissimo books sachbuch romanzo
costlier kostspielige remunerativo slower rasch lenta memoir buches racconto cheaper aufwändige redditizio slower gemächlich inesorabile novel romandebüt volumetto prohibitively kostenintensive rischioso slowed schnell rapidissimo storybooks büchlein saggio pricey aufwendige costosa slowing explosionsartig graduale blurb pamphlet ecclesiaste
expensiveness teures costosa slow langsamer lenti booked bücher libri costly teuren costose slowing langsames lente rebook büch libra
costlier teurem costosi slowed langsame lenta booking büche librare ruinously teurer dispendioso slowness langsamem veloce rebooked büches libre unaffordable teurerer dispendiose slows langsamen rapido books büchen librano
Table 1: The nearest neighbours of three example words (expensive, slow and book) in English, German and Italian before (top) and after (bottom) morph-fitting.
rich languages. Formalised as an instance of the post-processing semantic specialisation paradigm (Faruqui et al., 2015; Mrkšić et al., 2016), morphfitting is steered by a set of linguistic constraints derived from simple language-specific rules which describe (a subset of) morphological processes in a language. The constraints emphasise similarity on one side (e.g., by extracting morphological synonyms), and antonymy on the other (by extracting morphological antonyms), see Fig. 1 and Tab. 2.
The key idea of the fine-tuning process is to pull synonymous examples described by the constraints closer together in a transformed vector space, while at the same time pushing antonymous examples away from each other. The explicit post-hoc injection of morphological constraints enables: a) estimating more accurate vectors for low-frequency words if they are described by the constraints containing their relation with high-frequency words,1 thus tackling the data sparsity problem; and b) specialising the distributional space to distinguish between similarity and association, thus supporting language understanding applications such as dialogue state tracking (DST).
As a post-processor, morph-fitting allows the integration of morphological rules with any distributional vector space in any language: it treats an input distributional word vector space as a black box and fine-tunes it so that the transformed space reflects the knowledge coded in the input morphological constraints (e.g., Italian words rispettoso and irrispetosa should be far apart in the transformed vector space, see Fig. 1). Tab. 1 illustrates the effects of morph-fitting by qualitative examples in three languages: the vast majority of nearest neighbours are “morphological” synonyms.
We demonstrate the efficacy of morph-fitting in four languages (English, German, Italian, Rus-
1For instance, the vector for the word katalanischem which occurs only 9 times in the German Wikipedia will be pulled closer to the more reliable vectors for katalanisch and katalanischer, with frequencies of 2097 and 1383 respectively.
sian), yielding large and consistent improvements on benchmarking word similarity evaluation sets such as SimLex-999 (Hill et al., 2015), its multilingual extension (Leviant and Reichart, 2015), and SimVerb-3500 (Gerz et al., 2016). The improvements are reported for all four languages, and with a variety of input distributional spaces, verifying the robustness of the approach.
We then show that incorporating morph-fitted vectors into a state-of-the-art neural-network DST model results in improved tracking performance, especially for morphologically rich languages. We report an improvement of 4% on Italian, and 6% on German when using morph-fitted vectors instead of the distributional ones, setting a new state-of-theart DST performance for the two datasets.2
2 Morph-fitting: Methodology
Preliminaries In this work, we focus on four languages with varying levels of morphological complexity: English (EN), German (DE), Italian (IT), and Russian (RU). These correspond to languages in the Multilingual SimLex-999 dataset. Vocabularies Wen, Wde, Wit, Wru are compiled by retaining all word forms from the four Wikipedias with
2There are no readily available DST datasets for Russian.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
word frequency over 10, see Tab. 3. We then query these (large) vocabularies using a set of simple language-specific if-then-else rules to extract sets of linguistic constraints, see Tab. 2.3 These constraints (Sect. 2.2) are used as input for the vector space post-processing ATTRACT-REPEL algorithm (outlined in Sect. 2.1).
2.1 The ATTRACT-REPEL Model
The ATTRACT-REPEL model is an extension of PARAGRAM, proposed by Wieting et al. (2015). It provides a generic framework for incorporating similarity (e.g. successful and accomplished) and antonymy constraints (e.g. nimble and clumsy) into pre-trained word vectors. Given the initial vector space and collections of ATTRACT and REPEL constraints A and R, the model gradually modifies the space to bring the designated word vectors closer together or further apart. The method’s cost function consists of three terms. The first term pulls the ATTRACT examples (xl, xr) ∈ A closer together. If BA denotes the current mini-batch of ATTRACT examples, this term can be expressed as:
A(BA) = ∑
(xl,xr)∈BA
(ReLU (δatt + xltl − xlxr)
+ ReLU (δatt + xrtr − xlxr))
where δatt is the similarity margin which determines how much closer synonymous vectors should be to each other than to each of their respective negative examples. ReLU(x) = max(0, x) is the standard rectified linear unit (Nair and Hinton, 2010). The ‘negative’ example ti for each word xi in any ATTRACT pair is the word vector closest to xi among the examples in the current minibatch (distinct from its target synonym and xi itself). This means that this term forces synonymous words from the in-batch ATTRACT constraints to be closer to one another than to any other word in the current mini-batch.
The second term pushes antonyms away from each other. If (xl, xr) ∈ BR is the current minibatch of REPEL constraints, this term is:
R(BR) = ∑
(xl,xr)∈BR
(ReLU (δrpl + xlxr − xltr)
+ ReLU (δrpl + xlxr − xrtr)) 3A native speaker is able to easily come up with these sets of morphological rules (or at least with a reasonable subset of rules) without any linguistic training. What is more, the rules for DE, IT, and RU were created by non-native, non-fluent speakers with a limited knowledge of the three languages, exemplifying the simplicity and portability of the approach.
English German Italian
(discuss, discussed) (schottisch, schottischem) (golfo, golfi) (laugh, laughing) (damalige, damaligen) (minato, minata) (pacifist, pacifists) (kombiniere, kombinierte) (mettere, metto) (evacuate, evacuated) (schweigt, schweigst) (crescono, cresci) (evaluate, evaluates) (hacken, gehackt) (crediti, credite)
(dressed, undressed) (stabil, unstabil) (abitata, inabitato) (similar, dissimilar) (geformtes, ungeformt) (realtà, irrealtà) (formality, informality) (relevant, irrelevant) (attuato, inattuato)
In this case, each word’s ‘negative’ example is the (in-batch) word vector furthest away from it (and distinct from the word’s target antonym). The intuition is that we want antonymous words from the input REPEL constraints to be further away from each other than from any other word in the current mini-batch; δrpl is now the repel margin.
The final term of the cost function serves to retain the abundance of semantic information encoded in the starting distributional space. If xiniti is the initial distributional vector and V (B) is the set of all vectors present in the given mini-batch, this term (per mini-batch) is expressed as:
R(BA,BR) = ∑
xi∈V (BA∪BR)
λreg ∥∥∥xiniti − xi∥∥∥ 2
where λreg is the L2 regularisation constant.4 This term effectively pulls word vectors towards their initial (distributional) values, ensuring that relations encoded in initial vectors persist as long as they do not contradict the newly injected ones.
2.2 Language-Specific Rules and Constraints
Semantic Specialisation with Constraints The fine-tuning ATTRACT-REPEL procedure is entirely driven by the input ATTRACT and REPEL sets of constraints. These can be extracted from a variety of semantic databases such as WordNet (Fellbaum, 1998), the Paraphrase Database (Ganitkevitch et al., 2013; Pavlick et al., 2015), or BabelNet (Navigli and Ponzetto, 2012; Ehrmann et al., 2014) as done in prior work (Faruqui et al., 2015; Wieting et al.,
4We use hyperparameter values δatt = 0.6, δrpl = 0.0, λreg = 10
−9 from prior work without fine-tuning. We train all models for 10 epochs with AdaGrad (Duchi et al., 2011).
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
2015; Mrkšić et al., 2016, i.a.). In this work, we investigate another option: extracting constraints without curated knowledge bases in a spectrum of languages by exploiting inherent language-specific properties related to linguistic morphology. This relaxation ensures a wider portability of ATTRACTREPEL to languages and domains without readily available or adequate resources.
Extracting ATTRACT Pairs For the ATTRACT constraints, we focus on inflectional rather than on derivational morphology rules as the former preserve the full meaning of a word, modifying it only to reflect grammatical roles (e.g., verb tense, case markers; (read, reads)).5 This choice is guided by our intent to fine-tune the original vector space to improve the embedded semantic relations.
We define two rules for English, widely recognised as morphologically simple (Avramidis and Koehn, 2008; Cotterell et al., 2016). These are: (R1) if w1, w2 ∈Wen, where w2 = w1 + ing/ed/s, then add (w1, w2) and (w2, w1) to the set of ATTRACT constraints A. This rule yields pairs such as (look, looks), (look, looking), (look, looked).
If w[: −1] is a function which strips the last character from word w, the second rule is: (R2) if w1 ends with the letter e and w1 ∈Wen and w2 ∈ Wen, where w2 = w1[: −1] + ing/ed/s, then add (w1, w2) and (w2, w1) toA. This creates pairs such as (create, creates), (create, creating) and (create, created). Naturally, introducing more sophisticated rules is possible in order to cover for other special cases and morphological irregularities (e.g., sweep / swept), but in all our EN experiments, A is based on the two simple EN rules R1 and R2.
The other three languages, with more complicated morphology, yield a larger number of rules. In Italian, we rely on the sets of rules spanning: (1) regular formation of plural (libro / libri); (2) regular verb conjugation (aspettare / aspettiamo); (3) regular formation of past participle (aspettare / aspettato); and (4) rules regarding grammatical gender (bianco / bianca). Besides these, another set of rules is used for German and Russian: (5) regular declension (e.g., asiatisch / asiatischem).
5The core difference between inflectional and derivational morphology may be summarised in a few lines as follows: the former refers to a set of processes through which the word form expresses meaningful syntactic information, e.g., verb tense, without any change to the semantics of the word. On the other hand, the latter refers to the formation of new words with semantic shifts in meaning (Schone and Jurafsky, 2001; Haspelmath and Sims, 2013; Lazaridou et al., 2013; Zeller et al., 2013; Cotterell and Schütze, 2017).
Extracting REPEL Pairs As another source of implicit semantic signals, W also contains words which represent derivational antonyms: e.g., two words that denote concepts with opposite meanings, generated through a derivational process. We use a standard set of EN “antonymy” prefixes: APen = {dis, il, un, in, im, ir, mis, non, anti} (Fromkin et al., 2013). If w1, w2 ∈ Wen, where w2 is generated by adding a prefix from APen to w1, then (w1, w2) and (w2, w1) are added to the set of REPEL constraints R. This rule generates pairs such as (advantage, disadvantage) and (regular, irregular). An additional rule replaces the suffix -ful with -less, extracting antonyms such as (careful, careless).
Following the same principle, we use APde = {un, nicht, anti, ir, in, miss}, APit = {in, ir, im, anti}, and APru = {не, анти}. For instance, this generates an IT pair (rispettoso, irrispettoso) (see Fig. 1). For DE, we use another rule targeting suffix replacement: -voll is replaced by -los.
We further expand the set of REPEL constraints by transitively combining antonymy pairs from the previous step with inflectional ATTRACT pairs. This step yields additional constraints such as (rispettosa, irrispettosi) (see Fig. 1). The final A andR constraint counts are given in Tab. 3. The full sets of rules are available as supplemental material.
3 Experimental Setup
Training Data and Setup For each of the four languages we train the skip-gram with negative sampling (SGNS) model (Mikolov et al., 2013) on the latest Wikipedia dump of each language. We induce 300-dimensional word vectors, with the frequency cut-off set to 10. The vocabulary sizes |W | for each language are provided in Tab. 3.6 We label these collections of vectors SGNS-LARGE.
Other Starting Distributional Vectors We also analyse the impact of morph-fitting on other collections of well-known EN word vectors. These vectors have varying vocabulary coverage and are trained with different architectures. We test standard distributional models: Common-Crawl GloVe (Pennington et al., 2014), SGNS vectors (Mikolov et al., 2013) with various contexts (BOW = bag-ofwords; DEPS = dependency contexts), and training data (PW = Polyglot Wikipedia from Al-Rfou
6Other SGNS parameters were set to standard values (Baroni et al., 2014; Vulić and Korhonen, 2016b): 15 epochs, 15 negative samples, global learning rate: .025, subsampling rate: 1e− 4. Similar trends in results persist with d = 100, 500.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
et al. (2013); 8B = 8 billion token word2vec corpus), following (Levy and Goldberg, 2014) and (Schwartz et al., 2015). We also test the symmetricpattern based vectors of Schwartz et al. (2016) (SymPat-Emb), count-based PMI-weighted vectors reduced by SVD (Baroni et al., 2014) (Count-SVD), a model which replaces the context modelling function from CBOW with bidirectional LSTMs (Melamud et al., 2016) (Context2Vec), and two sets of EN vectors trained by injecting multilingual information: BiSkip (Luong et al., 2015) and MultiCCA (Faruqui and Dyer, 2014). We also experiment with a selection of standard distributional spaces in other languages from prior work (Dinu et al., 2015; Luong et al., 2015; Vulić and Korhonen, 2016a).
Morph-fixed Vectors A baseline which utilises an equal amount of knowledge as morph-fitting, termed morph-fixing, fixes the vector of each word to the distributional vector of its most frequent inflectional synonym, tying the vectors of lowfrequency words to their more frequent inflections. For each word w1, we construct a set of M + 1 words Ww1 = {w1, w′1, . . . , w′M} consisting of the word w1 itself and all M words which cooccur with w1 in the ATTRACT constraints. We then choose the word w′max from the set Ww1 with the maximum frequency in the training data, and fix all other word vectors in Ww1 to its word vector. The morph-fixed vectors (MFIX) serve as our primary baseline, as they outperformed another straightforward baseline based on stemming across all of our intrinsic and extrinsic experiments.
Morph-fitting Variants We analyse two variants of morph-fitting: (1) using ATTRACT constraints only (MFIT-A), and (2) using both ATTRACT and REPEL constraints (MFIT-AR).7
4 Intrinsic Evaluation: Word Similarity
Evaluation Setup and Datasets The first set of experiments intrinsically evaluates morph-fitted vector spaces on word similarity benchmarks, using Spearman’s rank correlation as the evaluation metric. First, we use the SimLex-999 dataset, as well as SimVerb-3500, a recent EN verb pair similarity dataset providing similarity ratings for 3,500 verb
7We also tried using another post-processing model (Mrkšić et al., 2016) in lieu of ATTRACT-REPEL. However, this model was computationally intractable with SGNSLARGE vectors. Moreover, it was consistently outperformed by ATTRACT-REPEL on vector spaces with smaller vocabularies.
pairs.8 SimLex-999 was translated to DE, IT, and RU by Leviant and Reichart (2015), and they crowdsourced similarity scores from native speakers. We use this dataset for our multilingual evaluation.9
Morph-SimLex We also introduce a synthetic dataset based on multilingual SimLex, termed Morph-SimLex. Since the original sets contain only word lemmas, they are unable to evaluate whether a representation model improves vectors for all synonymous word inflections. Therefore, we enrich the sets of pairs using the same set of ATTRACT rules from Sect. 2.2. In short, given a word pair (w1, w2) with a SimLex score sl1,2, we again construct sets Ww1 = {w1, w′1, . . . , w′M} and Ww2 = {w2, w′′1 , . . . , w′′N}, where Ww1 consists of w1 and all words which co-occur with w1 in the A constraints; the same holds for Ww2 . Morph-SimLex pairs are then generated by taking the Cartesian product between Ww1 and Ww2 , and assigning the same score sl1,2 to each such pair. The final dataset is constructed by repeating the procedure for each of the 999 SimLex pairs, yielding 13,213 EN pairs, 17,021 DE pairs, 18,281 IT pairs, and 10,289 RU pairs. We make this dataset available in hope it can aid further research on improving morphological relations in vector spaces.
Morph-fitting EN Word Vectors As the first experiment, we morph-fit a wide spectrum of EN distributional vectors induced by various architectures (see Sect. 3). The results on SimLex and SimVerb are summarised in Tab. 4. The results with EN SGNS-LARGE vectors are shown in Fig. 2a. Morphfitted vectors bring consistent improvement across all experiments, regardless of the quality of the initial distributional space. This finding confirms that the method is robust: its effectiveness does not depend on the architecture used to construct the initial space. To illustrate the improvements, note that the best score on SimVerb for a model trained on running text is achieved by Context2vec (ρ = 0.388); injecting morphological constraints into this vector space results in a gain of 7.1 ρ points.
Experiments on Other Languages We next extend our experiments to other languages, testing both morph-fitting variants. The results are sum-
8Unlike other gold standard resources such as WordSim353 (Finkelstein et al., 2002) or MEN (Bruni et al., 2014), SimLex and SimVerb provided explicit guidelines to discern between semantic similarity and association, so that related but non-similar words (e.g. cup and coffee) have a low rating.
9Since Leviant and Reichart (2015) re-scored the original EN SimLex, we use their EN SimLex version for consistency.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Evaluation Vectors SimLex-999 SimVerb-3500 1. SG-BOW2-PW (300) (Mikolov et al., 2013) .339→ .439 .277→ .381 2. GloVe-6B (300) (Pennington et al., 2014) .324→ .438 .286→ .405 3. Count-SVD (500) (Baroni et al., 2014) .267→ .360 .199→ .301 4. SG-DEPS-PW (300) (Levy and Goldberg, 2014) .376→ .434 .313→ .418 5. SG-DEPS-8B (500) (Bansal et al., 2014) .373→ .441 .356→ .473 6. MultiCCA-EN (512) (Faruqui and Dyer, 2014) .314→ .391 .296→ .354 7. BiSkip-EN (256) (Luong et al., 2015) .276→ .356 .260→ .333 8. SG-BOW2-8B (500) (Schwartz et al., 2015) .373→ .440 .348→ .441 9. SymPat-Emb (500) (Schwartz et al., 2016) .381→ .442 .284→ .373 10. Context2Vec (600) (Melamud et al., 2016) .371→ .440 .388→ .459
Table 4: The impact of morph-fitting (MFIT-AR used) on a representative set of EN vector space models. All results show the Spearman’s ρ correlation before and after morph-fitting. The numbers in parentheses refer to the vector dimensionality.
Vectors Distrib. MFIT-A MFIT-AR EN: GloVe-6B (300) .324 .376 .438 EN: SG-BOW2-PW (300) .339 .385 .439 DE: SG-DEPS-PW (300) (Vulić and Korhonen, 2016a) .267 .318 .325 DE: BiSkip-DE (256) (Luong et al., 2015) .354 .414 .421 IT: SG-DEPS-PW (300) (Vulić and Korhonen, 2016a) .237 .351 .391 IT: CBOW5-Wacky (300) (Dinu et al., 2015) .363 .417 .446
Table 5: Results on multilingual SimLex-999 (EN, DE, and IT) with two morph-fitting variants.
marised in Tab. 5, while Fig. 2a-2d show results for the morph-fitted SGNS-LARGE vectors. These scores confirm the effectiveness and robustness of morph-fitting across languages, suggesting that the idea of fitting to morphological constraints is indeed language-agnostic, given the set of languagespecific rule-based constraints. Fig. 2 also demonstrates that the morph-fitted vector spaces consistently outperform the morph-fixed ones.
Morph-SimLex performance across all languages shows even stronger relative gains over distributional and morph-fixed vectors. The original SimLex dataset only contains word lemmas. Consequently, it fails to penalise word vector collections with bad estimates of less-frequent word forms. The comparison between MFIT-A and MFIT-AR indicates that both sets of constraints are important
for the fine-tuning process: while MFIT-A already yields consistent gains over the initial spaces, a further refinement can be achieved by also incorporating the antonymous REPEL constraints.
5 Downstream Task: Dialogue State Tracking (DST)
Goal-oriented dialogue systems provide conversational interfaces for tasks such as booking flights or finding restaurants. In slot-based systems, application domains are specified using ontologies that define the search constraints which users can express. An ontology consists of a number of slots and their assorted slot values. In a restaurant search domain, sets of slot-values could include PRICE = [cheap, expensive] or FOOD = [Thai, Indian, ...]. The DST model is the first component of modern dialogue pipelines (Young, 2010). It serves to capture the intents expressed by the user at each dialogue turn and update the belief state. This is the system’s internal estimate of the user’s goals, used by the downstream dialogue manager to choose the system response. The following example shows the true dialogue state in a multi-turn dialogue:
User: What’s good in the southern part of town? inform(area=south) System: Vedanta is the top-rated Indian place. User: How about something cheaper? inform(area=south, price=cheap) System: Seven Days is very popular. Great hot pot. User: What’s the address? inform(area=south, price=cheap); request(address) System: Seven Days is at 66 Regent Street.
The Dialogue State Tracking Challenge (DSTC) shared task series formalised the evaluation and provided labelled DST datasets (Henderson et al., 2014a,b; Williams et al., 2016). While a plethora of DST models are available based on, e.g., handcrafted rules (Wang et al., 2014) or conditional random fields (Lee and Eskenazi, 2013), the recent DST methodology has seen a shift towards neuralnetwork architectures (Henderson et al., 2014c; Mrkšić et al., 2015; Liu and Perez, 2017, i.a.)
Model: Neural Belief Tracker To detect intents in user utterances, most existing models rely on either (or both): 1) Spoken Language Understanding models which require large amounts of annotated training data; or 2) hand-crafted, domain-specific lexicons which try to capture lexical and morphological variation. The Neural Belief Tracker (NBT) is a novel DST model which overcomes both issues
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Distributional MFix MFit-A MFit-AR 0.6
0.65
0.7
0.75
0.8
S im
L ex
(S p ea rm
an ’s ρ )
EN Vector Collections
D S T P erform ace (Joint)
SimLex Morph-SimLex DST
(a) English
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Distributional MFix MFit-A MFit-AR 0.6
0.65
0.7
0.75
0.8
S im
L ex
(S p ea rm
an ’s ρ )
DE Vector Collections
D S T P erform ace (Joint) SimLex Morph-SimLex DST
(b) German
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Distributional MFix MFit-A MFit-AR 0.6
0.65
0.7
0.75
0.8
S im
L ex
(S p ea rm
an ’s ρ )
IT Vector Collections
D S T P erform ace (Joint) SimLex Morph-SimLex DST
(c) Italian
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Distributional MFix MFit-A MFit-AR S im
L ex
(S p ea rm
an ’s ρ )
RU Vector Collections
SimLex Morph-SimLex
(d) Russian
Figure 2: An overview of the results (Spearman’s ρ correlation) for four languages on SimLex-999 (blue squares), Morph-SimLex-999 (red triangles), and the downstream DST performance (black diamonds) using SGNS-LARGE vectors (d = 300), see Tab. 3 and Sect. 3. The left y axis measures the intrinsic word similarity performance, while the right y axis provides the scale for the DST performance.
by reasoning purely over pre-trained word vectors (Mrkšić et al., 2016). The NBT learns to compose these vectors into intermediate utterance and context representations. These are then used to decide which of the ontology-defined intents (goals) have been expressed by the user. The NBT model keeps word vectors fixed during training, so that unseen, yet related words can be mapped to the right intent at test time (e.g. northern to north).
Data: Multilingual WOZ 2.0 Dataset Our DST evaluation is based on the WOZ dataset, released by Wen et al. (2017). In this Wizard-of-Oz setup, two Amazon Mechanical Turk workers assumed the role of the user and the system asking/providing restaurant information. Users typed instead of speaking, removing the need to deal with noisy speech recognition. In DSTC datasets, users would quickly adapt to the system’s inability to deal with complex queries. Conversely, the WOZ setup allowed them to use sophisticated language. The WOZ 2.0 release expanded the dataset to 1,200 dialogues (Mrkšić et al., 2016). In this work, we use translations of this dataset to Italian and German, provided by the authors of the original dataset.
Evaluation Setup The principal metric we use to
measure DST performance is joint goal accuracy, which represents the proportion of test set dialogue turns where all user goals expressed up to that point of the dialogue were decoded correctly (Henderson et al., 2014a). The NBT models for EN, DE and IT are trained using four variants of the SGNSLARGE vectors: 1) the initial distributional vectors; 2) morph-fixed; 3) and 4) the two variants of morphfitted vectors (see Sect. 3). Results and Discussion The diamond-dashed lines (against the right axes) in Fig. 2 show the DST performance of NBT models making use of the four vector collections. IT and DE benefit from both kinds of morph-fitting: IT performance increases 74.1→ 78.1 (MFIT-A) and DE performance rises even more: 60.6 → 66.3 (MFIT-AR), setting a new state-of-the-art score for both languages. The morph-fixed vectors do not enhance DST performance, probably because fixing word vectors to their highest frequency inflectional form eliminates useful semantic content encoded in the original vectors. On the other hand, morph-fitting makes use of this information, supplementing it with semantic relations between different morphological forms. These conclusions are in line with the SimLex and Morph-SimLex gains, where morph-fitting outper-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
forms distributional and morph-fixed vectors. English performance shows little variation across the four word vector collections investigated here. This corroborates our intuition that, as a morphologically simpler language, English stands to gain less from fine-tuning the morphological variation for downstream applications. This result again points at the discrepancy between intrinsic and extrinsic evaluation: the considerable gains in SimLex performance do not necessarily induce similar gains in downstream performance.
6 Related Work
Semantic Specialisation A standard approach to incorporating external information into vector spaces is to pull the representations of similar words closer together. Some models integrate such constraints into the training procedure, modifying the prior or the regularisation (Yu and Dredze, 2014; Xu et al., 2014; Bian et al., 2014; Kiela et al., 2015), or using a variant of the SGNS-style objective (Liu et al., 2015; Osborne et al., 2016). Another class of models, popularly termed retrofitting, injects lexical knowledge from available semantic databases (e.g., WordNet, PPDB) into pre-trained word vectors (Faruqui et al., 2015; Jauhar et al., 2015; Wieting et al., 2015; Nguyen et al., 2016; Mrkšić et al., 2016). Morph-fitting falls into the latter category. However, instead of resorting to curated knowledge bases, and experimenting solely with English, we show that the morphological richness of any language can be exploited as a source of inexpensive supervision for fine-tuning vector spaces, at the same time specialising them to better reflect true semantic similarity.
Word Vectors and Morphology The use of morphological resources to improve the representations of morphemes and words is an active area of research. The majority of proposed architectures encode morphological information, provided either as gold standard morphological resources (SylakGlassman et al., 2015) such as CELEX (Baayen et al., 1995) or as an external analyser such as Morfessor (Creutz and Lagus, 2007), along with distributional information jointly at training time in the language modelling (LM) objective (Luong et al., 2013; Botha and Blunsom, 2014; Qiu et al., 2014; Cotterell and Schütze, 2015; Bhatia et al., 2016, i.a.). The key idea is to learn a morphological composition function (Lazaridou et al., 2013; Cotterell and Schütze, 2017) which synthesises the
representation of a word given the representations of its constituent morphemes. Contrary to our work, these models typically coalesce all lexical relations.
Another class of models, operating at the character level, shares a similar methodology: such models compose token-level representations from subcomponent embeddings (subwords, morphemes, or characters) (dos Santos and Zadrozny, 2014; Ling et al., 2015; Cao and Rei, 2016; Kim et al., 2016; Wieting et al., 2016; Verwimp et al., 2017, i.a.).
In contrast to prior work, our model decouples the use of morphological information, now provided in the form of inflectional and derivational rules transformed into linguistic constraints, from the actual training. This pipelined approach results in a simpler, more portable model. In spirit, our work is similar to Cotterell et al. (2016), who formulate the idea of post-training specialisation in a generative Bayesian framework. Their work uses gold morphological lexicons; we show that competitive performance can be achieved using a nonexhaustive set of simple rules. Our framework facilitates the inclusion of antonyms at no extra cost and naturally extends to constraints from other sources (e.g., WordNet) in future work. Another practical difference is that we focus on similarity and evaluate morph-fitting in a well-defined downstream task where the artefacts of the distributional hypothesis are known to prompt statistical system failures.
7 Conclusion and Future Work
We have presented a novel morph-fitting method which injects morphological knowledge in the form of linguistic constraints into word vector spaces. The method makes use of implicit semantic signals encoded in inflectional and derivational rules which describe the morphological processes in a language. The results in intrinsic word similarity tasks show that morph-fitting improves vector spaces induced by distributional models across four languages. Finally, we have shown that the use of morph-fitted vectors boosts the performance of downstream language understanding models which rely on word representations as features, especially for morphologically rich languages such as German.
Future work will focus on other potential sources of morphological knowledge (Soricut and Och, 2015), porting the framework to other morphologically rich languages and downstream tasks, and on further refinements of the post-processing algorithm and constraints selection.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
The authors propose  morph-fitting , a method that retrofits any given set of trained word embeddings based on a morphologically-driven objective that (1) pulls inflectional forms of the same word together (as in  slow  and  slowing ) and (2) pushes derivational antonyms apart (as in  expensive  and  inexpensive ). With this, the authors aim to improve the representation of low-frequency inflections of words as well as mitigate the tendency of corpus-based word embeddings to assign similar representations to antonyms. The method is based on relatively simple manually-constructed morphological rules and is demonstrated on both English, German, Italian and Russian. The experiments include intrinsic word similarity benchmarks, showing notable performance improvements achieved by applying morph-fitting to several different corpus-based embeddings. Performance improvement yielding new state-of-the-art results is also demonstrated for German and Italian on an extrinsic task - dialog state tracking.   Strengths:  - The proposed method is simple and shows nice performance improvements across a number of evaluations and in several languages. Compared to previous knowledge-based retrofitting approaches (Faruqui et al., 2015), it relies on a few manually-constructed rules, instead of a large-scale knowledge base, such as an ontology.  - Like previous retrofitting approaches, this method is easy to apply to existing sets of embeddings and therefore it seems like the software that the authors intend to release could be useful to the community.  - The method and experiments are clearly described. u2028  Weaknesses:  - I was hoping to see some analysis of why the morph-fitted embeddings worked better in the evaluation, and how well that corresponds with the intuitive motivation of the authors.   - The authors introduce a synthetic word similarity evaluation dataset, Morph-SimLex. They create it by applying their presumably semantic-meaning-preserving morphological rules to SimLex999 to generate many more pairs with morphological variability. They do not manually annotate these new pairs, but rather use the original similarity judgements from SimLex999. The obvious caveat with this dataset is that the similarity scores are presumed and therefore less reliable. Furthermore, the fact that this dataset was generated by the very same rules that are used in this work to morph-fit word embeddings, means that the results reported on this dataset in this work should be taken with a grain of salt. The authors should clearly state this in their paper.  - (Soricut and Och, 2015) is mentioned as a future source for morphological knowledge, but in fact it is also an alternative approach to the one proposed in this paper for generating morphologically-aware word representations. The authors should present it as such and differentiate their work.  - The evaluation does not include strong morphologically-informed embedding baselines.   General Discussion:  With the few exceptions noted, I like this work and I think it represents a nice contribution to the community. The authors presented a simple approach and showed that it can yield nice improvements using various common embeddings on several evaluations and four different languages. I d be happy to see it in the conference.  Minor comments:  - Line 200: I found this phrasing unclear:  We then query  u2026 of linguistic constraints d.  - Section 2.1: I suggest to elaborate a little more on what the delta is between the model used in this paper and the one it is based on in Wieting 2015. It seemed to me that this was mostly the addition of the REPEL part.  - Line 217:  The method  cost function consists of three terms d - I suggest to spell this out in an equation.  - Line 223:  x and t in this equation (and following ones) are the vector representations of the words. I suggest to denote that somehow. Also, are the vectors L2-normalized before this process? Also, when computing  nearest neighbor  examples do you use cosine or dot-product? Please share these details.  - Line 297-299: I suggest to move this text to Section 3, and make the note that you did not fine-tune the params in the main text and not in a footnote.  - Line 327: (create, creates) seems like a wrong example for that rule. u2028  * I have read the author response