How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non–linearly encoded in the embedding space, which questions the cosine–based, unsupervised, evaluation methods. All results and analysis scripts are available online.
Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning).
In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non–linearly encoded in the embedding space, which questions the cosine–based, unsupervised, evaluation methods. All results and analysis scripts are available online.
1 Introduction
Using word embeddings remains a standard practice in modern NLP systems, both in shallow and deep architectures [Goldberg, 2015]. By encoding information about words in a relatively simple algebraic structure [Arora et al., 2016] they enable
fast transfer to the task of interest1. The importance of word representation learning has lead to developing multiple algorithms, but lack of principled evaluation hinders moving the field forward, which motivates developing more principled ways of evaluating word representations. Word embeddings are not only hard to evaluate, but also challenging to train. Recent practice shows that one often needs to tune algorithm, corpus and hyperparameters towards the target task [Lai et al., 2016, Sharp et al., 2016b], which challenges the promise of broad applicability of unsupervised pretraining.
Evaluation methods of word embeddings can be roughly divided into two groups: extrinsic and intrinsic [Schnabel and Labutov, 2015]. In the former approach embeddings are used in a downstream task (eg. POS tagging), while in the latter embeddings are tested directly for preserving syntactic of semantic relations. The most popular intrinsic task is Word Similarity (WS) which evaluates how well dot product between two vectors reproduce score assigned by human annotators. Intrinsic evaluations always assume a very specific model for recovering given property.
Despite popularity of word embeddings, there is no clear consensus what evaluation methods should be used, and both intrinsic and downstream evaluations are criticized [Tsvetkov et al., 2015a, Faruqui et al., 2016]. On top of that, different evaluation schemes usually lead to different rankings of embeddings [Schnabel and Labutov, 2015]. For instance, it has been shown [Baroni and Dinu, 2014] that neural-based word embeddings perform consistently better then count-based models and later, using WS and WA tasks, it was argued otherwise [Levy et al., 2015]. Recent research in evaluation methods focuses on representative or inter-
1Algebraic structure refers to the fact that words can be decomposed into a overcomplete basis, such that each word can be expressed as a sparse sum of base vectors
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
pretable set of tasks [Nayak et al., 2016, Köhn, 2015a], analysing intrinsic evaluation [Chiu et al., 2016, Faruqui et al., 2016], as well as proposing improvements to intrinsic evaluation [Avraham and Goldberg, 2016, Tsvetkov et al., 2015b].
In this paper we employ a transfer learning view, in which the main goal of representation learning is to make subsequent learning fast, i.e. use resulting word embeddings to maximize performance at the lowest sample complexity possible [Bengio et al., 2013, Glorot et al., 2011]2. Surprisingly, researchers rarely report model (using given word representation) performance under varying (benchmark) dataset sizes and model classes3, which is crucial for correct evaluation of transfer, especially given increasing importance of small data regime applications. Motivated by this, we propose an evaluation focused on data efficiency. To quantify precisely accessible information, we additionally propose focusing only on simple (supervised) tasks, as complex downstream tasks are challenging to interpret. In addition, we propose principled improvements to WS and WA tasks, which try to address some of the critiques both benchmarks have received in the literature [Faruqui et al., 2016], in authors’ opinion mostly due to their purely unsupervised nature.
2 Proposal
Our main goal is to better align evaluation of word embeddings with their transfer application. Future performance is correlated with the amount of easily accessible and useful information. By easily accessible information, we mean information that model can quickly learn to use. Useful information is defined as one that correlates well with the final task.
First argument for data efficiency focused evaluation is the growing evidence that pretrained word embeddings provide little benefit under various settings, especially deep learning models [Zhang et al., 2015, Zhang and Wallace, 2015, Andreas and Klein]. We hypothesize that most of the improvements (in downstream tasks) reported in literature are caused by small size of the supervised dataset, which is reasonable from the trans-
2Alternative goals might include maximizing interpretability, or analysing unsupervised corpora.
3What is claimed here is that vast majority of papers doesn’t take into consideration those factors. Nevertheless, there are notable exceptions [Andreas and Klein, Qu et al., 2015, Amir et al., 2017].
fer learning point of view. Therefore, measuring performance after seeing just a subset of the supervised dataset is crucial for comparing word embeddings. Another argument is the empiricial difference between how easily accessible is the information in various embeddings. As our experiment show, commonly used dense representations achieve different learning speeds. This effect should be even stronger for sparse representations, for which feature dimensions can have very strict semantic meaning [Faruqui and Dyer, 2015a]. An argument can be also made from theoretical point of view; it is easy to show that any injective (and thus not losing information) embedding preserves all information about corpora (see Appendix for details), i.e. having enough training data makes embeddings dispensable.
Second part of the proposal is to focus on simple supervised tasks to directly evaluate useful information content. In certain applications, like tagging, choosing the right, specialized, word embeddings is crucial for obtaining state of the art results [Sharp et al., 2016a, Lample et al., 2016]. We also confirm empirically that word embeddings trade off capacity between different information. In this work we pose hypothesis, that specialization of word embeddings can be best evaluated by checking what simple information is most easily recoverable. While word level classification problems (like noun classification) were proposed previously [Köhn, 2015b], here we also suggest including tests for recovery of relations between words (exemplified in experiments by Similarity and Analogy tasks) .
Importance of simple supervised tasks can be also seen in the light of algebraic structure that is encoded in word representation space. It has been observed in practice that word embeddings have useful information only in a small subspace [Sattigeri and Thiagarajan, 2016, Rothe and Schütze, 2016, Astudillo et al., 2015]. Thus, simple supervised tasks are closely aligned with the actual use of word embeddings and allow to quantify how quickly model can extract the most salient subspace (which leads to faster learning in general).
Our final remark is about diversity of models. Commonly used WS and WA datasets are solved by a constant models, i.e. model which does not learn from the data. We argue that such evaluation is not generally informative. If we are interested in how well our vector space helps solving a given
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Figure 1: Vertical axis is normalized difference between score of the best performing non-linear model and the best performing linear model (the higher, the better non-linear model is). Each box plot represents a single tested embedding.
problem, we should in theory fit all possible models and pick the one that has the best generalization capabilities. While this is impractical, it illustrates that fixing one specific model gives answer to a different question, thus drawing general conclusions from it can be highly biased. A good rule of thumb might be to include representatives of typical model classes, or at least match the model with class of models we are interested in (which rarely will be constant), which concludes our guidelines for a correct evaluation.
We leave out details from the proposal how to order embeddings, as this is determined by the specific research question given evaluation should answer. A sensible default is to report AUC of learning curve for each task, and pick set of tasks that are most interesting to the researcher.
To summarize:
• Evaluation should focus on data efficiency (if transfer is the main goal of representation learning).
• Tasks should be supervised and simple.
• Unless focus is on specific application, evaluation should focus on a diverse set of models (including nonlinear and linear ones) and datasets (testing for various information content).
If we follow those guidelines, we truly approximate (for a given trained embedding) generalization error under distribution of tasks, dataset size and classifiers,
E [ Lt(cl(VU(Xtm),Y t m)(VU(x),y) ] , (1)
VU denotes the embeddings trainedw on U, Lt denotes task t loss and expectation is taken with respect to:
• p(cl) – distribution of classifiers,
• p(Xtm,Ytm) – distribution of training datasets, where we first sample task t and then uniformly sample dataset Xtm of size m.
• x,y – i.i.d. examples following training data distribution.
Distribution over classifiers and tasks should be carefully tuned to researcher’s needs, as we will argue soon. Further theoretical analysis is included in Appendix, and in the rest of the paper we present practical arguments for the proposed evaluation scheme.
3 Experiments
In this section we define specific metrics and tasks and perform exemplary evaluation of several pretrained embeddings in the advocated setting. Specifically, we try to empirically answer several questions, all geared towards providing experimental validation for the three main points of proposal:
• Do supervised versions of WA and WS benchmarks provide additional insights?
• How stable is the ranking of embeddings under changing dataset size?
• Are there embeddings that benefit from nonlinear models?
The first question will aid understanding how useful are simple supervised tasks coupled with data efficiency. Second question shows that ranking of embeddings do change under transfer learning evaluation. Last question explores if any embeddings encode information in a “non-linear” fashion; while one of the main goals of representation learning is disentangling factors of variation, usually learned representations are entangled and dense, which poses interesting question how hard it is to extract useful patterns from them. In this paper we report only a subset of results with the most interesting conclusions, all results (along with code) are also made available online for further analysis.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Figure 2: Accuracy reached by 3 different models optimizing word similarity on SimLex dataset. Rightmost plot deptics performance of the traditionally used constant model. While NMT has been reported to have strong performance on SimLex (as shown on the rightmost plot), its relative gains diminish under supervised version of the benchmark (leftmost and central plot). Tested embeddings (ordered by performance on the rightmost plot): NMT, GloVe300, HPCAautoenc.
3.1 Datasets and models
Datasets are divided into 4 categories: Similarity, Analogy, Sentence and Single word. Analogy datasets are composed of quadruples (two pairs of words in a specific relation, for instance (king, queen, man, woman)). Similarity datasets are composed of pairs of words and assigned mean rank by human annotators. Sentence and Single word datasets have binary targets. In total our experimentation include 15 datasets:
• Similarity: SimLex999 [Hill et al., 2015], MEN [Bruni et al., 2014], WordSimilarity353 [Finkelstein et al., 2001] and Rare Words [Luong et al., 2013].
• Analogy: 4 categories from WordRep [Gao et al., 2014]4.
• Sentence: Stanford Sentiment Treebank [Socher et al., 2013] and News20 (3 binary datasets) [Tsvetkov et al., 2015a].
• Single word: Datasets constructed from lexicons collected in [Faruqui and Dyer, 2015b]: POS tagging (3 datasets for verb, noun and adjective), word sentiment (1 dataset), word color association (1 dataset) and WordNet synset membership (2 datasets).
Models for each datasets include both nonlinear and linear variants. When model is non-
4We experimented with MSR and Google datasets and observed that models easily overfit if the train and test sets share the same words (not 3-tuples). WordRep dataset is a set of pairs which we split into disjoint sets.
linear, for robustness we include in search a fallback to a simpler linear or constant model. Additionally, in the case of Similarity and Analogy we include commonly used constant models. Similarity between 2 vectors is approximated by their cosine similarity (cos(~v1, ~v2)). In the case of Analogy tasks embedding is evaluated for its ability to infer 4th word out from the first three and we use the following well-known constant models: 3COSADD (argmax~v∈V cos(~v, ~v2− ~v1+ ~v3)) and 3COSMUL (argmax~v∈V
ccos(~v, ~v3)ccos(~v, ~v2) ccos(~v, ~v1)+ )5. For each task class we evaluate a different set of classifiers:
• Similarity: cosine similarity, Random Forest (RF), Support Vector Regression (SVR) with RBF kernel6.
• Analogy: 3COSADD, 3COSMUL [Levy et al., 2015] and regression neural network (performing regression on the 4th word given the rest of the quadruple, see Appendix for further information).
• Sentence: Logistic Regression, Support Vector Machine (SVM) with RBF kernel taking as input averaged embedding vector and Convolutional Neural Network (CNN) [Kim, 2014] taking as input concatenation of embedding vectors.
• Single word: RF, SVM (with RBF kernel), Naive Bayes, k-Nearest Neighbor Classifier and Logistic Regression.
3.2 Embeddings
Our objective was to cover representatives of embeddings emerging from both shallow and deeper architectures. Deep embeddings are harder to train, so for the scope of this paper we decided to include pretrained and publicly available vectors7. Setup includes following “shallow” pretrained embeddings: GloVe (100 and 300 dimensions) [Pennington et al., 2014], Hellinger PCA (HPCA) [Lebret and Collobert, 2014], PDC (100 and 300 dimensions) and HDC (300 dimensions) [Sun et al., 2015], Additionally following “deep” embeddings are evaluated: Neural Translation Machine (NMT,
5ccos(~v1, ~v2) = 1+cos( ~v1, ~v2) 2 6We also tried RankSVM [Lee and Lin, 2014], but it did not perform better than other models, while being very computationally intensive.
7Vocabularies were lowercased and intersected before perfoming experiments. Vectors were normalized to a unit length.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
rank start rank end auc rank
GloVe100 2.09± 1.76 2.48± 1.55 2.17± 1.77 GloVe300 1.83± 2.63 1.91± 2.54 1.87± 2.57 HDC300 0.93± 1.34 1.06± 1.22 0.91± 1.40 HPCAautoenc 3.52± 2.04 3.68± 1.67 3.65± 2.03 HPCA 4.77± 2.33 4.75± 2.23 4.83± 2.29 NMT 3.90± 2.24 4.32± 1.93 4.28± 2.34 PDC300 0.64± 0.89 0.74± 1.07 0.65± 1.10 morphoRNNLM 4.13± 2.34 4.55± 2.02 4.26± 2.29
Table 1: First two column present ranks at the 30% and 100% splits of evaluation dataset averaged over all categories. Third column is rank computed by recommended default AUC of curve. As expected, when averaging over many tasks (of different dataset sizes), data efficiency is not changing final ordering (third column). On average rank increases as embeddings are becoming significantly different (as determined by ANOVA during rank computation).
activations of the deep model are extracted as word embeddings) [Hill et al., 2014], morphological embeddings (morph, which can learn morphological differences between words directly) [Luong et al., 2013] and HPCA variant trained using autoencoder architecture [Lebret and Collobert, 2015]. In some experiments we additionally include publicly available pretrained skip-gram embeddings on Google News corpora and skipn-gram embeddings trained on Wikipedia corpora [Ling et al., 2015] (used commonly in syntax demanding tasks, like tagging).
3.3 Results
For each dataset we first randomly select test set and run evaluation for increasing sizes of training dataset, thus scores approximate generalization error after seeing increasing amounts of data. Splits are repeated 6 times in total to reduce noise. Thus, for each task results are 6 learning curves, with a score for each subset of data (see Fig. 2).
Ranks of embeddings at each point are calculated using a greedy sequential procedure, where we assign embeddings the same rank if their scores (each point on the curve is represented by 6 scores) are not significantly different, as tested using pairwise ANOVA test. All results are available online8.
3.4 Learnable Similarity and Analogy tasks
Our first question was validating that adding learnable Similarity and Analogy tasks introduce any
8Results will be posted online upon publication.
additional insights. Positive answer to this question motivates introduction of simple tasks with varying dataset size, ideally defined on single or pair of words.
For solving Analogy we implemented a shallow neural network. Interestingly, WordRep authors [Gao et al., 2014] reported low accuracies (often even below 5%) on most analogy questions and we were able to improve absolute score upon the tested subset on average by absolute 11% (see Tab. 2). Having learnable models for Similarity and Analogy datasets enables reusing many publicly available datasets in the new context. Also, we can robustly evaluate if given information about relation between two words is present in the embedding based on analogy questions. In the case of HASCONTEXT and INSTANCEOF datasets no embeddings can recover analogy answers using static models (achieved accuracy is below 3%), but actually some embeddings do have information about the relations. In both cases HDC consistently outperforms other embeddings reaching around 25% accuracy, see Fig. 4 and Tab. 2.
In the case of Similarity dataset the best performing model was Support Vector Regression, similarly as in Analogy datasets we also improve over the constant models. What is more, we can draw novel conclusions. Interesting example is NMT performance on SimLex. It was claimed in [Hill et al., 2014] that NMT embeddings are better at encoding true similarity between words, but SVR on Glove embeddings performs better after training on the whole dataset (i.e. at the end of learning curve), see Fig.2.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
rank start rank end auc rank
GloVe100 1.80± 2.04 2.31± 1.83 1.86± 2.07 GloVe300 3.06± 3.17 3.14± 2.99 3.03± 3.09 HDC300 1.37± 1.68 1.37± 1.40 1.31± 1.69 HPCAautoenc 3.63± 2.30 3.80± 1.98 3.83± 2.28 HPCA 3.77± 2.77 3.60± 2.58 3.80± 2.79 NMT 3.74± 2.05 4.29± 1.74 4.23± 2.30 PDC300 0.60± 0.77 0.60± 1.01 0.51± 0.95 morphoRNNLM 3.00± 2.38 3.60± 2.16 3.17± 2.38
Table 3: First two column present ranks at the 30% and 100% splits of evaluation dataset averaged over all tasks for Single word datasets. Third column is rank computed by recommended default AUC of curve. Interestingly, information stored in Single word datasets is easily accessible even with small amount of data.
3.5 Rank stability under changing dataset size
Second question was how stable are the orderings under growing dataset. To this end we have measured rank at the beginning (30% of data) and end of training. Mean absolute value of change of ranking is approximately (averaged over all categories) 0.6 with standard deviation of 0.2. This means that usually an embeddings has a changed rank after training, which establishes usefulness of measuring data efficiency for the tested embeddings.
Interestingly, when averaged over many experiments, final ordering of embeddings tends to be similar, see Tab. 1 and Fig. 3. This is mainly because (tested) embeddings have different data efficiency properties for different tasks, i.e. none of embeddings is consistently more data efficient than others. On top of that standard deviation of both rank at the end and beginning is around 2.5, which further reinforces findings from [Schnabel and Labutov, 2015] that embeddings orderings are very task dependent.
Measuring data efficiency is crucial for a realistic (i.e. as close to application as possible) evaluation of representations. Besides a more accurate and practical ordering of embeddings, it also allows one to draw new conclusions, which is exemplified by differences between GloVe100 and GloVe300 (elaborated on in the next section). Another interesting point is that rank change after training on full dataset is relatively low for Single word datasets, which suggests that simple information about words like noun or verb is always quickly accessible to models, but more complicated information like relationships or similarities
between pair of words are not, see Tab. 3.
3.6 Linear vs non–linear models
Our last question was how stable is the ordering under changing model type. More specifically, are there embeddings especially fitted for use with linear models? Clearly some embeddings in fact are, see Fig. 1. This is an important empirical fact for practictioners, which motivates including such evaluation in experiments. In particular, it clearly shows that typically used evaluation does not answer the question “is there information about task X in the embedding Y” but only “is information about task X stored in embedding Y easily separable by a static (or linear) classifier”.
An illustrative example is the difference in performance between two pretrained GloVe embeddings of different dimensionality (100 and 300). It has been shown previously that lower dimensional GloVe embeddings are better at syntactic tasks [Lai et al.], but our evaluation reveals more complicated picture, that GloVe objective might encourage some sort of nonlinear encoding. We can see that by significantly better rank of GloVe100 at the beginning of learning of Single word datasets (mean rank 1.8), but lower at the end (mean rank 2.3), see Tab. 3. This is also visible when averaged over all categories, see Tab. 1.
3.7 Discussion
Performed eperiments show usefulness of the additional and more granular level of analysis enabled. Researcher can ask more precise questions, like “is it worth fitting syntax specific embeddings even when supervised dataset size is large?” (to which answer is positive based on our experiments) or “is HASINSTANCE relation encoded in the space?” (to which answer is also positive for some embeddings). Unfortunately, there is already a large volatility of final embeddings ordering when using standard evaluation, and our proposed scheme at times makes it even more challenging to decide which embeddings are optimal. This hints, that purely unsupervised large scale pretraining might not be suitable for NLP applications. Most importantly, evaluation should be more targeted, either at some specific application area, or at specific properties of representation.
One of the presented arguments for including supervised models for testing information content is algebraic interpretation of word embeddings [Arora et al., 2016]. The algebraic structure
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Figure 3: Rank at the beginning and end of learning for a subset of tested embedding (horizontal axis) and category (color). Task categories from left to right are: Sentence (blue), Single word (green), Similarity (red) and Analogy (purple).
Figure 4: Accuracy reached by Neural Network regression model (left plot) and 3COSMUL (right plot) optimizing analogy (HASCONTEXT relation from WordRep benchmark [Gao et al., 2014]).
present in the representation space enables one to decompose word embedding space into a set of concepts (so each word vector can be well approximated by a sum of few concepts)9. Theoretically, tasks defined on single words should test for existence of such concepts, but in our case including (supervised) Analogy tasks was very useful, as those tasks are still very challenging for current embeddings. For Analogy tasks (see Fig. 4) achieved accuracy scores are below 25%, whereas in the case of Single word average accuracy is around 80% (and fitting classifier adds on average only 2%). These higher order (or subspace) focused tasks are also well aligned with the application of word embeddings, because empirically models tend to focus on a small subspace in the vector space.
9This decomposition can be obtained using standard methods like k-SVD.
4 Conclusions
As exemplified by experiments, proposed evaluation reveals differences between embeddings along usually overlooked dimensions: data efficiency, non-linearity of downstream model and simple supervised tasks (including recovery of higher order relations between words). Interesting new conclusions can be reached, including differences between different size GloVe embeddings or performance of non-linear models on similarity benchmarks.
Additionally, obtained results reinforce conclusions from other published studies that there are no universally good embeddings and finding such might not be achievable, or a well posed problem. One should take great care when designing evaluation and specify what is the main focus. For instance, if the main goal of the word embeddings is to be useful in transfer, one should include advocated data efficiency metrics. New word embedding algorithms are moving away from typical pretraining scheme, with increasing focus on specialized word embeddings and applications under very limited dataset size, where fast learning is crucial. We hope that proposed evaluation methodology will help advance research in these scenarios.
A Theoretical analysis
Let us first try to answer the question what is the information about the task and how can it be measured given some data representation. For simplicity let us assume that task is a binary classification, but the same reasoning applies to multiclass, multilabel, regressions etc. A quite natural, machine learning perspective is to define information stored as a Bayes risk of optimal model trained to perform this task. Obviously, raw representation already has some non-negative Bayes risk, which cannot be reduced during any embedding. Actually, it is quite easy to show, that nearly every embedding preserves all the information contained in the source representation.
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
Observation 1. Every injective embedding preserves all the information about the task.
Proof. Let us assume that Bayesian optimal classifier (the one obtaining the Bayes risk RX ) on the input space X be called o. Furthermore, let our embedding (learned in arbitrary manner, supervised or not) be a function E : X → X ′. According to assumptions, E is injective, thus for every x ∈ X there exists unique x′ ∈ X ′ such that E(x) = x′. Let us call the corresponding inverse assignment E−1 (defined only on the image of E). Consequently we can define classifier on E(X) ⊂ X ′ through o′(x′) = o(E−1(x′)). It is now easy to show, that Bayes risks of these two models are exactly the same
RX = ∫ ∑
y
`(o(x), y)p(y|x)p(x)dx
= ∫ ∑ y `(o(E−1(E(x))), y)p(y|x)p(x)dx
= ∫ ∑ y `(o′(E(x), y)p(y|x)p(x)dx
= RX ′ .
This remains an open question how frequent in general are injective embeddings. If one considers continuous spaces as X then this is extremely small class of functions (especially if dim(X ′) ≤ dim(X )). However in case of natural language processing (and many other fields), the input space is actually discrete or even finite. In such case, non-injective embeddings are rare phenomenon. In particular, for any finite set, probability of selecting at random linear projection which gives non-injective embedding is zero.
The above reasoning is in some sense trivial, yet still worth underlying, as it gives an important notion of what should be measured when evaluating embeddings. Even though Bayes risk is the same for both spaces, the complexity of inferring o′ can be completely different from complexity of inferring o. We argue, that this is a crucial element - to measure how hard is to learn o′ (or any reasonable approximation). There are two basic dimensions of such analysis:
• check how complex the set of hypotheses H 3 o′ needs to be in order to be able to find it using given data,
• verify how well one can approximate o′ as a function of growing training size. In other words - how fast an estimator of o′ converges.
Thus, to really distinguish various embeddings, we should rather ask what is the best achievable performance under limited amount of data or under constrained class of models, which is theoretical argument for data efficiency oriented evaluation.
B Regression neural network for word analogy task
Let D be the dimension of a given word embedding. We assume that all embedded words have euclidean norm equal to one – this guarantees that the scalar product of two embedded words is also their cosine similarity. The word analogy task is defined in the following way: given (embedded) words v1, v2 and v3, predict word v4 that satisfies analogy “v1 is related to v2 as v3 is related to v4”. Our estimator of v4 is defined as:
v̂4 = −W1v1 +W2v2 +W3v3 + b ‖−W1v1 +W2v2 +W3v3 + b‖2
where the model trainable parameters are:
• W1, W2 and W3 – D×D matrices initialized with identities,
• b – D-dimensional vector initialized with zeros,
and the cost is defined as: − ∑ j 〈vj4, v̂ j 4〉.
The model was trained with gradient descent optimization on minibatches. Hyperparameters: learning rate, number of epochs, optimizer, batch size and (boolean) fallback to constant model were chosen using cross-validation. The actual prediction has two steps:
• calculate v̂4,
• choose (embedded) word v that minimizes 〈v, v̂4〉.
Observe that this model is initialized in such a way, that it is equivalent to 3COSADD – the idea is to check, if applying trainable affine transformations to input vectors would boost 3COSADD performance. It should also be noted that this approach turned out to be very computationally intensive.
This paper proposes a framework for evaluation of word embeddings based on data efficiency and simple supervised tasks. The main motivation is that word embeddings are generally used in a transfer learning setting, where evaluation is done based on how faster is to train a target model. The approach uses a set of simple tasks evaluated in a supervised fashion, including common benchmarks such as word similarity and word analogy. Experiments on a broad set of embeddings show that ranks tend to be task-specific and change according to the amount of training data used.  Strengths  - The transfer learning   data efficiency motivation is an interesting one, as it directly relates to the idea of using embeddings as a simple  semi-supervised  approach.  Weaknesses  - A good evaluation approach would be one that propagates to end tasks. Specifically, if the approach gives some rank R for a set of embeddings, I would like it to follow the same rank for an end task like text classification, parsing or machine translation. However, the approach is not assessed in this way so it is difficult to trust the technique is actually more useful than what is traditionally done. - The discussion about injective embeddings seems completely out-of-topic and does not seem to add to the paper's understanding. - The experimental section is very confusing. Section 3.7 points out that the analysis results in answers to questions as  is it worth fitting syntax specific embeddings even when supervised datset is large?  but I fail to understand where in the evaluation the conclusion was made. - Still in Section 3.7, the manuscript says  This hints, that purely unsupervised large scale pretraining might not be suitable for NLP applications . This is a very bold assumption and I again fail to understand how this can be concluded from the proposed evaluation approach. - All embeddings were obtained as off-the-shelf pretrained ones so there is no control over which corpora they were trained on. This limits the validity of the evaluation shown in the paper. - The manuscript needs proofreading, especially in terms of citing figures in the right places (why Figure 1, which is on page 3, is only cited in page 6?).  General Discussion  I think the paper starts with a very interesting motivation but it does not properly evaluate if their approach is good or not. As mentioned above, for any intrinsic evaluation approach I expect to see some study if the conclusions propagate to end tasks and this is not done in the paper. The lack of clarity and proofreading in the manuscript also hinders the understanding. In the future, I think the paper would vastly benefit from some extrinsic studies and a more controlled experimental setting (using the same corpora to train all embeddings, for instance). But in the current state I do not think it is a good addition to the conference.