In recent years word-embedding models have gained great popularity due to their remarkable performance on several tasks, including word analogy questions and caption generation. An unexpected “sideeffect” of such models is that their vectors often exhibit compositionality, i.e., adding two word-vectors results in a vector that is only a small angle away from the vector of a word representing the semantic composite of the original words, e.g., “man” + “royal” = “king”. In this work we provide a mathematical formalism for compositionality, and a theoretical justification for the presence of additive compositionality in word vectors learned using the Skip-Gram model. In particular, we show that additive compositionality holds in an even stricter sense (small distance rather than small angle) under certain assumptions on the process generating the corpus. As a corollary, we explain the success of vector calculus for solving word analogies. When these assumptions do not hold, we show that compositionality is no longer additive, and provide the correct composition operator. Finally, we establish a connection between the Skip-Gram model and the Sufficient Dimensionality Reduction (SDR) framework of Globerson and Tishby. SDR models provide information-theoretically optimal symbol-embeddings for problems where the training data takes the form of co-occurrence statistics. We prove that the parameters of Skip-Gram models can be readily modified to obtain the parameters of SDR models by simply adding information on symbol frequencies. This shows that the Skip-Gram model is essentially learning optimal word embeddings in the sense of Globerson and Tishby. Further, it implies that the heuristics commonly used to approximately fit Skip-Gram models can be used to fit SDR models.
In this work we provide a mathematical formalism for compositionality, and a theoretical justification for the presence of additive compositionality in word vectors learned using the Skip-Gram model. In particular, we show that additive compositionality holds in an even stricter sense (small distance rather than small angle) under certain assumptions on the process generating the corpus. As a corollary, we explain the success of vector calculus for solving word analogies. When these assumptions do not hold, we show that compositionality is no longer additive, and provide the correct composition operator.
Finally, we establish a connection between the Skip-Gram model and the Sufficient Dimensionality Reduction (SDR) framework of Globerson and Tishby. SDR models provide information-theoretically optimal symbol-embeddings for problems where the training data takes the form of co-occurrence statistics. We prove that the parameters of Skip-Gram models can be readily modified to obtain the parameters of SDR models by simply adding informa-
tion on symbol frequencies. This shows that the Skip-Gram model is essentially learning optimal word embeddings in the sense of Globerson and Tishby. Further, it implies that the heuristics commonly used to approximately fit Skip-Gram models can be used to fit SDR models.
1 Introduction
The idea of representing words as vectors has a long history in computational linguistics and machine learning. The general idea is to find a map from words to vectors such that wordsimilarity and vector-similarity are in correspondence. Whilst vector-similarity can be readily quantified in terms of distances and angles, quantifying word-similarity is a more ambiguous task. A key insight in that regard is to posit that the meaning of a word is captured by “the company it keeps” (Firth, 1957) and, therefore, that two words that keep company with similar words are likely to be similar themselves. To break the cyclicality of this definition it is common to consider models that only attempt to capture pairwise cooccurrence statistics1.
In the simplest case, one seeks vectors whose similarity approximates the co-occurrence frequencies. In more sophisticated methods cooccurrences are reweighed to suppress the effect of more frequent words (Rohde et al., 2006) and/or to emphasize pairs of words whose co-occurrence frequency maximally deviates from the independence assumption (Church and Hanks, 1990).
An alternative to seeking word-embeddings that reflect co-occurrence statistics is to extract the
1Normally, if a is similar to b and b is similar to c, it should be that a is more similar to c than it is to a random word. But pairwise-similarity models do not explicitly require that, i.e., do not penalize model parameters under which a and c are not particularly similar.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
vectorial representation of words from non-linear statistical language models, specifically neural networks. (Bengio et al., 2003) already proposed (i) associating with each vocabulary word a feature vector, (ii) expressing the probability function of word sequences in terms of the feature vectors of the words in the sequence, and (iii) learning simultaneously the vectors and the parameters of the probability function. This approach came into prominence recently through works of Mikolov et al. (see below) whose main departure from (Bengio et al., 2003) was to follow the suggestion of (Mnih and Hinton, 2007) and tradeaway the expressive capacity of general neuralnetwork models for the scalability to very large corpora afforded by the more restricted class of log-linear models.
An unexpected side effect of deriving wordembeddings via neural networks is that the wordvectors produced appear to enjoy (approximate) additive compositionality: adding two wordvectors often results in a vector whose nearest word-vector is of the word capturing the composition of the added words, e.g., “man” + “royal” = “king” (Mikolov et al., 2013c). This unexpected property allows one to use these vectors to answer word-analogy questions algebraically, e.g., answering the question “Man is to king as woman is to ?” by returning the word whose word-vector is nearest to the vector
v(king) - v(man) + v(woman).
In this work we focus on explaining the source of this phenomenon for the most prominent such model, namely the Skip-Gram (SG) model introduced in (Mikolov et al., 2013a). The SG model learns vector representations of words based on their patterns of co-occurrence in the training corpus as follows: it assigns to each word c in the vocabulary V , a “context” and a “target” vector, respectively uc and vc, which are to be used in order to predict the words that appear around each occurrence of c within a window of ∆ tokens. Specifically, the log probability of any target word w to occur at any position within distance ∆ of a context word c is taken to be proportional to the inner product between uc and vw, i.e., letting n = |V |,
p(w|c) = e uTc vw∑n
i=1 e uTc vi
. (1)
Further, SG assumes that the conditional proba-
bility of each possible set of words in a window around a context word c factorizes as the product of the respective conditional probabilities:
p(w−∆, . . . , w∆|c) = ∆∏
δ=−∆ δ 6=0
p(wδ|c). (2)
(Mikolov et al., 2013a) proposed learning the SG parameters on a training corpus by using maximum likelihood estimation under (1) and (2). Thus, if wi denotes the i-th word in the training corpus and T the length of the corpus, we seek the word vectors that maximize
1
T T∑ i=1 ∆∑ δ=−∆ δ 6=0 log p(wi+δ|wi) . (3)
As mentioned, the normalized context vectors obtained from maximizing (3) under (1) and (2) exhibit additive compositionality. For example, the cosine distance between the sum of the context vectors of the words “Vietnam” and “capital” and the context vector of the word “Hanoi” is small.
While there has been much interest in using algebraic operations on word vectors to carry out semantic operations like composition, the only published work which attempts a rigorous theoretical understanding of this phenomenon is (Arora et al., 2016). This work guarantees that word vectors can be recovered by factorizing the so-called PMI matrix, and that algebraic operations on these word vectors can be used to solve analogies, under certain conditions on the process that generated the training corpus. Specifically, the word vectors must be known, a priori before their recovery, to have been generated by randomly scaling uniformly sampled vectors from the unit sphere2. Further, the ith word in the corpus must have been selected with probability proportional to eu T wci , where the “discourse” vector ci governs the topic of the corpus at the ith word. The discourse vector is assumed to evolve according to a random walk on the unit sphere that has a uniform stationary distribution.
By way of contrast, our results assume nothing about the properties of the word vectors a priori. In fact, the connection we establish between the Skip-gram and Sufficient Dimensionality Reduction model of (Globerson and Tishby, 2003)
2More generally, it suffices that the word vectors have certain properties consistent with this sampling process.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
shows that the word vectors learned by Skip-Gram are information-theoretically optimal. Further, the context word c in the Skip-Gram model essentially serves the role that the discourse vector does in the PMI model of (Arora et al., 2016): the words neighboring c are selected with probability proportional to eu T c vw . We find the exact non-linear composition operator when no assumptions are made on the context word. When an analogous assumption to that of (Arora et al., 2016) is made, that the context words are uniformly distributed, we prove that the composition operator reduces to vector addition.
While our primary motivation has been to provide a better theoretical understanding of the popular SG model, our connection with the SDR method opens up the possibility of practical applicability of our approach more generally. In particular, there is the question of whether, for a given corpus, fitting an SG model will give good embeddings—while we are making reasonable linguistic assumptions about how to model words and the interdependencies of words in a corpus, it’s not clear that these have to hold universally on all corpuses to which we apply SG. However, the fact that when we fit an SG model we are fitting an SDR model (up to frequency information), and the fact that SDR models are informationtheoretically optimal in a certain sense, argues that regardless of whether the SG assumptions hold, SG always gives us optimal features in the following sense: the learned context embeddings and target embeddings preserve the maximal amount of mutual information between any pair of random variables X and Y consistent with the observed co-occurence matrix, where Y is the target word and X is the predictor word (in a min-max sense, since there are many ways of coupling X and Y , each of which may have different amounts of mutual information). Importantly, this statement requires no assumptions on the distribution P (X,Y ).
2 Compositionality of Skip-Gram
In this section, we first give a mathematical formulation of the intuitive notion of compositionality of words. We then prove that the composition operator for the Skip-Gram model in full generality is a non-linear function of the vectors of the words being composed. Under a single simplifying assumption, the operator linearizes and reduces to
the addition of the word vectors. Finally, we explain how linear compositionality allows for solving word analogies with vector algebra.
A natural way of capturing the compositionality of words is to say that the set of context words c1, . . . , cm has the same meaning as the single word c if for every other word w,
p(w|c1, . . . , cm) = p(w|c) .
Although this is an intuitively satisfying definition, we never expect it to hold exactly; instead, we replace exact equality with the minimization of KL-divergence. That is, we state that the best candidate for having the same meaning as the set of context words C is the word
arg min c∈V
DKL(p(·|C) | p(·|c)) . (4)
We refer to any vector that minimizes (4) as a synonym of the set of words C.
There are two natural concerns with (4). The first is that, in general, it is not clear how to define p(·|C). The second is that KL-divergence minimization is a hard problem, as it involves optimization over many high dimensional probability distributions. Our main result shows that both of these problems go away for any language model that satisfies two assumptions:
A1. For every word c, there exists Zc such that for every word w,
p(w|c) = 1 Zc exp(uTc vw) . (5)
A2. For every set of words C = {c1, c2, . . . , cm}, there exists ZC such that for every word w,
p(w|C) = p(w) 1−m
ZC
m∏ i=1 p(w|ci) . (6)
Clearly, the SG model satisfies A1 by definition. We prove that it also satisfies A2 when m ≤ ∆ (Lemma 1).
Theorem 1. In every word model that satisfies A1 and A2, for every set of words C = {c1, . . . , cm}, the synonyms of C satisfy∑
w∈V p(w|c)vw = ∑ w∈V p(w|C)vw . (7)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Theorem 1 characterizes the composition operator for any language model which satisfies our two assumptions; in general, this operator is not addition. Instead, a synonym c is a vector such that the average word vector under p(·|c) matches that under p(·|C). When the expectations in (7) can be computed, the composition operator can be implemented by solving a non-linear system of equations to find a vector u for which the left-hand side of (7) equals the right-hand side.
Our next result proves that although the composition operator is nontrivial in the general case, to recover vector addition as the composition operator, it suffices to assume that the word frequency is uniform.
Theorem 2. In every word model that satisfies A1, A2, and where p(w) = 1/|V | for everyw ∈ V , the synonym of C = {c1, . . . , cm} is
u1 + . . .+ um .
As word frequencies are typically much closer to a Zipf distribution (Piantadosi, 2014), the uniformity assumption of Theorem 2 is not realistic. That said, we feel it is important to point out that, as reported in (Mikolov et al., 2013b), additivity captures compositionality more accurately when the training set is manipulated so that the prior distribution of the words is made closer to uniform.
Using composition to solve analogies It has been observed that word vectors trained using nonlinear models like SG tend to encode semantic relationships between words as linear relationships between the word vectors (Mikolov et al., 2013b; Pennington et al., 2014; Levy and Goldberg, 2014). In particular, analogies of the form “man:woman::king:?” can often be solved by taking ? to be the word in the vocabulary whose context vector has the smallest angle with uwoman + (uking−uman). Theorems 1 and 2 offer insight into the solution such analogy questions.
We first consider solving an analogy of the form “m:w::k:?”” in the case where the composition operator is nonlinear. The fact that m and w share a relationship means m is a synonym for the set of words {w,R}, where R is a set of words encoding the relationship between m and w. Similarly, the fact that k and ? share the same relationship means k is a synonym for the set of words {?, R}.
By Theorem 1, we have that R and ? must satisfy∑ `∈V p(`|m)v` = ∑ `∈V
p(`|w,R)v` and∑ `∈V p(`|k)v` = ∑ `∈V p(`|?, R)v`.
We see that solving analogies when the composition operator is nonlinear requires the solution of two highly nonlinear systems of equations. In sharp contrast, when the composition operator is linear, the solution of analogies delightfully reduces to elementary vector algebra. To see this, we again begin with the assertion that the fact that m and w share a relationship means m is a synonym for the set of words {w,R}; Similarly, k is a synonym for {?, R}. By Theorem 2,
um = uw + ur and
uk = u? + ur,
which gives the expected relationship
u? = uk + (uw − um).
Note that because this expression for u? is in terms of k, w, andm, there is actually no need to assume that R is a set of actual words in V .
2.1 Proofs
Proof of Theorem 1. Note that p(w|C) equals
p(w)1−m
ZC
m∏ i=1 p(w|ci)
= p(w)1−m
ZC exp ( m∑ i=1 uTcivw − m∑ i=1 logZci )
= 1
Z p(w)1−m exp(uTCvw) ,
where Z = ZC ∏m i=1 Zi, and uC = ∑m i=1 ui.
Minimizing the KL-divergence
DKL(p(·|c1, . . . , cm)‖p(·|c))
as a function of c is equivalent to maximizing the negative cross-entropy as a function of uc, i.e., as maximizing
Q(uc) = Z ∑ w exp(uTCvw) p(w)m−1 (uTc vw − logZc) .
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Since Q is concave, the maximizers occur where its gradient vanishes. As∇ucQ equals
Z ∑ w exp(uTCvw) p(w)m−1 [ vw − ∑n `=1 exp(u T c v`)v`∑n k=1 exp(u T c vk) ] = ∑n `=1 exp(u T c v`)v`∑n
k=1 exp(u T c vk) − Z ∑ w exp(uTCvw)vw p(w)m−1
= ∑ w∈V p(w|c)vw − ∑ w∈V p(w|c1, . . . , cm)vw ,
we see that (7) follows. Proof of Theorem 2. Recall that uC = ∑m
i=1 ui. When p(w) = 1/|V | for all w ∈ V , the negative cross-entropy simplifies to
Q(uc) = Z ∑ w exp ( uTCvw ) (uTc vw − logZc) ,
and its gradient∇ucQ to
Z ∑ w exp(uC Tvw) [ vw − ∑n `=1 exp(u T c v`)v`∑n k=1 exp(u T c vk) ] = Z
∑ w exp(uC Tvw)vw − ∑ w exp(uTc vw)vw .
Thus,∇Q(uC) = 0 and since Q is concave, uC is its unique maximizer.
Lemma 1. The SG model satisfies assumption A2 when m ≤ ∆.
Proof of Lemma 1. First, assume that m = ∆. In the SG model target words are conditionally independent given a context word, i.e.,
p(c1, . . . , cm|w) = m∏ i=1 p(ci|w).
Applying Baye’s rule,
p(w|c1, . . . , cm) = p(c1, . . . , cm|w)p(w)
p(c1, . . . , cm)
= p(w)
p(c1, . . . , cm) m∏ i=1 p(ci|w)
= p(w)
p(c1, . . . , cm) m∏ i=1 p(w|ci)p(ci) p(w)
= p(w)1−m
ZC
m∏ i=1 p(w|ci) , (8)
where ZC = 1/ ( ∏m i=1 p(ci)). This establishes the result when m = ∆. The cases m < ∆ follow by marginalizing out ∆ −m context words in the equality 8.
Projection of synonyms onto the vocabulary Theorem 2 states that if there is a word c in the vocabulary V whose context vector equals the sum of the context vectors of the words c1, . . . , cm, then c has the same “meaning”, in the sense of (4), as the composition of the words c1, . . . , cm. For any given set of words C = {c1, . . . , cm}, it is unlikely that there exists a word c ∈ V whose context vector is exactly equal to the sum of the context vectors of the words c1, . . . , cm. Similarly, in Theorem 1, the solution(s) to (7) will most likely not equal the context vector of any word in V . In both cases, we thus need to project the vector(s) onto words in our vocabulary in some manner.
Since Theorem 1 holds for any prior over V , in theory, we could enumerate all words in V and find the word(s) that minimize the difference of the left hand side of (7) from the right hand side. In practice, it turns out that the angle between the context vector of a word w ∈ V and solutionvector(s) is a good proxy and one gets very good experimental results by selecting as the synonym of a collection of words, the word that minimizes the angle to the synonym vector.
Minimizing the angle has been empirically successful at capturing composition in multiple loglinear word models. One way to understand the success of this approach is to recall that each word c is characterized by a categorical distribution over all other words w, as stated in (1). The peaks of this categorical distribution are precisely the words with which c co-occurs most often. These words characterize c more than all the other words in the vocabulary, so it is reasonable to expect that a word c′ whose categorical distribution has similar peaks as the categorical distribution of c is similar in meaning to c. Note that the location of the peaks of p(·|c) are immune to the scaling of uc (athough the values of p(·|c) may change); thus, the wordsw which best characterize c are those for which vw has a high inner product with uc/‖uc‖2. Since ∣∣∣∣uTc vw‖uc‖2 − u T c′vw ‖uc′‖2 ∣∣∣∣≤ √ 2 ( 1− u T c uc′ ‖uc‖2‖uc′‖2 ) ‖vw‖2,
it is clear that if the angle between the context representations of c and c′ is small, the distributions p(w|c) and p(w|c′) will tend to have similar peaks.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
3 Skip-Gram learns a Sufficient Dimensionality Reduction Model
The Skip-Gram model assumes that the distribution of the neighbors of a word follows a specific exponential parametrization of a categorical distribution. There is empirical evidence that this model generates features that are useful for NLP tasks, but there is no a priori guarantee that the training corpus was generated in this manner. In this section, we provide theoretical support for the usefulness of the features learned even when the SkipGram model is misspecified.
To do so, we draw a connection between SkipGram and the Sufficient Dimensionality Reduction (SDR) factorization of Globerson and Tishby (Globerson and Tishby, 2003). The SDR model learns optimal3 embeddings for discrete random variables X and Y without assuming any parametric form on the distributions of X and Y , and it is useful in a variety of applications, including information retrieval, document classification, and association analysis (Globerson and Tishby, 2003). As it turns out, these embeddings, like Skip-Gram, are obtained by learning the parameters of an exponentially parameterized distribution. In Theorem 3 below, we show that if a SkipGram model is fit to the cooccurence statistics of X and Y , then the output can be trivially modified (by adding readily-available information on word frequencies) to obtain the parameters of an SDR model.
This connection is significant for two reasons: first, the original algorithm of (Globerson and Tishby, 2003) for learning SDR embeddings is expensive, as it involves information projections. Theorem 3 shows that if one can efficiently fit a Skip-Gram model, then one can efficiently fit an SDR model. This implies that Skip-Gram specific approximation heuristics like negativesampling, hierarchical softmas, and Glove, which are believed to return high-quality approximations to Skip-Gram parameters (Mikolov et al., 2013b; Pennington et al., 2014), can be used to efficiently approximate SDR model parameters. Second, (Globerson and Tishby, 2003) argues for the optimality of the SDR embedding in any domain where the training information on X and Y
3Optimal in an information-theoretic sense: they preserve the maximal mutual information between any pair of random variables with the observed coocurrence statistics, without regard to the underlying joint distribution.
consists of their coocurrence statistics; this optimality and the Skip-Gram/SDR connection argues for the use of Skip-Gram approximations in such domains, and supports the positive experimental results that have been observed in applications in network science (Grover and Leskovec, 2016), proteinomics (Asgari and Mofrad, 2015), and other fields.
As stated above, the SDR factorization solves the problem of finding information-theoretically optimal features, given co-occurrence statistics for a pair of discrete random variables X and Y . Associate a vector wi to the ith state of X , a vector hj to the jth state of Y , and let W = [wT1 · · ·wT|X|]
T and H be defined similarly. Globerson and Tishby show that such optimal features can be obtained from a low-rank factorization of the matrix G of co-occurence measurements: Gij counts the number of times state i of X has been observed to co-occur with state j of Y. The loss of this factorization is measured using the KL-divergence, and so the optimal features are obtained from solving the problem
arg min W,H DKL
( G
ZG ∥∥∥∥ 1ZW,H eWHT ) .
Here, ZG = ∑
ij Gij normalizes G into an estimate of the joint pmf of X and Y , and similarly ZW,H is the constant that normalizes eWH T into a joint pmf. The expression eWH T
denotes entrywise exponentiation of WHT .
Now we revisit the Skip-Gram training objective, and show that it differs from the SDR objective only slightly. Whereas the SDR objective measures the distance between the pmfs given by (normalized versions of) G and eWH T , the SkipGram objective measures the distance between the pmfs given by (normalized versions of) the rows of G and eWH T . That is, SDR emphasizes fitting the entire pmfs, while Skip-Gram emphasizes fitting conditional distributions.
Before presenting our main result, we state and prove the following lemma, which is of independent interest and is used in the proof of our main theorem. Recall that Skip-Gram represents each word c as a multinomial distribution over all other words w, and it learns the parameters for these distributions by a maximum likelihood estimation. It is known that learning model parameters by maximum likelihood estimation is equivalent to minimizing the KL-divergence of the learned
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
model from the empirical distribution; the following lemma establishes the KL-divergence that Skip-Gram minimizes. Lemma 2. Let G be the word co-occurrence matrix constructed from the corpus on which a SkipGram model is trained, in which case Gcw is the number of times word w occurs as a neighboring word of c in the corpus. For each word c, let gc denote the empirical frequency of the word in the corpus, so that
gc = ∑ w Gcw/ ∑ t,w Gt,w.
Given a positive vector x, let x̂ = x/‖x‖1. Then, the Skip-Gram model model parameters U =[ u1 · · · u|V | ]T and V = [v1 · · · u|V |]T minimize the objective∑
c
gcDKL(ĝc ‖ êuTc VT ),
where gc is the cth row of G.
Proof. Recall that Skip-Gram chooses U and V to maximize
Q = 1
T T∑ i=1 C∑ δ=−C δ 6=0 log p(wi+δ|wi) ,
where
p(w|c) = e uTc vw∑n
i=1 e uTc vi
.
This objective can be rewritten using the pairwise cooccurence statistics as
Q= 1
T ∑ c,w Gcw log p(w|c)
= 1
T ∑ c [(∑ t Gct )∑ w Gcw∑ tGct log p(w|c) ]
∝ 1 T ∑ c
[ ( ∑
tGct) ( ∑
twGtw) ∑ w Gcw∑ tGct log p(w|c)
]
= ∑ c gc (∑ w ( ĝc ) w log p(w|c) ) = ∑ c gc ( −DKL(ĝc ‖ p(·|c))−H(ĝc) ) ,
where H(·) denotes the entropy of a distribution. It follows that since Skip-Gram maximizes Q, it minimizes∑ c gcDKL(ĝc ‖ p(·|c))= ∑ c gcDKL(ĝc ‖ êuTc VT ).
We now prove our main theorem of this section, which states that SDR parameters can be obtained by augmenting the Skip-Gram embeddings to account for word frequencies.
Theorem 3. Let U,V be the results of fitting a Skip-Gram model to G, and consider the augmented matrices
Ũ = [U |α] and Ṽ = [V |1],
where
αc = log
( gc∑
w e uTc vw
) and gc = ∑ wGc,w∑ t,wGt,w .
Then, the features (Ũ, Ṽ) constitute a sufficient dimensionality reduction of G.
Proof. For convenience, let G denote the joint pdf matrix G/ZG, and let Ĝ denote the matrix obtained by normalizing each row of G to be a probability distribution. Then, it suffices to show that DKL(G ‖ qW,H) is minimized over the set of probability distributions{
qW,H ∣∣∣∣ qW,H(w, c) = 1Z (eWHT )cw } ,
when W = Ũ and H = Ṽ. To establish this result, we use a chain rule for the KL-divergence. Recall that if we denote the expected KL-divergence between two marginal pmfs by
DKL(p(·|c)‖q(·|c))
= ∑ c p(c) (∑ w p(w|c) log ( p(w|c) q(w|c) )) ,
then the KL-divergence satisfies the chain rule:
DKL(p(w, c)‖q(w, c)) = DKL(p(c)‖q(c)) +DKL(p(w|c)‖q(w|c)).
Using this chain rule, we get
DKL(G ‖ qW,H(w, c)) (9) =DKL(g ‖ qW,H(c))+DKL(Ĝ‖qW,H(w|c)).
Note that the second term in this sum is, in the notation of Lemma 2, DKL(Ĝ‖qW,H(w|c)) = ∑ c gcDKL(ĝc ‖ êwTc HT ),
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
so the matrices U and V that are returned by fitting the Skip-Gram model minimize the second term in this sum. We now show that the augmented matrices W = Ũ and H = Ṽ also minimize this second term, and in addition they make the first term vanish.
To see that the first of these claims holds, i.e., that the augmented matrices make the second term in (9) vanish, note that
qŨ,Ṽ(w|c) ∝ e ũTc ṽw = eu T c vw+αc ∝ qU,V(w|c),
and the constant of proportionality is independent of w. It follows that qŨ,Ṽ(w|c) = qU,V(w|c) and
DKL(Ĝ ‖ qŨ,Ṽ(w|c)) = DKL(Ĝ ‖ qU,V(w|c)).
Thus, the choice W = Ũ and H = Ṽ minimizes the second term in (9).
To see that the augmented matrices make the first term in (9) vanish, observe that when W = Ũ and H = Ṽ, we have that qŨ,Ṽ(c) = g by construction. This can be verified by calculation:
qŨ,Ṽ(c) = ∑ w qŨ,Ṽ(w, c)∑ w,t qŨ,Ṽ(w, t) = ∑ w e uTc vw+αc∑ w,t e uTt vw+αt
=
(∑ w e uTc vw )
eαc∑ t (∑ w e uTt vw ) eαt
=
[ (eUV T 1) eα ] c
1T [ (eUV T 1) eα ] . Here, the notation x y denotes entry-wise multiplication of vectors.
Since αc = log(gc) − log (( eUV T 1 ) c ) , we
have
qŨ,Ṽ(c) =
[ (eUV T 1) eα ] c
1T [ (eUV T 1) eα ] = gc∑ t gt = gc.
The choice W = Ũ and H = Ṽ makes the first term in (9) vanish, and it also minimizes the second term in (9). Thus, it follows that the features (Ũ, Ṽ) constitute a sufficient dimensionality reduction of G.
This paper delves into the mathematical properties of the skip-gram model, explaining the reason for its success on the analogy task and for the general superiority of additive composition models. It also establishes a link between skip-gram and Sufficient Dimensionality Reduction.  I liked the focus of this paper on explaining the properties of skip-gram, and generally found it inspiring to read. I very much appreciate the effort to understand the assumptions of the model, and the way it affects (or is affected by) the composition operations that it is used to perform. In that respect, I think it is a very worthwhile read for the community.  My main criticism is however that the paper is linguistically rather naive. The authors' use of 'compositionality' (as an operation that takes a set of words and returns another with the same meaning) is extremely strange. Two words can of course be composed and produce a vector that is a) far away from both; b) does not correspond to any other concept in the space; c) still has meaning (productivity wouldn't exist otherwise!) Compositionality in linguistic terms simply refers to the process of combining linguistic constituents to produce higher-level constructs. It does not assume any further constraint, apart from some vague (and debatable) notion of semantic transparency. The paper's implication (l254) that composition takes place over sets is also wrong: ordering matters hugely (e.g. 'sugar cane' is not 'cane sugar'). This is a well-known shortcoming of additive composition.   Another important aspect is that there are pragmatic factors that make humans prefer certain phrases to single words in particular contexts (and the opposite), naturally changing the underlying distribution of words in a large corpus. For instance, talking of a 'male royalty' rather than a 'king' or 'prince' usually has implications with regard to the intent of the speaker (here, perhaps highlighting a gender difference). This means that the equation in l258 (or for that matter the KL-divergence modification) does not hold, not because of noise in the data, but because of fundamental linguistic processes. This point may be addressed by the section on SDR, but I am not completely sure (see my comments below).  In a nutshell, I think the way that the authors present composition is flawed, but the paper convinces me that this is indeed what happens in skip-gram, and I think this is an interesting contribution.   The part about Sufficient Dimensionality Reduction seems a little disconnected from the previous argument as it stands. I'm afraid I wasn't able to fully follow the argument, and I would be grateful for some clarification in the authors' response. If I understand it well, the argument is that skip-gram produces a model where a word's neighbours follow some exponential parametrisation of a categorical distribution, but it is unclear whether this actually reflects the distribution of the corpus (as opposed to what happens in, say, a pure count-based model). The fact that skip-gram performs well despite not reflecting the data is that it implements some form of SDR, which does not need to make any assumption about the underlying form of the data. But then, is it fair to say that the resulting representations are optimised for tasks where geometrical regularities are important, regardless of the actual pattern of the data? I.e. there some kind of denoising going on?  Minor comments:  - The abstract is unusually long and could, I think, be shortened.  - para starting l71: I think it would be misconstrued to see circularity here. Firth observed that co-occurrence effects were correlated with similarity judgements, but those judgements are the very cognitive processes that we are trying to model with statistical methods. Co-occurrence effects and vector space word representations are in some sense 'the same thing', modelling an underlying linguistic process we do not have direct observations for. So pair-wise similarity is not there to break any circularity, it is there because it better models the kind of judgements humans known to make.  - l296: I think 'paraphrase' would be a better word than 'synonym' here, given that we are comparing a set of words with a unique lexical item.  - para starting l322: this is interesting, and actually, a lot of the zipfian distribution (the long tail) is fairly uniform.  - l336: it is probably worth pointing out that the analogy relation does not hold so well in practice and requires to 'ignore' the first returned neighbour of the analogy computation (which is usually one of the observed terms).  - para starting l343: I don't find it so intuitive to say that 'man' would be a synonym paraphrase of anything involving 'woman'. The subtraction involved in the analogy computation is precisely not a straightforward composition operation, as it involves an implicit negation.   - A last, tiny general comment. It is usual to write p(w|c) to mean the probability of a word given a context, but in the paper 'w' is actually the context and 'c' the target word. It makes reading a little bit harder... Perhaps change the notation?  Literature:  The claim that Arora (2016) is the only work to try and understand vector composition is a bit strong. For instance, see the work by Paperno & Baroni on explaining the success of addition as a composition method over PMI-weighted vectors:  D. Paperno and M. Baroni. 2016. When the whole is less than the sum of its parts: How composition affects PMI values in distributional semantic vectors. Computational Linguistics 42(2): 345-350.  *** I thank the authors for their response and hope to see this paper accepted.