Mapping cognitive ontologies to and from the brain
Imaging neuroscience links brain activation maps to behavior and cognition via correlational studies. Due to the nature of the individual experiments, based on eliciting neural response from a small number of stimuli, this link is incomplete, and unidirectional from the causal point of view. To come to conclusions on the function implied by the activation of brain regions, it is necessary to combine a wide exploration of the various brain functions and some inversion of the statistical inference. Here we introduce a methodology for accumulating knowledge towards a bidirectional link between observed brain activity and the corresponding function. We rely on a large corpus of imaging studies and a predictive engine. Technically, the challenges are to find commonality between the studies without denaturing the richness of the corpus. The key elements that we contribute are labeling the tasks performed with a cognitive ontology, and modeling the long tail of rare paradigms in the corpus. To our knowledge, our approach is the first demonstration of predicting the cognitive content of completely new brain images.
To that end, we propose a method that predicts the experimental paradigms across different studies.

The authors provide a methodology for analyzing fMRI
      data from a database containing 19 studies, which allows making reverse
      inferences of brain regions specifically related to a given term, as
      opposed to forward inferences that are usually used in fMRI studies but
      that do not provide specificity. The authors apply this method to predict
      cognitive ontology terms for new brain images (not used in training).
      Differently from ref.[28] (Yarkoni et al., 2011), the authors do not use
      peak coordinates text-mined from journal articles, but analyze actual fMRI
      contrasts annotated with specific terms. Multivoxel pattern
      recognition studies and automated meta-analyses over numerous neuroimaging
      experiments are becoming increasingly popular and it's only a question of
      time when they will become a common practice in the neuroimaging field,
      which still suffers from problems of low statistical power, simplistic
      assumptions (of candidate regions) and limited comparability. The current
      paper provides an interesting methodology for obtaining reasonably
      accurate reverse inferences using a relatively small number of studies
      with few subjects. Although a similar database has been used for
      classification of mental states in ref.[18] (Poldrack et al., 2009), the
      current paper focuses on several difficult issues not resolved in [18],
      such as predictability across different studies, mitigating various
      confounding effects, and using cognitive ontology terms instead of classes
      based on task/study. Although the paper is good and appropriate
      for NIPS, it could be improved in several ways: - the last sentence of
      the abstract is too general and a more specific claim of novelty (such as
      predicting ontology terms across different studies) should be used; -
      to my knowledge, ref.[28] does not only use Talairach coordinates, as
      stated in the beginning of para.2 of the introduction; - assertion in
      section 3.1 that images from many different teams are used in the database
      does not seem to be very accurate, as the vast majority of studies seem to
      have only 2 different senior authors (one of which is sometimes the first
      author). More specific information regarding the number of different
      scanners and truly independent study designs should be reported. It should
      also be discussed whether prediction across different studies is
      successful also for these truly independent studies (i.e. with no related
      study being in the training set) or only for similar studies (when a
      training set contains at least one study with shared authors, scanners or
      designs). Currently only mean performances are reported, which are not
      very informative in this regard. - It would be useful to discuss why
      "buttons" and not "saccades" are inversely correlated with "none" (section
      4.1); - measures of "precision" and "recall" should be explicitly
      defined in the text or footnotes even if they are standard in certain
      fields; - what are the chance levels for predictions in fig.2? -
      without them it's very difficult (or impossible) to evaluate the quality
      of predictions. - it is not clear what are the criteria of
      classification into 3 "operating regimes" in fig.2 and what exactly do
      these regimes mean (if nothing particular, such classification should be
      avoided); - what is the rationale behind choices of coordinates for
      brain slices in fig.3 maps? In addition, the following
      typos/errors should be corrected: - in the abstract it should be
      "nature" instead of "natural" (line 2) and "bidirectional" better than
      "bilateral" (line 8); - in the 3rd sentence of section 4.1 (lines 3-4)
      I suppose it should be "visual" instead of "not visual"; - in the 5th
      line from the end of page 7, it should be "albeit" instead of "ableit";
      - on page 8, there should be "fewer" instead of "less" in line 5 of
      para.1 and the last line of para.2, "many" instead of "much" in the last
      line of para.1 and "whereas" instead of "will" in line 9 of para.2.
      Finally, it's more conventional for references to be numbered in
      the order of their presentation, not alphabetically. ---
      The authors addressed most of my major concerns regarding
      weaknesses of the study, therefore, assuming their responses and
      additional data are incorporated into the final version and that minor
      points are corrected as well, this is a solid paper that should be useful
      for the NIPS community. A few more points regarding author responses:
      - The analysis of training on Dehaene's data / testing on Poldrack's
      and vice versa is interesting and should be mentioned in the final version
      as well. It is not clear which term is "read" (no such term appeared in
      the paper) More detailed explanations regarding the substantial
      deterioration of "auditory" scores would be helpful. - Chance levels
      should also be provided for the Poldrack vs. Dehaene testing. It is also a
      bit strange how the "saccades" term with both precision and recall close
      to 0 (fig. 2) can be above chance. - I agree with Reviewer 3 that fig.
      3 should be presented/explained in a more informative way, indicating
      statistical significances. As the involvement of cerebellum in cognition
      is not so well known as in motor response, this result is particularly
      interesting and it would be useful to discuss it in the text. Regarding
      the prediction of auditory stimuli by voxels in the visual cortex, it's
      not clear based on the shown data if the result is genuine, but if it is,
      it may be related to evidence of multisensory integration between primary
      sensory areas (see the work of Micah M.
      Murray)


In recent years word-embedding models have gained great popularity due to their remarkable performance on several tasks, including word analogy questions and caption generation. An unexpected “sideeffect” of such models is that their vectors often exhibit compositionality, i.e., adding two word-vectors results in a vector that is only a small angle away from the vector of a word representing the semantic composite of the original words, e.g., “man” + “royal” = “king”. In this work we provide a mathematical formalism for compositionality, and a theoretical justification for the presence of additive compositionality in word vectors learned using the Skip-Gram model. In particular, we show that additive compositionality holds in an even stricter sense (small distance rather than small angle) under certain assumptions on the process generating the corpus. As a corollary, we explain the success of vector calculus for solving word analogies. When these assumptions do not hold, we show that compositionality is no longer additive, and provide the correct composition operator. Finally, we establish a connection between the Skip-Gram model and the Sufficient Dimensionality Reduction (SDR) framework of Globerson and Tishby. SDR models provide information-theoretically optimal symbol-embeddings for problems where the training data takes the form of co-occurrence statistics. We prove that the parameters of Skip-Gram models can be readily modified to obtain the parameters of SDR models by simply adding information on symbol frequencies. This shows that the Skip-Gram model is essentially learning optimal word embeddings in the sense of Globerson and Tishby. Further, it implies that the heuristics commonly used to approximately fit Skip-Gram models can be used to fit SDR models.
In this work we provide a mathematical formalism for compositionality, and a theoretical justification for the presence of additive compositionality in word vectors learned using the Skip-Gram model. In particular, we show that additive compositionality holds in an even stricter sense (small distance rather than small angle) under certain assumptions on the process generating the corpus. As a corollary, we explain the success of vector calculus for solving word analogies. When these assumptions do not hold, we show that compositionality is no longer additive, and provide the correct composition operator.
Finally, we establish a connection between the Skip-Gram model and the Sufficient Dimensionality Reduction (SDR) framework of Globerson and Tishby. SDR models provide information-theoretically optimal symbol-embeddings for problems where the training data takes the form of co-occurrence statistics. We prove that the parameters of Skip-Gram models can be readily modified to obtain the parameters of SDR models by simply adding informa-
tion on symbol frequencies. This shows that the Skip-Gram model is essentially learning optimal word embeddings in the sense of Globerson and Tishby. Further, it implies that the heuristics commonly used to approximately fit Skip-Gram models can be used to fit SDR models.
1 Introduction
The idea of representing words as vectors has a long history in computational linguistics and machine learning. The general idea is to find a map from words to vectors such that wordsimilarity and vector-similarity are in correspondence. Whilst vector-similarity can be readily quantified in terms of distances and angles, quantifying word-similarity is a more ambiguous task. A key insight in that regard is to posit that the meaning of a word is captured by “the company it keeps” (Firth, 1957) and, therefore, that two words that keep company with similar words are likely to be similar themselves. To break the cyclicality of this definition it is common to consider models that only attempt to capture pairwise cooccurrence statistics1.
In the simplest case, one seeks vectors whose similarity approximates the co-occurrence frequencies. In more sophisticated methods cooccurrences are reweighed to suppress the effect of more frequent words (Rohde et al., 2006) and/or to emphasize pairs of words whose co-occurrence frequency maximally deviates from the independence assumption (Church and Hanks, 1990).
An alternative to seeking word-embeddings that reflect co-occurrence statistics is to extract the
1Normally, if a is similar to b and b is similar to c, it should be that a is more similar to c than it is to a random word. But pairwise-similarity models do not explicitly require that, i.e., do not penalize model parameters under which a and c are not particularly similar.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
vectorial representation of words from non-linear statistical language models, specifically neural networks. (Bengio et al., 2003) already proposed (i) associating with each vocabulary word a feature vector, (ii) expressing the probability function of word sequences in terms of the feature vectors of the words in the sequence, and (iii) learning simultaneously the vectors and the parameters of the probability function. This approach came into prominence recently through works of Mikolov et al. (see below) whose main departure from (Bengio et al., 2003) was to follow the suggestion of (Mnih and Hinton, 2007) and tradeaway the expressive capacity of general neuralnetwork models for the scalability to very large corpora afforded by the more restricted class of log-linear models.
An unexpected side effect of deriving wordembeddings via neural networks is that the wordvectors produced appear to enjoy (approximate) additive compositionality: adding two wordvectors often results in a vector whose nearest word-vector is of the word capturing the composition of the added words, e.g., “man” + “royal” = “king” (Mikolov et al., 2013c). This unexpected property allows one to use these vectors to answer word-analogy questions algebraically, e.g., answering the question “Man is to king as woman is to ?” by returning the word whose word-vector is nearest to the vector
v(king) - v(man) + v(woman).
In this work we focus on explaining the source of this phenomenon for the most prominent such model, namely the Skip-Gram (SG) model introduced in (Mikolov et al., 2013a). The SG model learns vector representations of words based on their patterns of co-occurrence in the training corpus as follows: it assigns to each word c in the vocabulary V , a “context” and a “target” vector, respectively uc and vc, which are to be used in order to predict the words that appear around each occurrence of c within a window of ∆ tokens. Specifically, the log probability of any target word w to occur at any position within distance ∆ of a context word c is taken to be proportional to the inner product between uc and vw, i.e., letting n = |V |,
p(w|c) = e uTc vw∑n
i=1 e uTc vi
. (1)
Further, SG assumes that the conditional proba-
bility of each possible set of words in a window around a context word c factorizes as the product of the respective conditional probabilities:
p(w−∆, . . . , w∆|c) = ∆∏
δ=−∆ δ 6=0
p(wδ|c). (2)
(Mikolov et al., 2013a) proposed learning the SG parameters on a training corpus by using maximum likelihood estimation under (1) and (2). Thus, if wi denotes the i-th word in the training corpus and T the length of the corpus, we seek the word vectors that maximize
1
T T∑ i=1 ∆∑ δ=−∆ δ 6=0 log p(wi+δ|wi) . (3)
As mentioned, the normalized context vectors obtained from maximizing (3) under (1) and (2) exhibit additive compositionality. For example, the cosine distance between the sum of the context vectors of the words “Vietnam” and “capital” and the context vector of the word “Hanoi” is small.
While there has been much interest in using algebraic operations on word vectors to carry out semantic operations like composition, the only published work which attempts a rigorous theoretical understanding of this phenomenon is (Arora et al., 2016). This work guarantees that word vectors can be recovered by factorizing the so-called PMI matrix, and that algebraic operations on these word vectors can be used to solve analogies, under certain conditions on the process that generated the training corpus. Specifically, the word vectors must be known, a priori before their recovery, to have been generated by randomly scaling uniformly sampled vectors from the unit sphere2. Further, the ith word in the corpus must have been selected with probability proportional to eu T wci , where the “discourse” vector ci governs the topic of the corpus at the ith word. The discourse vector is assumed to evolve according to a random walk on the unit sphere that has a uniform stationary distribution.
By way of contrast, our results assume nothing about the properties of the word vectors a priori. In fact, the connection we establish between the Skip-gram and Sufficient Dimensionality Reduction model of (Globerson and Tishby, 2003)
2More generally, it suffices that the word vectors have certain properties consistent with this sampling process.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
shows that the word vectors learned by Skip-Gram are information-theoretically optimal. Further, the context word c in the Skip-Gram model essentially serves the role that the discourse vector does in the PMI model of (Arora et al., 2016): the words neighboring c are selected with probability proportional to eu T c vw . We find the exact non-linear composition operator when no assumptions are made on the context word. When an analogous assumption to that of (Arora et al., 2016) is made, that the context words are uniformly distributed, we prove that the composition operator reduces to vector addition.
While our primary motivation has been to provide a better theoretical understanding of the popular SG model, our connection with the SDR method opens up the possibility of practical applicability of our approach more generally. In particular, there is the question of whether, for a given corpus, fitting an SG model will give good embeddings—while we are making reasonable linguistic assumptions about how to model words and the interdependencies of words in a corpus, it’s not clear that these have to hold universally on all corpuses to which we apply SG. However, the fact that when we fit an SG model we are fitting an SDR model (up to frequency information), and the fact that SDR models are informationtheoretically optimal in a certain sense, argues that regardless of whether the SG assumptions hold, SG always gives us optimal features in the following sense: the learned context embeddings and target embeddings preserve the maximal amount of mutual information between any pair of random variables X and Y consistent with the observed co-occurence matrix, where Y is the target word and X is the predictor word (in a min-max sense, since there are many ways of coupling X and Y , each of which may have different amounts of mutual information). Importantly, this statement requires no assumptions on the distribution P (X,Y ).
2 Compositionality of Skip-Gram
In this section, we first give a mathematical formulation of the intuitive notion of compositionality of words. We then prove that the composition operator for the Skip-Gram model in full generality is a non-linear function of the vectors of the words being composed. Under a single simplifying assumption, the operator linearizes and reduces to
the addition of the word vectors. Finally, we explain how linear compositionality allows for solving word analogies with vector algebra.
A natural way of capturing the compositionality of words is to say that the set of context words c1, . . . , cm has the same meaning as the single word c if for every other word w,
p(w|c1, . . . , cm) = p(w|c) .
Although this is an intuitively satisfying definition, we never expect it to hold exactly; instead, we replace exact equality with the minimization of KL-divergence. That is, we state that the best candidate for having the same meaning as the set of context words C is the word
arg min c∈V
DKL(p(·|C) | p(·|c)) . (4)
We refer to any vector that minimizes (4) as a synonym of the set of words C.
There are two natural concerns with (4). The first is that, in general, it is not clear how to define p(·|C). The second is that KL-divergence minimization is a hard problem, as it involves optimization over many high dimensional probability distributions. Our main result shows that both of these problems go away for any language model that satisfies two assumptions:
A1. For every word c, there exists Zc such that for every word w,
p(w|c) = 1 Zc exp(uTc vw) . (5)
A2. For every set of words C = {c1, c2, . . . , cm}, there exists ZC such that for every word w,
p(w|C) = p(w) 1−m
ZC
m∏ i=1 p(w|ci) . (6)
Clearly, the SG model satisfies A1 by definition. We prove that it also satisfies A2 when m ≤ ∆ (Lemma 1).
Theorem 1. In every word model that satisfies A1 and A2, for every set of words C = {c1, . . . , cm}, the synonyms of C satisfy∑
w∈V p(w|c)vw = ∑ w∈V p(w|C)vw . (7)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Theorem 1 characterizes the composition operator for any language model which satisfies our two assumptions; in general, this operator is not addition. Instead, a synonym c is a vector such that the average word vector under p(·|c) matches that under p(·|C). When the expectations in (7) can be computed, the composition operator can be implemented by solving a non-linear system of equations to find a vector u for which the left-hand side of (7) equals the right-hand side.
Our next result proves that although the composition operator is nontrivial in the general case, to recover vector addition as the composition operator, it suffices to assume that the word frequency is uniform.
Theorem 2. In every word model that satisfies A1, A2, and where p(w) = 1/|V | for everyw ∈ V , the synonym of C = {c1, . . . , cm} is
u1 + . . .+ um .
As word frequencies are typically much closer to a Zipf distribution (Piantadosi, 2014), the uniformity assumption of Theorem 2 is not realistic. That said, we feel it is important to point out that, as reported in (Mikolov et al., 2013b), additivity captures compositionality more accurately when the training set is manipulated so that the prior distribution of the words is made closer to uniform.
Using composition to solve analogies It has been observed that word vectors trained using nonlinear models like SG tend to encode semantic relationships between words as linear relationships between the word vectors (Mikolov et al., 2013b; Pennington et al., 2014; Levy and Goldberg, 2014). In particular, analogies of the form “man:woman::king:?” can often be solved by taking ? to be the word in the vocabulary whose context vector has the smallest angle with uwoman + (uking−uman). Theorems 1 and 2 offer insight into the solution such analogy questions.
We first consider solving an analogy of the form “m:w::k:?”” in the case where the composition operator is nonlinear. The fact that m and w share a relationship means m is a synonym for the set of words {w,R}, where R is a set of words encoding the relationship between m and w. Similarly, the fact that k and ? share the same relationship means k is a synonym for the set of words {?, R}.
By Theorem 1, we have that R and ? must satisfy∑ `∈V p(`|m)v` = ∑ `∈V
p(`|w,R)v` and∑ `∈V p(`|k)v` = ∑ `∈V p(`|?, R)v`.
We see that solving analogies when the composition operator is nonlinear requires the solution of two highly nonlinear systems of equations. In sharp contrast, when the composition operator is linear, the solution of analogies delightfully reduces to elementary vector algebra. To see this, we again begin with the assertion that the fact that m and w share a relationship means m is a synonym for the set of words {w,R}; Similarly, k is a synonym for {?, R}. By Theorem 2,
um = uw + ur and
uk = u? + ur,
which gives the expected relationship
u? = uk + (uw − um).
Note that because this expression for u? is in terms of k, w, andm, there is actually no need to assume that R is a set of actual words in V .
2.1 Proofs
Proof of Theorem 1. Note that p(w|C) equals
p(w)1−m
ZC
m∏ i=1 p(w|ci)
= p(w)1−m
ZC exp ( m∑ i=1 uTcivw − m∑ i=1 logZci )
= 1
Z p(w)1−m exp(uTCvw) ,
where Z = ZC ∏m i=1 Zi, and uC = ∑m i=1 ui.
Minimizing the KL-divergence
DKL(p(·|c1, . . . , cm)‖p(·|c))
as a function of c is equivalent to maximizing the negative cross-entropy as a function of uc, i.e., as maximizing
Q(uc) = Z ∑ w exp(uTCvw) p(w)m−1 (uTc vw − logZc) .
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Since Q is concave, the maximizers occur where its gradient vanishes. As∇ucQ equals
Z ∑ w exp(uTCvw) p(w)m−1 [ vw − ∑n `=1 exp(u T c v`)v`∑n k=1 exp(u T c vk) ] = ∑n `=1 exp(u T c v`)v`∑n
k=1 exp(u T c vk) − Z ∑ w exp(uTCvw)vw p(w)m−1
= ∑ w∈V p(w|c)vw − ∑ w∈V p(w|c1, . . . , cm)vw ,
we see that (7) follows. Proof of Theorem 2. Recall that uC = ∑m
i=1 ui. When p(w) = 1/|V | for all w ∈ V , the negative cross-entropy simplifies to
Q(uc) = Z ∑ w exp ( uTCvw ) (uTc vw − logZc) ,
and its gradient∇ucQ to
Z ∑ w exp(uC Tvw) [ vw − ∑n `=1 exp(u T c v`)v`∑n k=1 exp(u T c vk) ] = Z
∑ w exp(uC Tvw)vw − ∑ w exp(uTc vw)vw .
Thus,∇Q(uC) = 0 and since Q is concave, uC is its unique maximizer.
Lemma 1. The SG model satisfies assumption A2 when m ≤ ∆.
Proof of Lemma 1. First, assume that m = ∆. In the SG model target words are conditionally independent given a context word, i.e.,
p(c1, . . . , cm|w) = m∏ i=1 p(ci|w).
Applying Baye’s rule,
p(w|c1, . . . , cm) = p(c1, . . . , cm|w)p(w)
p(c1, . . . , cm)
= p(w)
p(c1, . . . , cm) m∏ i=1 p(ci|w)
= p(w)
p(c1, . . . , cm) m∏ i=1 p(w|ci)p(ci) p(w)
= p(w)1−m
ZC
m∏ i=1 p(w|ci) , (8)
where ZC = 1/ ( ∏m i=1 p(ci)). This establishes the result when m = ∆. The cases m < ∆ follow by marginalizing out ∆ −m context words in the equality 8.
Projection of synonyms onto the vocabulary Theorem 2 states that if there is a word c in the vocabulary V whose context vector equals the sum of the context vectors of the words c1, . . . , cm, then c has the same “meaning”, in the sense of (4), as the composition of the words c1, . . . , cm. For any given set of words C = {c1, . . . , cm}, it is unlikely that there exists a word c ∈ V whose context vector is exactly equal to the sum of the context vectors of the words c1, . . . , cm. Similarly, in Theorem 1, the solution(s) to (7) will most likely not equal the context vector of any word in V . In both cases, we thus need to project the vector(s) onto words in our vocabulary in some manner.
Since Theorem 1 holds for any prior over V , in theory, we could enumerate all words in V and find the word(s) that minimize the difference of the left hand side of (7) from the right hand side. In practice, it turns out that the angle between the context vector of a word w ∈ V and solutionvector(s) is a good proxy and one gets very good experimental results by selecting as the synonym of a collection of words, the word that minimizes the angle to the synonym vector.
Minimizing the angle has been empirically successful at capturing composition in multiple loglinear word models. One way to understand the success of this approach is to recall that each word c is characterized by a categorical distribution over all other words w, as stated in (1). The peaks of this categorical distribution are precisely the words with which c co-occurs most often. These words characterize c more than all the other words in the vocabulary, so it is reasonable to expect that a word c′ whose categorical distribution has similar peaks as the categorical distribution of c is similar in meaning to c. Note that the location of the peaks of p(·|c) are immune to the scaling of uc (athough the values of p(·|c) may change); thus, the wordsw which best characterize c are those for which vw has a high inner product with uc/‖uc‖2. Since ∣∣∣∣uTc vw‖uc‖2 − u T c′vw ‖uc′‖2 ∣∣∣∣≤ √ 2 ( 1− u T c uc′ ‖uc‖2‖uc′‖2 ) ‖vw‖2,
it is clear that if the angle between the context representations of c and c′ is small, the distributions p(w|c) and p(w|c′) will tend to have similar peaks.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
3 Skip-Gram learns a Sufficient Dimensionality Reduction Model
The Skip-Gram model assumes that the distribution of the neighbors of a word follows a specific exponential parametrization of a categorical distribution. There is empirical evidence that this model generates features that are useful for NLP tasks, but there is no a priori guarantee that the training corpus was generated in this manner. In this section, we provide theoretical support for the usefulness of the features learned even when the SkipGram model is misspecified.
To do so, we draw a connection between SkipGram and the Sufficient Dimensionality Reduction (SDR) factorization of Globerson and Tishby (Globerson and Tishby, 2003). The SDR model learns optimal3 embeddings for discrete random variables X and Y without assuming any parametric form on the distributions of X and Y , and it is useful in a variety of applications, including information retrieval, document classification, and association analysis (Globerson and Tishby, 2003). As it turns out, these embeddings, like Skip-Gram, are obtained by learning the parameters of an exponentially parameterized distribution. In Theorem 3 below, we show that if a SkipGram model is fit to the cooccurence statistics of X and Y , then the output can be trivially modified (by adding readily-available information on word frequencies) to obtain the parameters of an SDR model.
This connection is significant for two reasons: first, the original algorithm of (Globerson and Tishby, 2003) for learning SDR embeddings is expensive, as it involves information projections. Theorem 3 shows that if one can efficiently fit a Skip-Gram model, then one can efficiently fit an SDR model. This implies that Skip-Gram specific approximation heuristics like negativesampling, hierarchical softmas, and Glove, which are believed to return high-quality approximations to Skip-Gram parameters (Mikolov et al., 2013b; Pennington et al., 2014), can be used to efficiently approximate SDR model parameters. Second, (Globerson and Tishby, 2003) argues for the optimality of the SDR embedding in any domain where the training information on X and Y
3Optimal in an information-theoretic sense: they preserve the maximal mutual information between any pair of random variables with the observed coocurrence statistics, without regard to the underlying joint distribution.
consists of their coocurrence statistics; this optimality and the Skip-Gram/SDR connection argues for the use of Skip-Gram approximations in such domains, and supports the positive experimental results that have been observed in applications in network science (Grover and Leskovec, 2016), proteinomics (Asgari and Mofrad, 2015), and other fields.
As stated above, the SDR factorization solves the problem of finding information-theoretically optimal features, given co-occurrence statistics for a pair of discrete random variables X and Y . Associate a vector wi to the ith state of X , a vector hj to the jth state of Y , and let W = [wT1 · · ·wT|X|]
T and H be defined similarly. Globerson and Tishby show that such optimal features can be obtained from a low-rank factorization of the matrix G of co-occurence measurements: Gij counts the number of times state i of X has been observed to co-occur with state j of Y. The loss of this factorization is measured using the KL-divergence, and so the optimal features are obtained from solving the problem
arg min W,H DKL
( G
ZG ∥∥∥∥ 1ZW,H eWHT ) .
Here, ZG = ∑
ij Gij normalizes G into an estimate of the joint pmf of X and Y , and similarly ZW,H is the constant that normalizes eWH T into a joint pmf. The expression eWH T
denotes entrywise exponentiation of WHT .
Now we revisit the Skip-Gram training objective, and show that it differs from the SDR objective only slightly. Whereas the SDR objective measures the distance between the pmfs given by (normalized versions of) G and eWH T , the SkipGram objective measures the distance between the pmfs given by (normalized versions of) the rows of G and eWH T . That is, SDR emphasizes fitting the entire pmfs, while Skip-Gram emphasizes fitting conditional distributions.
Before presenting our main result, we state and prove the following lemma, which is of independent interest and is used in the proof of our main theorem. Recall that Skip-Gram represents each word c as a multinomial distribution over all other words w, and it learns the parameters for these distributions by a maximum likelihood estimation. It is known that learning model parameters by maximum likelihood estimation is equivalent to minimizing the KL-divergence of the learned
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
model from the empirical distribution; the following lemma establishes the KL-divergence that Skip-Gram minimizes. Lemma 2. Let G be the word co-occurrence matrix constructed from the corpus on which a SkipGram model is trained, in which case Gcw is the number of times word w occurs as a neighboring word of c in the corpus. For each word c, let gc denote the empirical frequency of the word in the corpus, so that
gc = ∑ w Gcw/ ∑ t,w Gt,w.
Given a positive vector x, let x̂ = x/‖x‖1. Then, the Skip-Gram model model parameters U =[ u1 · · · u|V | ]T and V = [v1 · · · u|V |]T minimize the objective∑
c
gcDKL(ĝc ‖ êuTc VT ),
where gc is the cth row of G.
Proof. Recall that Skip-Gram chooses U and V to maximize
Q = 1
T T∑ i=1 C∑ δ=−C δ 6=0 log p(wi+δ|wi) ,
where
p(w|c) = e uTc vw∑n
i=1 e uTc vi
.
This objective can be rewritten using the pairwise cooccurence statistics as
Q= 1
T ∑ c,w Gcw log p(w|c)
= 1
T ∑ c [(∑ t Gct )∑ w Gcw∑ tGct log p(w|c) ]
∝ 1 T ∑ c
[ ( ∑
tGct) ( ∑
twGtw) ∑ w Gcw∑ tGct log p(w|c)
]
= ∑ c gc (∑ w ( ĝc ) w log p(w|c) ) = ∑ c gc ( −DKL(ĝc ‖ p(·|c))−H(ĝc) ) ,
where H(·) denotes the entropy of a distribution. It follows that since Skip-Gram maximizes Q, it minimizes∑ c gcDKL(ĝc ‖ p(·|c))= ∑ c gcDKL(ĝc ‖ êuTc VT ).
We now prove our main theorem of this section, which states that SDR parameters can be obtained by augmenting the Skip-Gram embeddings to account for word frequencies.
Theorem 3. Let U,V be the results of fitting a Skip-Gram model to G, and consider the augmented matrices
Ũ = [U |α] and Ṽ = [V |1],
where
αc = log
( gc∑
w e uTc vw
) and gc = ∑ wGc,w∑ t,wGt,w .
Then, the features (Ũ, Ṽ) constitute a sufficient dimensionality reduction of G.
Proof. For convenience, let G denote the joint pdf matrix G/ZG, and let Ĝ denote the matrix obtained by normalizing each row of G to be a probability distribution. Then, it suffices to show that DKL(G ‖ qW,H) is minimized over the set of probability distributions{
qW,H ∣∣∣∣ qW,H(w, c) = 1Z (eWHT )cw } ,
when W = Ũ and H = Ṽ. To establish this result, we use a chain rule for the KL-divergence. Recall that if we denote the expected KL-divergence between two marginal pmfs by
DKL(p(·|c)‖q(·|c))
= ∑ c p(c) (∑ w p(w|c) log ( p(w|c) q(w|c) )) ,
then the KL-divergence satisfies the chain rule:
DKL(p(w, c)‖q(w, c)) = DKL(p(c)‖q(c)) +DKL(p(w|c)‖q(w|c)).
Using this chain rule, we get
DKL(G ‖ qW,H(w, c)) (9) =DKL(g ‖ qW,H(c))+DKL(Ĝ‖qW,H(w|c)).
Note that the second term in this sum is, in the notation of Lemma 2, DKL(Ĝ‖qW,H(w|c)) = ∑ c gcDKL(ĝc ‖ êwTc HT ),
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
so the matrices U and V that are returned by fitting the Skip-Gram model minimize the second term in this sum. We now show that the augmented matrices W = Ũ and H = Ṽ also minimize this second term, and in addition they make the first term vanish.
To see that the first of these claims holds, i.e., that the augmented matrices make the second term in (9) vanish, note that
qŨ,Ṽ(w|c) ∝ e ũTc ṽw = eu T c vw+αc ∝ qU,V(w|c),
and the constant of proportionality is independent of w. It follows that qŨ,Ṽ(w|c) = qU,V(w|c) and
DKL(Ĝ ‖ qŨ,Ṽ(w|c)) = DKL(Ĝ ‖ qU,V(w|c)).
Thus, the choice W = Ũ and H = Ṽ minimizes the second term in (9).
To see that the augmented matrices make the first term in (9) vanish, observe that when W = Ũ and H = Ṽ, we have that qŨ,Ṽ(c) = g by construction. This can be verified by calculation:
qŨ,Ṽ(c) = ∑ w qŨ,Ṽ(w, c)∑ w,t qŨ,Ṽ(w, t) = ∑ w e uTc vw+αc∑ w,t e uTt vw+αt
=
(∑ w e uTc vw )
eαc∑ t (∑ w e uTt vw ) eαt
=
[ (eUV T 1) eα ] c
1T [ (eUV T 1) eα ] . Here, the notation x y denotes entry-wise multiplication of vectors.
Since αc = log(gc) − log (( eUV T 1 ) c ) , we
have
qŨ,Ṽ(c) =
[ (eUV T 1) eα ] c
1T [ (eUV T 1) eα ] = gc∑ t gt = gc.
The choice W = Ũ and H = Ṽ makes the first term in (9) vanish, and it also minimizes the second term in (9). Thus, it follows that the features (Ũ, Ṽ) constitute a sufficient dimensionality reduction of G.

This paper delves into the mathematical properties of the skip-gram model, explaining the reason for its success on the analogy task and for the general superiority of additive composition models. It also establishes a link between skip-gram and Sufficient Dimensionality Reduction.  I liked the focus of this paper on explaining the properties of skip-gram, and generally found it inspiring to read. I very much appreciate the effort to understand the assumptions of the model, and the way it affects (or is affected by) the composition operations that it is used to perform. In that respect, I think it is a very worthwhile read for the community.  My main criticism is however that the paper is linguistically rather naive. The authors' use of 'compositionality' (as an operation that takes a set of words and returns another with the same meaning) is extremely strange. Two words can of course be composed and produce a vector that is a) far away from both; b) does not correspond to any other concept in the space; c) still has meaning (productivity wouldn't exist otherwise!) Compositionality in linguistic terms simply refers to the process of combining linguistic constituents to produce higher-level constructs. It does not assume any further constraint, apart from some vague (and debatable) notion of semantic transparency. The paper's implication (l254) that composition takes place over sets is also wrong: ordering matters hugely (e.g. 'sugar cane' is not 'cane sugar'). This is a well-known shortcoming of additive composition.   Another important aspect is that there are pragmatic factors that make humans prefer certain phrases to single words in particular contexts (and the opposite), naturally changing the underlying distribution of words in a large corpus. For instance, talking of a 'male royalty' rather than a 'king' or 'prince' usually has implications with regard to the intent of the speaker (here, perhaps highlighting a gender difference). This means that the equation in l258 (or for that matter the KL-divergence modification) does not hold, not because of noise in the data, but because of fundamental linguistic processes. This point may be addressed by the section on SDR, but I am not completely sure (see my comments below).  In a nutshell, I think the way that the authors present composition is flawed, but the paper convinces me that this is indeed what happens in skip-gram, and I think this is an interesting contribution.   The part about Sufficient Dimensionality Reduction seems a little disconnected from the previous argument as it stands. I'm afraid I wasn't able to fully follow the argument, and I would be grateful for some clarification in the authors' response. If I understand it well, the argument is that skip-gram produces a model where a word's neighbours follow some exponential parametrisation of a categorical distribution, but it is unclear whether this actually reflects the distribution of the corpus (as opposed to what happens in, say, a pure count-based model). The fact that skip-gram performs well despite not reflecting the data is that it implements some form of SDR, which does not need to make any assumption about the underlying form of the data. But then, is it fair to say that the resulting representations are optimised for tasks where geometrical regularities are important, regardless of the actual pattern of the data? I.e. there some kind of denoising going on?  Minor comments:  - The abstract is unusually long and could, I think, be shortened.  - para starting l71: I think it would be misconstrued to see circularity here. Firth observed that co-occurrence effects were correlated with similarity judgements, but those judgements are the very cognitive processes that we are trying to model with statistical methods. Co-occurrence effects and vector space word representations are in some sense 'the same thing', modelling an underlying linguistic process we do not have direct observations for. So pair-wise similarity is not there to break any circularity, it is there because it better models the kind of judgements humans known to make.  - l296: I think 'paraphrase' would be a better word than 'synonym' here, given that we are comparing a set of words with a unique lexical item.  - para starting l322: this is interesting, and actually, a lot of the zipfian distribution (the long tail) is fairly uniform.  - l336: it is probably worth pointing out that the analogy relation does not hold so well in practice and requires to 'ignore' the first returned neighbour of the analogy computation (which is usually one of the observed terms).  - para starting l343: I don't find it so intuitive to say that 'man' would be a synonym paraphrase of anything involving 'woman'. The subtraction involved in the analogy computation is precisely not a straightforward composition operation, as it involves an implicit negation.   - A last, tiny general comment. It is usual to write p(w|c) to mean the probability of a word given a context, but in the paper 'w' is actually the context and 'c' the target word. It makes reading a little bit harder... Perhaps change the notation?  Literature:  The claim that Arora (2016) is the only work to try and understand vector composition is a bit strong. For instance, see the work by Paperno & Baroni on explaining the success of addition as a composition method over PMI-weighted vectors:  D. Paperno and M. Baroni. 2016. When the whole is less than the sum of its parts: How composition affects PMI values in distributional semantic vectors. Computational Linguistics 42(2): 345-350.  *** I thank the authors for their response and hope to see this paper accepted.


Zero-Shot Relation Extraction via Reading Comprehension
We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn high-quality relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relations that are only specified at test-time, for which we have no labeled training examples. Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relations with high accuracy, and that zero-shot generalization to unseen relations is possible, at lower accuracy levels, setting the bar for future work on this task.
1 Introduction
Relation extraction systems populate knowledge bases with facts (relations) from an unstructured text corpus. When the type of facts (schema) are predefined, one can use crowdsourcing (Liu et al., 2016) or distant supervision (Hoffmann et al., 2011) to collect examples and train an extraction model for each relation. However, these approaches are incapable of extracting relation types that were not specified in advance and observed during training. In this paper, we propose an alternative approach for relation extraction, which can potentially extract relations of new types that were neither specified nor observed a priori.
Relation Question Template educated at(x, y) Where did x graduate from? In which university did x study? What is x’s alma mater? occupation(x, y) What did x do for a living? What is x’s job? What is the profession of x? spouse(x, y) Who is x’s spouse? Who did x marry? Who is x married to?
We show that it is possible to reduce relation extraction to the problem of answering simple reading comprehension questions. We map each relation R(x, y) to at least one parametrized naturallanguage question qx whose answer is y. For example, the relation educated at(x, y) can be mapped to “Where did x study?” and “Which university did x graduate from?”. Given a particular entity x (“Turing”) and a text that mentions x (“Turing obtained his PhD from Princeton”), a non-null answer to any of these questions (“Princeton”) asserts the relation and also fills the slot y. Figure 1 illustrates a few more examples.
This reduction enables new ways of framing the learning problem. In particular, it allows us to perform zero-shot learning: define new relations “on the fly”, after the model has already been trained. More specifically, the zero-shot scenario assumes access to labeled data for N relations. This data is used to train a reading comprehension model through our reduction. However, at test time, we are asked about a previously unseen relation RN+1. Rather than providing labeled data for the new relation, we simply list questions that define the relation’s slot values. Assuming a good reading comprehension model has been learned, the correct values should be extracted.
Our zero-shot setup includes innovations both
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
in data and models. We use distant supervision for a relatively large number of relations (120) from Wikidata (Vrandečić, 2012), which are easily gathered in practice via the WikiReading dataset (Hewlett et al., 2016). We introduce a crowdsourcing approach for gathering and verifying the questions for each relation. This process produced about 10 questions per relation on average, yielding a dataset of over 30,000,000 questionsentence-answer examples in total. Because questions are paired with relations, not specific examples, this overall procedure has very modest costs.
The key modeling challenge is that most existing reading comprehension problem formulations assume the answer to the question is always present in the given text. However, for relation extraction, this premise does not hold, and the model needs to reliably determine when a question is not answerable. We show that a recent state-of-the-art neural approach for reading comprehension (Seo et al., 2016) can be directly extended with variables to model answerability, and trained directly on our new dataset. This modeling approach is another key advantage of our reduction: as machine reading models improve with time, so should our ability to extract relations.
Experiments demonstrate that our approach generalizes to new paraphrases of questions from the training set, while incurring only a minor loss in performance (4% relative F1 reduction). Furthermore, translating relation extraction to the realm of reading comprehension allows us to extract a significant portion of previously unseen relations, from virtually zero to an F1 of 41%. Our analysis suggests that our model is able to generalize to these cases by learning typing information that occurs across many relations (e.g. the answer to “Where” is a location), as well as detecting relation paraphrases to a certain extent. We also find that there are many feasible cases that our model does not quite master, providing an interesting challenge for future work.
2 Approach
We consider the slot-filling challenge in relation extraction, in which we are given a knowledgebase relation R, an entity e, and a sentence s. For example, consider the relation occupation, the entity “Steve Jobs”, and the sentence “Steve Jobs was an American businessman, inventor, and industrial designer”. Our goal is to find
a set of text spans A in s for which R(e, a) holds for each a ∈ A. In our example, A = {businessman, inventor, industrial designer}. The empty set is also a valid answer (A = ∅) when s does not contain any phrase that satisfies R(e, ?). We observe that given a natural-language question q that expresses R(e, ?) (e.g. “What did Steve Jobs do for a living?”), solving the reading comprehension problem of answering q from s is equivalent to solving the slot-filling challenge.
The challenge now becomes one of querification: translating R(e, ?) into q. Rather than querify R(e, ?) for every entity e, we propose a method of querifying the relation R. We treat e as a variable x, querify the parametrized query R(x, ?) (e.g. occupation(x, ?)) as a question template qx (“What did x do for a living?”), and then instantiate this template with the relevant entities, creating a tailored natural-language question for each entity e (“What did Steve Jobs do for a living?”). This process, schema querification, is by an order of magnitude more efficient than instance querification because annotating a relation type automatically annotates all of its instances.
Applying schema querification to N relations from a pre-existing relation-extraction dataset converts it into a reading-comprehension dataset. We then use this dataset to train a readingcomprehension model, which given a sentence s and a question q returns a set of text spans A within s that answer q (to the best of its ability).
In the zero-shot scenario, we are given a new relation RN+1(x, y) at test-time, which was neither specified nor observed beforehand. For example, the deciphered(x, y) relation, as in “Turing and colleagues came up with a method for efficiently deciphering the Enigma”, is too domainspecific to exist in common knowledge-bases. We then querify RN+1(x, y) into qx (“Which code did x break?”) or qy (“Who cracked y?”), and run our reading-comprehension model for each sentence in the document(s) of interest, while instantiating the question template with different entities that might participate in this relation. Each time the model returns a non-null answer a for a given question qe, it extracts the relation RN+1(e, a).
Ultimately, all we need to do for a new relation is define our information need in the form of a question. Our approach provides a naturallanguage API for application developers who are interested in incorporating a relation-extraction
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Relation Question Sentence & Answers
educated at What is Albert Einstein’s alma mater? Albert Einstein was awarded a PhD by the Universityof Zürich, with his dissertation titled... occupation What did Steve Jobs do for a living? Steve Jobs was an American businessman, inventor,and industrial designer.
spouse Who is Angela Merkel married to? Angela Merkel’s second and current husband is quantumchemist and professor Joachim Sauer, who has largely...
Figure 2: Examples from our reading-comprehension dataset. Each instance contains a relation R, a question q, a sentence s, and an answer set A. The question explicitly mentions an entity e, which also appears in s. For brevity, answers are underlined instead of being displayed in a separate column.
component in their programs; no linguistic knowledge or pre-defined schema is needed. To implement our approach, we require two components: training data and a reading-comprehension model. In Section 3, we construct a large relationextraction dataset and querify it using an efficient crowdsourcing procedure. We then adapt an existing state-of-the-art reading-comprehension model to suit our problem formulation (Section 4).
3 Dataset
To collect reading-comprehension examples as in Figure 2, we first gather labeled examples for the task of relation-slot filling. Slot-filling examples are similar to reading-comprehension examples, but contain a knowledge-base query R(e, ?) instead of a natural-language question; e.g. spouse(Angela Merkel, ?) instead of “Who is Angela Merkel married to?”. We collect many slot-filling examples via distant supervision, and then convert their queries into natural language.
Slot-Filling Data We use the WikiReading dataset (Hewlett et al., 2016) to collect labeled slot-filling examples. WikiReading was collected by aligning each Wikidata (Vrandečić, 2012) relation R(e, a) with the corresponding Wikipedia article D for the entity e, under the reasonable assumption that the relation can be derived from the article’s text. Each instance in this dataset contains a relation R, an entity e, a document D, and an answer a. We used distant supervision to select the specific sentences in which each R(e, a) manifests. Specifically, we took the first sentence s in D to contain both e and a. We then grouped instances by R, e, and s to merge all the answers for R(e, ?) given s into one answer set A.
Schema Querification We then conducted two annotation phases on Mechanical Turk to querify relations: collection and verification.
For each relation R, we present the annotator with 4 example sentences, where the entity e in each sentence s is masked by the variable x. In addition, we underline the extractable answers a ∈ A that appear in s (see Figure 3). The annotator must then come up with a question about x whose answer, given each sentence s, is the underlined span within that sentence. For example, “In which country is x?” captures the exact set of answers for each sentence in Figure 3. Asking a more general question, such as “Where is x?” might return false positives (“North America” in sentence 2).
Each worker produced 3 different question templates for each example set. For each relation, we sampled 3 different example sets, and hired 3 different annotators for each set. We ran one instance of this annotation phase where the workers were also given, in addition to the example set, the name of the relation (e.g. country), and another instance where it was hidden. Overall, this process collected 40 unique question templates on average, out of a potential 54.
In the verification phase, we measure the question templates’ quality by sampling additional sentences and instantiating each question template with the example entity e. Annotators are then asked to answer the question from the sentence s, or mark it as unanswerable; if the annotators’ answers match A, the question template is valid. We discarded the templates that were not answered correctly in the majority of the examples (6/10).1
Overall, we applied schema querification to 178 relations that had at least 100 examples each (accounting for 99.77% of the data), costing roughly $1,250. After the verification phase, we were left with 1,192 high-quality question templates span-
1We used this relatively lenient measure because many annotators selected the correct answer, but with a slightly incorrect span; e.g. “American businessman” instead of “businessman”. We therefore used token-overlap F1 as a secondary filter, requiring an average score of at least 0.75.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
(1) The wine is produced in the X region of France. (2) X, the capital of Mexico, is the most populous city in North America. (3) X is an unincorporated and organized territory of the United States. (4) The X mountain range stretches across the United States and Canada.
Figure 3: An example of the annotator’s input when querifying the country(x, ?) relation. The annotator is required to ask a question about x whose answer is, for each sentence, the underlined spans.
ning 120 relations.2 We then join these templates with our slot-filling dataset along relations, instantiating each template qx with its matching entities. This process yields a reading-comprehension dataset of over 30,000,000 examples, where each instance contains the original relation R (unobserved by the machine), a question q, a sentence s, and the set of answers A (see Figure 2).
Negative Examples Following the methodology of InfoboxQA (Morales et al., 2016), we generate negative examples by matching (for the same entity e) a question q that pertains to one relation with a sentence s that expresses another relation. We also assert that the sentence does not contain the answer to q. For instance, we match “Who is Angela Merkel married to?” with a sentence about her occupation: “Angela Merkel is a German politician who is currently the Chancellor of Germany”. This process generated 2,249,637 negative examples. While this is a relatively naive method of generating negative examples, our analysis shows that about a third of negative examples contain good distractors (see Section 6).
4 Model
Given a sentence s and a question q, our algorithm either returns an answer span3 a within s, or indicates that there is no answer.
The task of obtaining answer spans to naturallanguage questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016). In SQuAD, every question is answerable from the text, which is why these models assume that there exists a correct answer span. Therefore, we modify an existing model in a way that allows it to decide whether an answer exists. We first give a high-level description of the original model, and then describe our modification.
258 relations had zero questions after verification due to noisy distant supervision and little annotator quality control.
3While our setting allows for multiple answer spans per question, our algorithm assumes a single span; in practice, less than 5% of our data has multiple answers.
We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q. The model predicts the start and end positions ystart,yend of the answer span in s. BiDAF uses recurrent neural networks to encode contextual information within s and q alongside an attention mechanism to align parts of q with s and vice-versa.
The outputs of the BiDAF model are the confidence scores of ystart and yend, for each potential start and end. We denote these scores as zstart, zend ∈ RN , where N is the number of words in the sentence s. In other words, zstarti indicates how likely the answer is to start at position i of the sentence (the higher the more likely); similarly, zendi indicates how likely the answer is to end at that index. Assuming the answer exists, we can transform these confidence scores into pseudo-probability distributions pstart,pend via softmax. The probability of each i-to-j-span of the context can therefore be defined by:
P (a = si...j) = p start i p end j (1)
where pi indicates the i-th element of the vector pi, i.e. the probability of the answer starting at i. Seo et al. (2016) obtain the span with the highest probability during post-processing.
To allow the model to signal that there is no answer, we concatenate a trainable bias b to the end of both confidences score vectors zstart, zend. The new score vectors z̃start, z̃end ∈ RN+1 are defined as z̃start = [zstart; b] and similarly for z̃end, where [; ] indicates row-wise concatenation. Hence, the last elements of z̃start and z̃end indicate the model’s confidence that the answer has no start or end, respectively. We apply softmax to these augmented vectors to obtain pseudo-probability distributions, p̃start, p̃end. This means that the probability the model assigns to a null answer is:
P (a = ∅) = p̃startN+1p̃endN+1. (2)
If P (a = ∅) is higher than the probability of the best span, argmaxi,j≤N P (a = si...j), then the model deems that the question cannot be answered
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
from the sentence. Conceptually, adding the bias enables the model to be sensitive to the absolute values of the raw confidence scores zstart, zend. We are essentially setting and learning a threshold b that decides whether the model is sufficiently confident of the best candidate answer span.
While this threshold provides us with a dynamic per-example decision of whether the instance is answerable, we can also set a global confidence threshold pmin; if the best answer’s confidence is below that threshold, we infer that there is no answer. In Section 5.3 we use this global threshold to get a broader picture of the model’s performance.
5 Experiments
To understand how well our method can generalize to unseen data, we design experiments for unseen entities (Section 5.1), unseen question templates (Section 5.2), and unseen relations (Section 5.3).
Evaluation Metrics Each instance is evaluated by comparing the tokens in the labeled answer set with those of the predicted span.4 Precision is the true positive count divided by the number of times the system returned a non-null answer. Recall is the true positive count divided by the number of instances that have an answer.
Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them. The typical training set was an order of 1 million examples, for which 3 epochs were enough for convergence. All training sets had a ratio of 1:1 positive and negative examples, which was chosen to match the test sets’ ratio.
5.1 Unseen Entities
We show that our reading-comprehension approach works well in a typical relation-extraction setting by testing it on unseen entities and texts.
Setup We partitioned our dataset along entities in the question, and randomly clustered each entity into one of three groups: train, dev, or test. For example, examples about Alan Turing appeared only in the training set, while examples regarding Steve Jobs were exclusive to the test set. We then sampled 1,000,000 examples for train, 1,000 for dev,
4We ignore word order, case, punctuation, and articles (“a”, “an”, “the”). We also ignore “and”, which often appears when a single span captures multiple correct answers (e.g. “United States and Canada”).
and 10,000 for test. This partition also ensures that the sentences at test time are different from those in train, since the sentences are gathered from the Wikipedia article of each entity.
Results Our model correctly predicts the vast majority of test instances, yielding 89.44% F1 (87.66% precision, 91.32% recall). This result suggests that the model does indeed generalize to unseen entities. An analysis of 50 erroneous examples shows that 36% of errors can be attributed to annotation errors (chiefly missing entries in Wikidata), and an additional 42% result from inaccurate span selection (e.g. “8 February 1985” instead of “1985”), for which our model is fully penalized. In total, only 18% of our sample were pure system errors, suggesting that our model is performing even better in practice. This result demonstrates that reducing relation extraction to reading comprehension is indeed a viable approach for our Wikipedia slot-filling task.
5.2 Unseen Question Templates
We test our method’s ability to generalize to new descriptions of the same relation, by holding out a question template for each relation during training.
Setup We created 10 folds of train/dev/test samples of the data, in which one question template for each relation was held out for the test set, and another for the development set. For instance, “What did x do for a living?” may appear only in the training set, while “What is x’s job?” is exclusive to the test set. Each split was stratified by sampling N examples per question template (N = 1000, 10, 50 for train, dev, test, respectively). This process created 10 training sets of 966,000 examples with matching development and test sets of 940 and 4,700 examples each.
Comparison System To compare this scenario with one where the question templates were previously observed, we replicated the existing test sets and replaced the unseen question templates with templates from the training set. Revisiting our example, we convert test-set occurrences of “What is x’s job?” with “What did x do for a living?”.
Results Table 1 shows that our approach is able to generalize to unseen question templates. Our system’s performance on unseen questions is nearly as strong as for previously observed templates (losing roughly 3.5 points in F1).
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Precision Recall F1 Seen Templates 86.73% 86.54% 86.63% Unseen Templates 84.37% 81.88% 83.10%
Table 1: Performance on previously seen vs. unseen question templates. Both results use the same trained model, but Seen Templates tests on examples with templates that appeared in the training set, while Unseen Templates tests on examples with unobserved templates.
Precision Recall F1 KB Relation 19.32% 2.54% 4.32% NL Relation 40.50% 28.56% 33.40% Single Template 37.18% 31.24% 33.90% Multiple Templates 43.61% 36.45% 39.61% Question Ensemble 45.85% 37.44% 41.11%
Table 2: Performance on previously unseen relations. KB Relation does not contain any linguistic information about the relation, NL Relation uses the relation’s name as a natural-language expression, Single Template observes one template per relation during training, while Multiple Templates observes multiple templates per relation during training. Question Ensemble uses the trained Multiple Templates model, but is provided with 3 templates per instance at test time.
5.3 Unseen Relations
Finally, we test our approach in a true zero-shot setting, where the test-time question templates are not only unobserved during training, they describe new, unseen relations.
Setup We created 10 folds of train/dev/test samples, partitioned along relations: 84 relations for training, 12 for development, and 24 for test. For example, when the examples of educated at are allocated to the test set, none of them appear in the training set. Each split was stratified by sampling N examples per relation (N = 10000, 50, 500 for train, dev, test, respectively). This process created 10 training sets of 840,000 examples each with matching development and test sets of 600 and 12,000 examples per fold.
Comparison Systems We compare our querification approach to two alternatives. First, we consider using formal knowledge-base relations instead of questions; these are essentially indicators (e.g. relation1) that tie training examples of the same relation together. We expect this alternative to fail, since the new relation’s description does not contain any tangible information. The second
alternative uses the relation’s name as a naturallanguage string instead of a question. For example, educated at will be interpreted as “educated at”. Unlike the previous baseline, this method allows our machine reading system to interpret its lexical components via word and character embeddings. We also consider a weakened version of our approach where, during training, only one question template per relation is observed. This setting should somewhat hinder the model’s ability to learn paraphrases (e.g. “Where” and “Which country”, active and passive voice, etc).
Question Ensemble We also evaluated how asking (at test time) about the same relation in multiple ways improves performance. This scenario may be more realistic in actual applications, where extractor authors could easily provide a few question paraphrases to define their target relation. We created an ensemble by sampling 3 questions per test example and predicting the answer for each. We then chose the answer with the highest sum of confidence scores.
Results Table 2 shows the performance of each setting; Figure 4 extends these results by applying a global threshold on the answers’ confidence scores to generate precision/recall curves (see Section 4). As expected, knowledge-base relations are insufficient in a zero-shot setting, and must be interpreted as natural-language to allow for some generalization. The difference between using a single question template and the relation’s name appears to be minor. However, training on a variety of question templates rather than on a single expression substantially increases performance. We conjecture that multiple phrasings of the same
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Verbatim Relation András Dombai plays for what team?András Dombai... ...currently plays as a goalkeeper for FC Tatabánya.
Type Which airport is most closely associated with Royal Jordanian?Royal Jordanian Airlines... ...from its main base at Queen Alia International Airport...
Global Relation Who was responsible for directing Les petites fugues?Les petites fugues is a 1979 Swiss comedy film directed by Yves Yersin.
Type When was The Snow Hawk released?The Snow Hawk is a 1925 film...
Specific Relation Who started Fürstenberg China?The Fürstenberg China Factory was founded... ...by Johann Georg von Langen...
Type What voice type does Étienne Lainez have? Étienne Lainez... ...was a French operatic tenor...
relation allows our model to learn answer-type paraphrases that occur across many relations (see further discussion in Section 6). There is also an advantage to having multiple questions at test time; an ensemble of multiple questions per instance allows for a slight but significant improvement in both precision and recall.
6 Analysis
To understand how our method extracts unseen relations, we analyzed 100 random examples, of which 60 had answers in the sentence and 40 did not (negative examples).
For negative examples, we checked whether a distractor – an incorrect answer of the correct answer type – appears in the sentence. For example, the question “Who is John McCain married to?” does not have an answer in “John McCain chose Sarah Palin as his running mate”, but “Sarah Palin” is of the correct answer type. We noticed that 14 negative examples (35%) contain distractors. When pairing these examples with the results from the unseen relations experiment in Section 5.3, we found that our method answered 2/14 of the distractor examples incorrectly, compared to only 1/26 of the easier examples. It appears that while most of the negative examples are easy, a significant portion of them are not trivial.
For positive examples, we observed that some instances can be solved by matching the relation in the sentence to that in the question, while others rely more on the answer’s type. Moreover, we notice that each cue can be further categorized according to the type of information needed to detect it: (1) when part of the question appears verbatim in the text, (2) when the phrasing in the text deviates from the question in a way that is typical of other relations as well (e.g. syntactic variability), (3) when the phrasing in the text deviates
Relation Type Verbatim 5% 12% Global 25% 8% Specific 28% 22%
Table 3: The distribution of cues by type, based on a sample of 60.
Relation Type Verbatim 33% 43% Global 73% 60% Specific 18% 46%
Table 4: Our method’s accuracy on subsets of examples pertaining to different cue types. Results in italics are based on a sample of less than 10.
from the question in a way that is unique to this relation (e.g. lexical variability). We name these categories verbatim, global, and specific, respectively. Figure 5 illustrates all the different types of cues we discuss in our analysis.
We selected the most important cue for solving each instance. If there were two important cues, each one was counted as half. Table 3 shows their distribution. Type cues appear to be somewhat more dominant than relation cues (58% vs. 42%). Half of the cues are relation-specific, whereas global cues account for one third of the cases and verbatim cues for one sixth. This is an encouraging result, because we can potentially learn to accurately recognize verbatim and global cues from other relations. However, our method was only able to exploit these cues partially.
We paired these examples with the results from the unseen relations experiment in Section 5.3 to see how well our method performs in each category. Table 4 shows the results for the Multiple Templates setting. On one hand, the model appears agnostic to whether the relation cue is verbatim, global, or specific, and is able to correctly answer these instances with similar accuracy (there
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
is no clear trend due to the small sample size). For examples that rely on typing information, the trend is much clearer; our model is much better at detecting global type cues than specific ones.
Based on these observations, we think that the primary sources of our model’s ability to generalize to new relations are: global type detection, which is acquired from training on many different relations, and relation paraphrase detection (of all types), which probably relies on its pre-trained word embeddings.
7 Related Work
Zero-Shot Relation Extraction There has been relatively little work on zero-shot learning in information extraction. Rocktäschel et al. (2015) and Demeester et al. (2016) show that injecting precomputed inference rules between naturallanguage relations and knowledge-base relations in a universal-schema setting (Riedel et al., 2013), allows their system to predict whether a given entity pair takes part in an unseen knowledgebase relation. In their setting, all natural-language relations are observed at train time, while all knowledge-base relations are hidden until test time. This setting is akin to our unseen question templates experiment (Section 5.2) since a naturallanguage description of each target relation appears in the training data. The “pure” zero-shot scenario – in which no manifestation of the test relation was observed – is substantially more challenging (see Section 5.3).
Bronstein et al. (2015) offered an almost unsupervised approach for event-trigger identification. In their setting, a few seed trigger words are given at test time, and by using a collection of lexical inference and similarity features (tuned by different event types during training), their system is able to detect the desired event-type. We focus instead on the complementary challenge of slot filling, where natural-language questions align perhaps more directly with the values we hope to recover.
Open information extraction (Mausam et al., 2012) can be seen as a type of zero-shot extraction, since there is no need for relation-specific training data. However, such systems often treat every possible string as a new relation while we hope to, through the reduction to reading comprehension, extract a canonical slot value independent of how the original text is phrased.
Reading Comprehension Many reading comprehension formulations have been proposed recently (Richardson et al., 2013; Berant et al., 2014; Hermann et al., 2015; Weston et al., 2015; Hill et al., 2015; Rajpurkar et al., 2016). A typical example involves a text, a question (or cloze), and assumes that the answer is mentioned in the text. In comparison, our questions are relatively direct and easy, but we must also predict when the question is not answerable, to support relation extraction.
Querification Some recent question-answering datasets were collected by expressing knowledgebase assertions in natural language. The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.g. educated at(Turing, Princeton)), collecting roughly 100,000 natural-language questions to support QA against a knowledge graph. Morales et al. (2016) used a similar process to collect questions from Wikipedia infoboxes, yielding the 15,000-example InfoboxQA dataset.
For the task of identifying predicate-argument structures, QA-SRL (He et al., 2015) was proposed as an open schema for semantic roles, in which the relation between an argument and a predicate is expressed as a natural-language question containing the predicate (“Where was someone educated?”) whose answer is the argument (“Princeton”). The authors collected about 19,000 question-answer pairs from 3,200 sentences.
In these efforts, the costs scale linearly in the number of instances, requiring significant investments for large datasets. In contrast, schema querification can generate an enormous amount of data for a fraction of the cost by labeling at the relation level; in this work, we generated 30,000,000 examples with a budget of $1,250 (see Section 3).
8 Conclusion
We showed that relation extraction can be reduced to a reading comprehension problem, allowing us to generalize to unseen relations that are defined on-the-fly in natural language. However, the problem of zero-shot relation extraction is far from solved, and poses an interesting challenge to both the information extraction and machine reading communities. As research into machine reading progresses, we may find that more tasks can benefit from a similar approach. We hope the contributions and insights presented in this paper will support future work in this avenue.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

- Strengths: Zero-shot relation extraction is an interesting problem. The authors have created a large dataset for relation extraction as question answering which would likely be useful to the community.  - Weaknesses: Comparison and credit to existing work is severely lacking. Contributions of the paper don't seen particularly novel.  - General Discussion:  The authors perform relation extraction as reading comprehension. In order to train reading comprehension models to perform relation extraction, they create a large dataset of 30m  querified d (converted to natural language) relations by asking mechanical turk annotators to write natural language queries for relations from a schema. They use the reading comprehension model of Seo et al. 2016, adding the ability to return  no relation, d as the original model must always return an answer. The main motivation result of the paper appears to be that the authors can perform zero-shot relation extraction, extracting relations only seen at test time.  This paper is well-written and the idea is interesting. However, there are insufficient experiments and comparison to previous work to convince me that the paper  contributions are novel and impactful.  First, the authors are missing a great deal of related work: Neelakantan at al. 2015 (https:  arxiv.org abs 1504.06662) perform zero-shot relation extraction using RNNs over KB paths. Verga et al. 2017 (https:  arxiv.org abs 1606.05804) perform relation extraction on unseen entities. The authors cite Bordes et al. (https:  arxiv.org pdf 1506.02075.pdf), who collect a similar dataset and perform relation extraction using memory networks (which are commonly used for reading comprehension). However, they merely note that their data was annotated at the  relation d level rather than at the triple (relation, entity pair) level u2026 but couldn t Bordes et al. have done the same in their annotation? If there is some significant difference here, it is not made clear in the paper. There is also a NAACL 2016 paper (https:  www.aclweb.org anthology N N16 N16-2016.pdf) which performs relation extraction using a new model based on memory networks u2026 and I m sure there are more. Your work is so similar to much of this work that you should really cite and establish novelty wrt at least some of them as early as the introduction -- that's how early I was wondering how your work differed, and it was not made clear.  Second, the authors neither 1) evaluate their model on another dataset or 2) evaluate any previously published models on their dataset. This makes their empirical results extremely weak. Given that there is a wealth of existing work that performs the same task and the lack of novelty of this work, the authors need to include experiments that demonstrate that their technique outperforms others on this task, or otherwise show that their dataset is superior to others (e.g. since it is much larger than previous, does it allow for better generalization?)