Reasoning about implied relationships (e.g. paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems. This paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships. Our pairwise embeddings are computed as a compositional function on word representations, which is learned by maximizing the pointwise mutual information (PMI) with the contexts in which the two words co-occur. We add these representations to the cross-sentence attention layer of existing inference models (e.g. BiDAF for QA, ESIM for NLI), instead of extending or replacing existing word embeddings. Experiments show a gain of 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI. Our representations also aid in better generalization with gains of around 6-7% on adversarial SQuAD datasets, and 8.8% on the adversarial entailment test set by Glockner et al. (2018). Reasoning about relationships between pairs of words is crucial for cross sentence inference problems such as question answering (QA) and natural language inference (NLI). In NLI, for example, given the premise  golf is prohibitively expensive , inferring that the hypothesis  golf is a cheap pastime  is a contradiction requires one to know that expensive and cheap are antonyms. Recent work (Glockner et al., 2018) has shown that current models, which rely heavily on unsupervised single-word embeddings, struggle to learn such relationships. In this paper, we show that they can be learned with word pair vectors ( pair2vec 1 ), 1 https://github.com/mandarjoshi90/ pair2vec X Y Contexts with X and Y baths hot cold too X or too Y neither X nor Y in X , Y Portland Oregon the X metropolitan area in Y X International Airport in Y food X are maize, Y , etc crop wheat dry X , such as Y , more X circles appeared in Y fields XOS comes with Y play Android Google the X team at Y X is developed by Y Table 1: Example word pairs and their contexts. which are trained unsupervised, and which significantly improve performance when added to existing cross-sentence attention mechanisms. Unlike single-word representations, which typically model the co-occurrence of a target word x with its context c , our word-pair representations are learned by modeling the three-way co-occurrence between words ( x, y ) and the context c that ties them together, as seen in Table 1. While similar training signals have been used to learn models for ontology construction (Hearst, 1992; Snow et al., 2005; Turney, 2005; Shwartz et al., 2016) and knowledge base completion (Riedel et al., 2013), this paper shows, for the first time, that large scale learning of pairwise embeddings can be used to directly improve the performance of neural cross-sentence inference models. More specifically, we train a feedforward network R ( x, y ) that learns representations for the individual words x and y , as well as how to compose them into a single vector. Training is done by maximizing a generalized notion of the pointwise mutual information (PMI) among x , y , and their context c using a variant of negative sampling (Mikolov et al., 2013a). Making R ( x, y ) a compositional function on individual words alleviates the sparsity that necessarily comes with embedding pairs of words, even at a very large scale. We show that our embeddings can be added to existing cross-sentence inference models, such as BiDAF++ (Seo et al., 2017; Clark and Gardner, 2018) for QA and ESIM (Chen et al., 2017) for NLI. Instead of changing the word embeddings that are fed into the encoder, we add the pretrained pair representations to higher layers in the network where cross sentence attention mechanisms are used. This allows the model to use the background knowledge that the pair embeddings implicitly encode to reason about the likely relationships between the pairs of words it aligns. Experiments show that simply adding our word-pair embeddings to existing high-performing models, which already use ELMo (Peters et al., 2018), results in sizable gains. We show 2.72 F1 points over the BiDAF++ model (Clark and Gardner, 2018) on SQuAD 2.0 (Rajpurkar et al., 2018), as well as a 1.3 point gain over ESIM (Chen et al., 2017) on MultiNLI (Williams et al., 2018). Additionally, our approach generalizes well to adversarial examples, with a 6-7% F1 increase on adversarial SQuAD (Jia and Liang, 2017) and a 8.8% gain on the Glockner et al. (2018) NLI benchmark. An analysis of pair2vec on word analogies suggests that it complements the information in single-word representations, especially for encyclopedic and lexicographic relations. Extending the distributional hypothesis to word pairs, we posit that similar word pairs tend to occur in similar contexts, and that the contexts provide strong clues about the likely relationships that hold between the words (see Table 1). We assume a dataset of ( x, y, c ) triplets, where each instance depicts a word pair ( x, y ) and the context c in which they appeared. We learn two compositional representation functions, R ( x, y ) and C ( c ) , to encode the pair and the context, respectively, as d dimensional vectors (Section 2.1). The functions are trained using a variant of negative sampling, which tries to embed word pairs ( x, y ) close to the contexts c with which they appeared (Section 2.2). individual words. The word-pair representation function R ( x, y ) first embeds and normalizes the individual words with a shared lookup matrix E a : x = E a ( x ) (cid:107) E a ( x ) (cid:107) y = E a ( y ) (cid:107) E a ( y ) (cid:107) These vectors, along with their element-wise product, are fed into a four-layer perceptron: R ( x, y ) = MLP 4 ( x , y , x  y ) The context c = c 1 ...c n is encoded as a d dimensional vector using the function C ( c ) . C ( c ) embeds each token c i with a lookup matrix E c , contextualizes it with a single-layer Bi-LSTM, and then aggregates the entire context with attentive pooling: c i = E c ( c i ) h 1 ... h n = BiLSTM ( c 1 ... c n ) w = softmax i ( kh i ) C ( c ) = (cid:88) i w i Wh i where W  R d  d and k  R d . All parameters, including the lookup tables E a and E c , are trained. Our representation is similar to two recently-proposed frameworks by Washio and Kato (2018a,b), but differs in that: (1) they use dependency paths as context, while we use surface form; (2) they encode the context as either a lookup table or the last state of a unidirectional LSTM. We also use a different objective, which we discuss next. To optimize our representation functions, we consider two variants of negative sampling (Mikolov et al., 2013a): bivariate and multivariate. The original bivariate objective models the two-way distribution of context and (monolithic) word pair co-occurrences, while our multivariate extension models the three-way distribution of word-word-context co-occurrences. We further augment the multivariate objective with typed sampling to up-sample harder negative examples. We discuss the impact of the bivariate and multivariate objectives (and other components) in Section 4.3. aspires to make R ( x, y ) and C ( c ) similar (have high inner products) for ( x, y, c ) that were observed together in the data. At the same time, we wish Bivariate J 2 NS ( x, y, c ) = log  ( R ( x, y )  C ( c )) + (cid:80) k c i =1 log  (cid:0)  R ( x, y )  C ( c Ni ) (cid:1) Multivariate J 3 NS ( x, y, c ) = J 2 NS ( x, y, c ) + (cid:80) k x i =1 log  (cid:0)  R ( x Ni , y )  C ( c ) (cid:1) + (cid:80) k y i =1 log  (cid:0)  R ( x, y Ni )  C ( c ) (cid:1) Table 2: The bivariate and multivariate negative sampling objectives. to keep our pair vectors dis -similar from random context vectors. In a straightforward application of the original (bivariate) negative sampling objective, we could generate a negative example from each observed ( x, y, c ) instance by replacing the original context c with a randomly-sampled context c N (Table 2, J 2 NS ). Assuming that the negative contexts are sampled from the empirical distribution P (  ,  , c ) (with P ( x, y, c ) being the portion of ( x, y, c ) instances in the dataset), we can follow Levy and Goldberg (2014) to show that this objective converges into the pointwise mutual information (PMI) between the word pair and the context. This objective mainly captures co-occurrences of monolithic pairs and contexts, and is limited by the fact that the training data, by construction, only contains pairs occurring within a sentence. For better generalization to cross-sentence tasks, where the pair distribution differs from that of the training data, we need a multivariate objective that captures the full three-way ( x, y, c ) interaction. Multivariate Negative Sampling We introduce negative sampling of target words, x and y , in addition to negative sampling of contexts c (Table 2, J 3 NS ). Our new objective also converges to a novel multivariate generalization of PMI, different from previous PMI extensions that were inspired by information theory (Van de Cruys, 2011) and heuristics (Jameel et al., 2018). 2 Following Levy and Goldberg (2014), we can show that when replacing target words in addition to contexts, our objective will converge 3 to the optimal value in Equation 1: R ( x, y )  C ( c ) = log P ( x, y, c ) Z x,y,c (1) 2 See supplementary material for their exact formulations. where Z x,y,c , the denominator, is: Z x,y,c = k c P (  ,  , c ) P ( x, y,  ) + k x P ( x,  ,  ) P (  , y, c ) + k y P (  , y,  ) P ( x,  , c ) (2) This optimal value deviates from previous generalizations of PMI by having a linear mixture of marginal probability products in its denominator. By introducing terms such as P ( x,  , c ) and P (  , y, c ) , the objective penalizes spurious correlations between words and contexts that disregard the other target word. For example, it would assign the pattern  X is a Y  a high score with ( banana , fruit ), but a lower score with ( cat , fruit ). Typed Sampling In multivariate negative sampling, we typically replace x and y by sampling from their unigram distributions. In addition to this, we also sample uniformly from the top 100 words according to cosine similarity using distributional word vectors. This is done to encourage the model to learn relations between specific instances as opposed to more general types. For example, using California as a negative sample for Oregon helps the model to learn that the pattern  X is located in Y  fits the pair ( Portland , Oregon ), but not the pair ( Portland , California ). Similar adversarial constraints were used in knowledge base completion (Toutanova et al., 2015) and word embeddings (Li et al., 2017). 4 3 Integrating pair2vec into Models We first present a general outline for incorporating pair2vec into attention-based architectures, and then discuss changes made to BiDAF++ and ESIM. The key idea is to inject our pairwise representations into the attention layer by reusing the cross-sentence attention weights. In addition to attentive pooling over single word representations, we also pool over cross-sentence word pair embeddings (Figure 1). 4 Applying typed sampling also changes the value to which our objective will converge, and will replace the unigram probabilities in Equation (2) to reflect the type-based distribution. Pair Representation We assume that we are given two sequences a = a 1 ...a n and b = b 1 ...b m . We represent the word-pair embeddings between a and b using the pretrained pair2vec model as: r i,j = (cid:20) R ( a i , b j ) (cid:107) R ( a i , b j ) (cid:107) ; R ( b j , a i ) (cid:107) R ( b j , a i ) (cid:107) (cid:21) (3) We include embeddings in both directions, R ( a i , b j ) and R ( b j , a i ) , because the many relations can be expressed in both directions; e.g., hypernymy can be expressed via  X is a type of Y  as well as  Y such as X . We take the L 2 normalization of each direction's pair embedding because the heavy-tailed distribution of word pairs results in significant variance of their norms. Base Model Let a 1 ... a n and b 1 ... b m be the vector representations of sequences a and b , as produced by the input encoder (e.g. ELMo embeddings contextualized with model-specific BiL-STMs). Furthermore, we assume that the base model computes soft word alignments between a and b via co-attention (4, 5), which are then used to compute b -aware representations of a : s i,j = f att ( a i , b j ) (4)  = softmax j ( s i,j ) (5) b i = m (cid:88) j =0  i,j b j (6) a inf i = (cid:2) a i ; b i (cid:3) (7) The symmetric term b infj is defined analogously. We refer to a inf and b inf as the inputs to the inference layer, since this layer computes some function over aligned word pairs, typically via a feedforward network and LSTMs. The inference layer is followed by aggregation and output layers. Injecting pair2vec We conjecture that the inference layer effectively learns word-pair relationships from training data, and it should, therefore, help to augment its input with pair2vec . We augment a infi (7) with the pair vectors r i,j (3) by concatenating a weighted average of the pair vectors r i,j involving a i , where the weights are the same  i,j computed via attention in (5): r i = (cid:88) j  i,j r i,j (8) a infi = (cid:2) a i ; b i ; r i (cid:3) (9) The symmetric term b infj is defined analogously. We augment the inference layer in the BiDAF++ model with pair2vec . BiDAF++ is an improved version of the BiDAFNoAnswer (Seo et al., 2017; Levy et al., 2017) which includes self-attention and ELMo embeddings from Peters et al. (2018). We found this variant to be stronger than the baselines presented in Rajpurkar et al. (2018) by over 2.5 F1. We use BiDAF++ as a baseline since its architecture is typical for QA systems, and, until recently, was state-of-the-art on SQuAD 2.0 and other benchmarks. BiDAF++ Let a and b be the outputs of the passage and question encoders respectively (in place of the standard p and q notations). The inference layer's inputs a infi are defined similarly to the generic model's in (7), but also contain an aggregation of the elements in a , with better-aligned elements receiving larger weights:  = softmax i (max j s i,j ) (10) a i = (cid:88) i  i a i (11) a infi = (cid:2) a i ; b i ; a i  b i ; a (cid:3) (12) In the later layers, a inf is recontextualized using a BiGRU and self attention. Finally a prediction layer predicts the start and end tokens. For NLI, we augment the ESIM model (Chen et al., 2017), which was previously state-of-the-art on both SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) benchmarks. ESIM Let a and b be the outputs of the premise and hypothesis encoders respectively (in place of the standard p and h notations). The inference layer's inputs a infi (and b infj ) are defined similarly to the generic model's in (7): a infi = (cid:2) a i ; b i ; a i  b i ; a i  b i (cid:3) (14) In the later layers, a inf and b inf are projected, recontextualized, and converted to a fixed-length vector for each sentence using multiple pooling schemes. These vectors are then passed on to an output layer, which predicts the class. A similar augmentation of ESIM was recently proposed in KIM (Chen et al., 2018). However, their pair vectors are composed of WordNet features, while our pair embeddings are learned directly from text (see further discussion in Section 6). For experiments on QA (Section 4.1) and NLI (Section 4.2), we use our full model which includes multivariate and typed negative sampling. We discuss ablations in Section 4.3 Benchmark BiDAF + pair2vec  SQuAD 2.0 EM 65.66 68.02 +2.36 F1 68.86 71.58 +2.72 AddSent EM 37.50 44.20 +6.70 F1 42.55 49.69 +7.14 AddOneSent EM 48.20 53.30 +5.10 F1 54.02 60.13 +6.11 Table 3: Performance on SQuAD 2.0 and adversarial SQuAD (AddSent and AddOneSent) benchmarks, with and without pair2vec . Data We use the January 2018 dump of English Wikipedia, containing 96M sentences to train pair2vec . We restrict the vocabulary to the 100K most frequent words. Preprocessing removes all out-of-vocabulary words in the corpus. We consider each word pair within a window of 5 in the preprocessed corpus, and subsample 5 instances based on pair probability with a threshold of 5  10  7 . We define the context as one word each to the left and right, and all the words in between each pair, replacing both target words with place-holders X and Y (see Table 1). More details can be found in the supplementary material. We experiment on the SQuAD 2.0 QA benchmark (Rajpurkar et al., 2018), as well as the adversarial datasets of SQuAD 1.1 (Rajpurkar et al., 2016; Jia and Liang, 2017). Table 3 shows the performance of BiDAF++, with ELMo , before and after adding pair2vec . Experiments on SQuAD 2.0 show that our pair representations improve performance by 2.72 F1. Moreover, adding pair2vec also results in better generalization on the adversarial SQuAD datasets with gains of 7.14 and 6.11 F1. 5 Like in word2vec , subsampling reduces the size of the dataset and speeds up training. For this, we define the word pair probability as the product of unigram probabilities. We also record a gain of 8.8% absolute over ESIM on the Glockner et al. (2018) dataset, setting a new state of the art. Following standard practice (Glockner et al., 2018), we train all models on a combination of SNLI (Bowman et al., 2015) and MultiNLI. Glockner et al. (2018) show that with the exception of KIM (Chen et al., 2018), which uses WordNet features, several NLI models fail to generalize to this setting which involves lexical inference. For a fair comparison with KIM on the Glockner test set, we replace ELMo with GLoVE embeddings, and still outperform KIM by almost halving the error rate. Ablating parts of pair2vec shows that all components of the model (Section 2) are useful. We ablate each component and report the EM and F1 on the development set of SQuAD 2.0 (Table 6). The full model, which uses a 4-layer MLP for R ( x, y ) and trains with multivariate negative sampling, achieves the highest F1 of 72.68. We experiment with two alternative composition functions, a 2-layer MLP ( Composition: 2 Layers ) and element-wise multiplication ( Compo-0.0 0.2 0.4 0.6 0.8 1.0 0 20 40 60 80 100 A cc u r a c y Derivational Lexicographic Inflectional Encyclopedic Figure 2: Accuracy as a function of the interpolation parameter  (see Eq. sition: Multiply ), which yield significantly smaller gains over the baseline BiDAF++ model. This demonstrates the need for a deep composition function. Eliminating sampling of target words ( x, y ) from the objective ( Objective: Bivariate NS ) results in a drop of 0.7 F1, accounting for about a quarter of the overall gain. This suggests that while the bulk of the signal is mined from the pair-context interactions, there is also valuable information in other interactions as well. We also test whether specific pre-training of word pair representations is useful by replacing pair2vec embeddings with the vector offsets of pre-trained word embeddings ( Unsupervised: Pair Dist ). We follow the PairDistance method for word analogies (Mikolov et al., 2013b), and represent the pair ( x, y ) as the L2 normalized difference of single-word vectors: ( x  y ) / (cid:107) x  y (cid:107) . We use the same fastText (Bojanowski et al., 2017) word vectors with which we initialized pair2vec before training. We observe a gain of only 0.34 F1 over the baseline. In Section 4, we showed that pair2vec adds information complementary to single-word representations like ELMo. Here, we ask what this extra information is, and try to characterize which word relations are better captured by pair2vec . To that end, we evaluate performance on a word analogy dataset with over 40 different relation types (Section 5.1), and observe how pair2vec fills hand-crafted relation patterns (Section 5.2). Word Analogy Dataset Given a word pair ( a, b ) and word x , the word analogy task involves predicting a word y such that a : b :: x : y . We use the Bigger Analogy Test Set (BATS, Glad-kova et al., 2016) which contains four groups of relations: encyclopedic semantics (e.g., person-profession as in Einstein physicist ), lexicographic semantics (e.g., antonymy as in cheap expensive ), derivational morphology (e.g., noun forms as in oblige obligation ), and inflectional morphology (e.g., noun-plural as in bird birds ). Each group contains 10 sub-relations. where a , b , x , and y represent fastText embeddings 6 and r a,b , r x,y represent the pair2vec embedding for the word pairs ( a, b ) and ( x, y ) , respectively;  is the linear interpolation parameter. Following prior work (Mikolov et al., 2013b), we return the highest-scoring y in the entire vocabulary, excluding the given words a , b , and x . Results Figure 2 shows how the accuracy on each category of relations varies with  . For all four groups, adding pair2vec to 3CosAdd results in significant gains. In particular, the biggest relative improvements are observed for encyclopedic (356%) and lexicographic (51%) relations. Table 7 shows the specific relations in which pair2vec made the largest absolute impact. The gains are particularly significant for relations where fastText embeddings provide limited signal. For example, the accuracy for substance meronyms goes from 3.8% to 14.5%. In some cases, there is also a synergistic effect; for instance, in noun+less , pair2vec alone scored 0% accuracy, but mixing it with 3CosAdd , which got 4.8% on its own, yielded 16% accuracy. These results, alongside our experiments in Section 4, strongly suggest that pair2vec encodes information complementary to that in single-word embedding methods such as fastText and ELMo. To further explore how pair2vec encodes such complementary information, we consider a setting similar to that of knowledge base completion: given a Hearst-like context pattern c and a single word x , predict the other word y from the entire vocabulary. We rank candidate words y based on the scoring function in our training objective: R ( x, y )  C ( c ) . We use a fixed set of example relations and manually define their predictive context patterns and a small set of candidate words x . Table 8 shows the top three y words. The model embeds ( x, y ) pairs close to contexts that reflect their relationship. For example, substituting Portland in the city-state pattern ( in X, Y. ), the top two words are Oregon and Maine , both US states with cities named Portland. When used with the city-city pattern ( from X to Y. ), the top two words are Salem and Astoria , both cities in Oregon. The word-context interaction often captures multiple relations; for example, Monet is used to refer to the painter ( profession ) as well as his paintings. As intended, pair2vec captures the three-way word-word-context interaction, and not just the two-way word-context interaction (as in single-word embeddings). This profound difference allows pair2vec to complement single-word embeddings with additional information. Pretrained Word Embeddings Many state-of-the-art models initialize their word representations using pretrained embeddings such as word2vec (Mikolov et al., 2013a) or ELMo (Peters et al., 2018). These representations are typically trained using an interpretation of the Distributional Hy-Relation Context X Y (Top pothesis (Harris, 1954) in which the bivariate distribution of target words and contexts is modeled. Our work deviates from the word embedding literature in two major aspects. First, our goal is to represent word pairs , not individual words. Second, our new PMI formulation models the trivari-ate word-word-context distribution. Experiments show that our pair embeddings can complement single-word embeddings. Mining Textual Patterns There is extensive literature on mining textual patterns to predict relations between words (Hearst, 1992; Snow et al., 2005; Turney, 2005; Riedel et al., 2013; Van de Cruys, 2014; Toutanova et al., 2015; Shwartz and Dagan, 2016). These approaches focus mostly on relations between pairs of nouns (perhaps with the exception of VerbOcean (Chklovski and Pantel, 2004)). More recently, they have been expanded to predict relations between unrestricted pairs of words (Jameel et al., 2018; Espinosa Anke and Schockaert, 2018), assuming that each word-pair was observed together during pretraining. Washio and Kato (2018a,b) relax this assumption with a compositional model that can represent any pair, as long as each word appeared (individually) in the corpus. These methods are evaluated on either intrinsic relation prediction tasks, such as BLESS (Ba-roni and Lenci, 2011) and CogALex (Santus et al., 2016), or knowledge-base population benchmarks, e.g. FB15 (Bordes et al., 2013). To the best of our knowledge, our work is the first to integrate pattern-based methods into modern high-performing semantic models and evaluate their impact on complex end-tasks like QA and NLI. Integrating Knowledge in Complex Models Ahn et al. (2016) integrate Freebase facts into a language model using a copying mechanism over fact attributes. Yang and Mitchell (2017) modify the LSTM cell to incorporate WordNet and NELL knowledge for event and entity extraction. For cross-sentence inference tasks, Weissenborn et al. (2017), Bauer et al. (2018), and Mihaylov and Frank (2018) dynamically refine word representations by reading assertions from ConceptNet and Wikipedia abstracts. Our approach, on the other hand, relies on a relatively simple extension of existing cross-sentence inference models. Furthermore, we do not need to dynamically retrieve and process knowledge base facts or Wikipedia texts, and just pretrain our pair vectors in advance. KIM (Chen et al., 2017) integrates word-pair vectors into the ESIM model for NLI in a very similar way to ours. However, KIM's word-pair vectors contain only hand-engineered word-relation indicators from WordNet, whereas our word-pair vectors are automatically learned from unlabeled text. Our vectors can therefore reflect relation types that do not exist in WordNet (such as profession ) as well as word pairs that do not have a direct link in WordNet (e.g. bronze and statue ); see Table 8 for additional examples. We presented new methods for training and using word pair embeddings that implicitly represent background knowledge. Our pair embeddings are computed as a compositional function of the individual word representations, which is learned by maximizing a variant of the PMI with the contexts in which the the two words co-occur. Experiments on cross-sentence inference benchmarks demonstrated that adding these representations to existing models results in sizable improvements for both in-domain and adversarial settings. Published concurrently with this paper, BERT (Devlin et al., 2018), which uses a masked language model objective, has reported dramatic gains on multiple semantic benchmarks including question-answering, natural language inference, and named entity recognition. Potential avenues for future work include multitasking BERT with pair2vec in order to more directly incorporate reasoning about word pair relations into the BERT objective. We would like to thank Anna Rogers (Gladkova), Qian Chen, Koki Washio, Pranav Rajpurkar, and Robin Jia for their help with the evaluation. We are also grateful to members of the UW and FAIR NLP groups, and anonymous reviewers for their thoughtful comments and suggestions.
abstain objective method objective abstain abstain abstain abstain abstain other abstain result other method abstain abstain result method abstain result result abstain abstain abstain method method abstain abstain other other abstain abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain other abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain method method abstain abstain abstain method abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other other other method objective objective abstain other other other other other objective other other other abstain method other abstain abstain objective method abstain abstain abstain other other